Compute, Time and Energy Characterization of Encoder-Decoder Networks with Automatic Mixed Precision Training

Siddharth Samsi, Michael Jones, Mark M. Veillette

arXiv preprint · 18 August 2020

In one sentence. Characterizes compute, time and energy costs of training a UNet for precipitation nowcasting, showing mixed-precision and distributed training cut training time without losing accuracy.

Abstract

Deep neural networks have shown great success in many diverse fields. The training of these networks can take significant amounts of time, compute and energy. As datasets get larger and models become more complex, the exploration of model architectures becomes prohibitive. In this paper we examine the compute, energy and time costs of training a UNet based deep neural network for the problem of predicting short term weather forecasts (called precipitation Nowcasting). By leveraging a combination of data distributed and mixed-precision training, we explore the design space for this problem. We also show that larger models with better performance come at a potentially incremental cost if appropriate optimizations are used. We show that it is possible to achieve a significant improvement in training time by leveraging mixed-precision training without sacrificing model performance. Additionally, we find that a 1549% increase in the number of trainable parameters for a network comes at a relatively smaller 63.22% increase in energy usage for a UNet with 4 encoding layers.

Labels

Labels and summary are generated by Claude and reviewed weekly. Wrong? Edit this entry.