Training a machine translation system takes a week or two of
GPU time.  Bored waiting for systems to train, we thought it would be
fun to use multiple GPUs. But that consumes a lot of bandwidth, so
optimizing communication is critical for multiple nodes.  Most of the
prior work experimented with vision and speech with convolutional neural
networks that use parameters for every data point.  In contrast, machine
translation's recurrent networks are dominated by word embeddings and
only a few words appear in a batch of ~90 sentences, which makes
gradients very skewed.  We find these gradients can be exploited to
compress communication and discuss strategies for further optimization,
such as tuning your momentum.