Catalog

The challenge of distributed training

Algorithmic challenges

1. Data parallelism or model parallelism
2. Synchronous or asynchronous
3. Large batch , Affect the accuracy of the model
4. Warm up , Adjust learning rate （ Linear rise ,LARC/LARS）
5. Add noise to the gradient
6. The choice of optimizer （SGD,Momentum,Adam,Rmsprop）
7. Balance speed and accuracy

Engineering challenges

• CPU and GPU Uneven performance improvement
• Expand vertically first , And then expand horizontally
• GPU model ,NVLink,NVSwitch,DGX,10G/25G/100G/200G Matching and selection
• Mixing accuracy
• GPU Direct RDMA（Infiniband）
• from CPU Uninstall some operations to GPU（e.g. Data pre