parallel training

Time-stamp: <2022-02-04 11:44:47"timestamp-wrapper"><2022-02-04 Fri 11:44>

Parallel Pytorch

• Tsinghua TDS, https://github.com/TsinghuaAI/TDS
• DeepSpeed
• Fully Sharded Data Parallel: faster AI training with fewer GPUs, https://engineering.fb.com/2021/07/15/open-source/fsdp/
• DietGPU, https://github.com/facebookresearch/dietgpu

Fast dataloader

• FFCV, https://ffcv.io/