2021 High Performance Computing Practical Lecture 10.2 Distributed Deep Learning Part2 👨‍💻

Practical Lecture 10.2 - Distributed Deep Learning 👨‍💻 - Part Two Invited Lecture given by PhD Student Rocco Sedona https://www.fz-juelich.de/SharedDocs/Personen/IAS/JSC/EN/staff/sedona_r.html Advanced Scientific Computing 16 university lectures with additional practical lectures for hands-on exercises in context University of Iceland, School of Engineering and Natural Sciences Faculty of Industrial Engineering, Mechanical Engineering and Computer Science Spring 2021 Course Outline 1. High Performance Computing 2. Parallel Programming with MPI 3. Parallelization Fundamentals 4. Advanced MPI Techniques 5. Parallel Algorithms & Data Structures 6. Parallel Programming with OpenMP 7. Hybrid Programming & Patterns 8. Debugging & Profiling & Performance Analysis 9. Accelerators & Graphical Processing Units 10. Parallel & Scalable Machine & Deep Learning 11. HPC in Health & Neurosciences 12. Computational Fluid Dynamics & Finite Elements 13. Systems Biology & Bioinformatics 14. Molecular Systems & Material Sciences 15. Terrestrial Systems & Climate 16. Epilogue Lecture Outline Part One: Introduction Key concepts from Lecture 10.1 Other Important Concepts Distributed Training: Theory Part Two: Frameworks Horovod DeepSpeed Others A Remote Sensing Application