Utilizing accelerators in Apache Spark presents opportunities for significant speedup of ETL, ML and DL applications. In this talk i’ll describe acceleration of Spark data shuffle across network and GPU accelerators using UCX library. Will show our existing solutions on speeding up host-to-host data transfer, using one-sided communications, device-to-device acceleration for Rapids Spark using tag API, and our roadmap to unify these two solutions.
Presented by Peter Rudenko, NVIDIA.
Learn more about the UCF Consortium at www.ucfconsortium.org.
