HPNL: A High-Performance Light Weight Network Library for Big Data Application

In this video from the 2019 OpenFabrics Workshop in Austin, Jian Zhang form Intel presents: HPNL: A High-Performance Light Weight Network Library for Big Data Application. "Nowadays data is growing at a faster rate than ever before and it presents new challenges for large-scale Big Data analytics. Spark is expected to achieve high throughput & ultra-low latency for different workload. However, previous studies showed it can be improved by using RDMA networking. New emerging persistent memory technologies like DCPMM can offer persistency with memory-like speed, combining RDMA and persistent memory create tremendous opportunities for Spark Shuffle acceleration. We present high-performance network library (HPNL), a light weight network library built on Libfabric for big data application. It provides protocol-independent networking framework, C/JAVA API and high-level abstraction to let developer easily replace other TCP/IP based network library, like ASIO or Netty, without knowing the low-level details of RDMA programming model. We will showcase the benchmark result compared with other network libraries. One design principle of HPNL is to ease storage and network stack integration and supports API to access remote persistent memory. We will also present a new Spark Shuffle Manager based on HPNL, which leveraging non-volatile persistent memory as shuffle storage and RDMA for network transmission. Our evaluation shows that this approach significantly improves the Spark end-to-end job execution time by up to 10x." Learn more: https://www.openfabrics.org/2019-workshop-agenda-and-abstracts/ Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter