Friday, 18 September 2020

Award Winner 2015

Scalable End-to-End Data I/O over Enterprise and Data-Center Networks

by Yufei Ren

Press Release

Data-intensive applications in commercial cloud and scientific computing demand ultra-high-speed  end-to-end  data  access  capability  between  data  storage  and  computing locations.  Meanwhile,  advancements  in  hardware  and  systems  continuously  change  the landscape  of  data  centers’  core  capabilities,  i.e.,  computing,  networking,  and  storage.  The  two  trends  expose  new  research  and  development  challenges  and  opportunities  to bring  the  bare-metal  capacity  of  state-of-the-art  hardware  to  the  rising  needs  for  high performance by applications.

This  dissertation  focuses  on  designing  and  implementing  scalable  end-to-end  data transfer  solutions  and  storage  I/O  systems  over  ultra-high-speed  networks.  Particularly, we  present  a  scalable  memory-centric  software  framework  for  high-speed  data  transfer. At the data plane level, we introduce the hardware remote direct memory access (RDMA) of  HPC  to  ensure  zero-copy  along  the  entire  data  path  and  to  eliminate  the  overhead  of copying data between the layers of traditional network I/O stack. At the storage layer, we present  the  importance  of  data  locality  and  implement  a  data-affinity-aware  caching mechanism  in  the  storage  servers  with  non-uniform  memory  architecture  (NUMA).    At the control plane, we divide an end-to-end data path into pipelined stages, each of which uses  an  asynchronous  processing  model  to  efficiently  perform  resource-aware  task scheduling  and  optimization.  We  applied  this  framework  area  to  our  reference  systems design which consists of a front-end RDMA-based data transfer protocol for moving data within or across data centers over high-speed connectivity, and a backend NUMA-aware caching for accessing storage area networks efficiently. We rigorously tested the software implementation   and   demonstrated   its   viability   in   enabling   various   data-intensive applications  that  constantly  move  bulk  data  over  high-speed  enterprise  and  data-center networks.

Full Text

Hosted by SPEC RG