MCD: Overcoming the Data Download Bottleneck in Data Centers

2013

The data download problem in data centers describes the increasingly common task of coordinated loading of identical data to a large number of nodes. Data download is seen as a significant problem in exascale HPC applications. Uncoor-dinated reading from a central file server creates contention at the file server and its network interconnect. We propose and evaluation a reliable multicast based approach to solve the data download problem. The MCD system builds a logical multi-rooted tree based on the physical network topology and uses the logical view for a two-phase approach. In the first phase, the data is multicasted to all nodes. In the second phase, the logical tree is used for an effi…