0000000001155570
AUTHOR
Alberto Miranda
GekkoFS - A Temporary Distributed File System for HPC Applications
We present GekkoFS, a temporary, highly-scalable burst buffer file system which has been specifically optimized for new access patterns of data-intensive High-Performance Computing (HPC) applications. The file system provides relaxed POSIX semantics, only offering features which are actually required by most (not all) applications. It is able to provide scalable I/O performance and reaches millions of metadata operations already for a small number of nodes, significantly outperforming the capabilities of general-purpose parallel file systems. The work has been funded by the German Research Foundation (DFG) through the ADA-FS project as part of the Priority Programme 1648. It is also support…
Streamlining distributed Deep Learning I/O with ad hoc file systems
With evolving techniques to parallelize Deep Learning (DL) and the growing amount of training data and model complexity, High-Performance Computing (HPC) has become increasingly important for machine learning engineers. Although many compute clusters already use learning accelerators or GPUs, HPC storage systems are not suitable for the I/O requirements of DL workflows. Therefore, users typically copy the whole training data to the worker nodes or distribute partitions. Because DL depends on randomized input data, prior work stated that partitioning impacts DL accuracy. Their solutions focused mainly on training I/O performance on a high-speed network but did not cover the data stage-in pro…
Random Slicing: Efficient and Scalable Data Placement for Large-Scale Storage Systems
The ever-growing amount of data requires highly scalable storage solutions. The most flexible approach is to use storage pools that can be expanded and scaled down by adding or removing storage devices. To make this approach usable, it is necessary to provide a solution to locate data items in such a dynamic environment. This article presents and evaluates the Random Slicing strategy, which incorporates lessons learned from table-based, rule-based, and pseudo-randomized hashing strategies and is able to provide a simple and efficient strategy that scales up to handle exascale data. Random Slicing keeps a small table with information about previous storage system insert and remove operations…
GekkoFS — A Temporary Burst Buffer File System for HPC Applications
Many scientific fields increasingly use high-performance computing (HPC) to process and analyze massive amounts of experimental data while storage systems in today’s HPC environments have to cope with new access patterns. These patterns include many metadata operations, small I/O requests, or randomized file I/O, while general-purpose parallel file systems have been optimized for sequential shared access to large files. Burst buffer file systems create a separate file system that applications can use to store temporary data. They aggregate node-local storage available within the compute nodes or use dedicated SSD clusters and offer a peak bandwidth higher than that of the backend parallel f…
Collaborative Governance in Portugal – Yes or No? A critical approach based on an empirical survey
Collaborative governance can be defined as a concerted type of decision-making and collective action in which government bodies and private sector stakeholders interact as equal partners with the aim of achieving outcomes of interest for both parties. In practical terms, this is a type of governance where special meetings and negotiations are held – typically in somewhat elitist and secretive settings – with the aim of defining shared responsibilities and benefits associated with joint public-private ventures. Academic authors in favour of the adoption of collaborative governance in Portugal tend to present a rather positive view on the merits of this approach. They argue that it is potenti…