Search results for "Namespace"
showing 4 items of 4 documents
One Phase Commit: A Low Overhead Atomic Commitment Protocol for Scalable Metadata Services
2012
As the number of client machines in high end computing clusters increases, the file system cannot keep up with the resulting volume of requests, using a centralized metadata server. This problem will be even more prominent with the advent of the exascale computing age. In this context, the centralized metadata server represents a bottleneck for the scaling of the file system performance as well as a single point of failure. To overcome this problem, file systems are evolving from centralized metadata services to distributed metadata services. The metadata distribution raises a number of additional problems that must be taken into account. In this paper we will focus on the problem of managi…
Streamlining distributed Deep Learning I/O with ad hoc file systems
2021
With evolving techniques to parallelize Deep Learning (DL) and the growing amount of training data and model complexity, High-Performance Computing (HPC) has become increasingly important for machine learning engineers. Although many compute clusters already use learning accelerators or GPUs, HPC storage systems are not suitable for the I/O requirements of DL workflows. Therefore, users typically copy the whole training data to the worker nodes or distribute partitions. Because DL depends on randomized input data, prior work stated that partitioning impacts DL accuracy. Their solutions focused mainly on training I/O performance on a high-speed network but did not cover the data stage-in pro…
Middleware with QoS Support to Control Intelligent Systems
2008
This paper presents the architecture of a middleware for control component-based distributed systems. The fundamental idea underlying this architecture is the use of a hierarchical communications structure called "logical namespace tree" and a structured set of control processes interconnected, called "logical sensors graph". This architecture is named frame sensor adapter control (FSA-Ctrl). The aim of FSA-Ctrl is to provide a simple interface to specify the control actions and ensure a communication with some specified QoS parameters restrictions. In this architecture both systems, communication and control, manages the QoS policies. The communication system is based on the data distribut…
NVMM-Oriented Hierarchical Persistent Client Caching for Lustre
2021
In high-performance computing (HPC), data and metadata are stored on special server nodes and client applications access the servers’ data and metadata through a network, which induces network latencies and resource contention. These server nodes are typically equipped with (slow) magnetic disks, while the client nodes store temporary data on fast SSDs or even on non-volatile main memory (NVMM). Therefore, the full potential of parallel file systems can only be reached if fast client side storage devices are included into the overall storage architecture. In this article, we propose an NVMM-based hierarchical persistent client cache for the Lustre file system (NVMM-LPCC for short). NVMM-LPC…