Search results for "Distributed computing"
showing 10 items of 622 documents
P2P-PL: A pattern language to design efficient and robust peer-to-peer systems
2017
To design peer-to-peer (P2P) software systems is a challenging task, because of their highly decentralized nature, which may cause unexpected emergent global behaviors. The last fifteen years have seen many P2P applications to come out and win favor with millions of users. From success histories of applications like BitTorrent, Skype, MyP2P we have learnt a number of useful design patterns. Thus, in this article we present a P2P pattern language (shortly, P2P-PL) which encompasses all the aspects that a fully effective and efficient P2P software system should provide, namely consistency of stored data, redundancy, load balancing, coping with asymmetric bandwidth, decentralized security. The…
Almost disjoint spanning trees: relaxing the conditions for completely independent spanning trees
2017
International audience; The search of spanning trees with interesting disjunction properties has led to the introduction of edge-disjoint spanning trees, independent spanning trees and more recently completely independent spanning trees. We group together these notions by dening (i, j)-disjoint spanning trees, where i (j, respectively) is the number of vertices (edges, respectively) that are shared by more than one tree. We illustrate how (i, j)-disjoint spanning trees provide some nuances between the existence of disjoint connected dominating sets and completely independent spanning trees. We prove that determining if there exist two (i, j)-disjoint spanning trees in a graph G is NP-comple…
GekkoFS - A Temporary Distributed File System for HPC Applications
2018
We present GekkoFS, a temporary, highly-scalable burst buffer file system which has been specifically optimized for new access patterns of data-intensive High-Performance Computing (HPC) applications. The file system provides relaxed POSIX semantics, only offering features which are actually required by most (not all) applications. It is able to provide scalable I/O performance and reaches millions of metadata operations already for a small number of nodes, significantly outperforming the capabilities of general-purpose parallel file systems. The work has been funded by the German Research Foundation (DFG) through the ADA-FS project as part of the Priority Programme 1648. It is also support…
Sorted deduplication: How to process thousands of backup streams
2016
The requirements of deduplication systems have changed in the last years. Early deduplication systems had to process dozens to hundreds of backup streams at the same time while today they are able to process hundreds to thousands of them. Traditional approaches rely on stream-locality, which supports parallelism, but which easily leads to many non-contiguous disk accesses, as each stream competes with all other streams for the available resources. This paper presents a new exact deduplication approach designed for processing thousands of backup streams at the same time on the same fingerprint index. The underlying approach destroys the traditionally exploited temporal chunk locality and cre…
DelveFS - An Event-Driven Semantic File System for Object Stores
2020
Data-driven applications are becoming increasingly important in numerous industrial and scientific fields, growing the need for scalable data storage, such as object storage. Yet, many data-driven applications cannot use object interfaces directly and often have to rely on third-party file system connectors that support only a basic representation of objects as files in a flat namespace. With sometimes millions of objects per bucket, this simple organization is insufficient for users and applications who are usually only interested in a small subset of objects. These huge buckets are not only lacking basic semantic properties and structure, but they are also challenging to manage from a tec…
Challenges and Solutions for Tracing Storage Systems
2018
IBM Spectrum Scale’s parallel file system General Parallel File System (GPFS) has a 20-year development history with over 100 contributing developers. Its ability to support strict POSIX semantics across more than 10K clients leads to a complex design with intricate interactions between the cluster nodes. Tracing has proven to be a vital tool to understand the behavior and the anomalies of such a complex software product. However, the necessary trace information is often buried in hundreds of gigabytes of by-product trace records. Further, the overhead of tracing can significantly impact running applications and file system performance, limiting the use of tracing in a production system. In…
Intelligent Cloud Storage Management for Layered Tiers
2018
Today, the cloud offers a large array of possibilities for storage, with this flexibility comes also complexity. This complexity stems from the variety of storage mediums, such as, blob storage or NoSQL tables, and also from the different cost tiers within these systems. A strategic thinking to navigate this complex cloud storage landscape is important, not only for cost saving but also for prioritizing information, this prioritization has wider implications in other domains such as the Big Data realm, especially for governance and efficiency. In this paper we propose a strategy centered around probabilistic graphical model (PGM), this heuristic oriented management and organizational strate…
Using an Object-Oriented Approach for Scalable Flexibility in Manufacturing
2012
In this paper, it is briefly explained how to configure the different resource modules that actually control the tasks of the physical device and put together a work cell consisting of other FMS resources. Finally, an example on building a workcell using a hierarchical approach is presented.
Real-Time Routing Selection in Flexible Manufacturing Systems
1993
Routing flexibility is one of the main peculiarities of Flexible Manufacturing Systems. This paper proposes three methods for real-time routing selection. The first one makes decisions comparing the current workload of machines in each alternative path. The second method considers the current workloads at the bottleneck machines in each allowed route. The third approach makes real-time decisions minimizing a merit index that represents a measure of the still required resource amount. The index is computed by short discrete-event simulation runs. Some case studies evaluate and compare the proposed approaches.
DEMAND Project: Bottom-Up Aggregation of Prosumers in Distribution Networks
2018
The paper explains the concept of the DEMAND project, whose aim is to develop a technical framework for allowing a Bottom-Up aggregation of prosumers connected to the distribution grid. Load and generation aggregation is a very current issue given the great potential that the coordinated management of distributed resources has on power systems operation and design. The novelty in DEMAND is the absence of a physical aggregator and the recourse to a virtual aggregation environment (VAE) for allowing the exchange of information among the prosumers in order to provide a service to the DSO. After a general description of the research project, the paper presents the services that the aggregated c…