Search results for " distributed"
showing 10 items of 235 documents
AnySeq: A High Performance Sequence Alignment Library based on Partial Evaluation
2020
Sequence alignments are fundamental to bioinformatics which has resulted in a variety of optimized implementations. Unfortunately, the vast majority of them are hand-tuned and specific to certain architectures and execution models. This not only makes them challenging to understand and extend, but also difficult to port to other platforms. We present AnySeq - a novel library for computing different types of pairwise alignments of DNA sequences. Our approach combines high performance with an intuitively understandable implementation, which is achieved through the concept of partial evaluation. Using the AnyDSL compiler framework, AnySeq enables the compilation of algorithmic variants that ar…
A Big Data Approach for Sequences Indexing on the Cloud via Burrows Wheeler Transform
2020
Indexing sequence data is important in the context of Precision Medicine, where large amounts of ``omics'' data have to be daily collected and analyzed in order to categorize patients and identify the most effective therapies. Here we propose an algorithm for the computation of Burrows Wheeler transform relying on Big Data technologies, i.e., Apache Spark and Hadoop. Our approach is the first that distributes the index computation and not only the input dataset, allowing to fully benefit of the available cloud resources.
Parallel In-Memory Evaluation of Spatial Joins
2019
The spatial join is a popular operation in spatial database systems and its evaluation is a well-studied problem. As main memories become bigger and faster and commodity hardware supports parallel processing, there is a need to revamp classic join algorithms which have been designed for I/O-bound processing. In view of this, we study the in-memory and parallel evaluation of spatial joins, by re-designing a classic partitioning-based algorithm to consider alternative approaches for space partitioning. Our study shows that, compared to a straightforward implementation of the algorithm, our tuning can improve performance significantly. We also show how to select appropriate partitioning parame…
Burrows Wheeler Transform on a Large Scale: Algorithms Implemented in Apache Spark
2021
With the rapid growth of Next Generation Sequencing (NGS) technologies, large amounts of "omics" data are daily collected and need to be processed. Indexing and compressing large sequences datasets are some of the most important tasks in this context. Here we propose algorithms for the computation of Burrows Wheeler transform relying on Big Data technologies, i.e., Apache Spark and Hadoop. Our algorithms are the first ones that distribute the index computation and not only the input dataset, allowing to fully benefit of the available cloud resources.
Concurrent Computing with Shared Replicated Memory
2019
The behavioural theory of concurrent systems states that any concurrent system can be captured by a behaviourally equivalent concurrent Abstract State Machine (cASM). While the theory in general assumes shared locations, it remains valid, if different agents can only interact via messages, i.e. sharing is restricted to mailboxes. There may even be a strict separation between memory managing agents and other agents that can only access the shared memory by sending query and update requests to the memory agents. This article is dedicated to an investigation of replicated data that is maintained by a memory management subsystem, whereas the replication neither appears in the requests nor in th…
Self-stabilizing Balls & Bins in Batches
2016
A fundamental problem in distributed computing is the distribution of requests to a set of uniform servers without a centralized controller. Classically, such problems are modeled as static balls into bins processes, where $m$ balls (tasks) are to be distributed to $n$ bins (servers). In a seminal work, Azar et al. proposed the sequential strategy \greedy{d} for $n=m$. When thrown, a ball queries the load of $d$ random bins and is allocated to a least loaded of these. Azar et al. showed that $d=2$ yields an exponential improvement compared to $d=1$. Berenbrink et al. extended this to $m\gg n$, showing that the maximal load difference is independent of $m$ for $d=2$ (in contrast to $d=1$). W…
Low-Power Wide-Area Networks for Sustainable IoT
2019
Low-power wide-area (LPWA) networks are attracting extensive attention because of their abilities to offer low-cost and massive connectivity to Internet of Things (IoT) devices distributed over wide geographical areas. This article provides a brief overview on the existing LPWA technologies and useful insights to aid the large-scale deployment of LPWA networks. Particularly, we first review the currently competing candidates of LPWA networks, such as narrowband IoT (NB-IoT) and long range (LoRa), in terms of technical fundamentals and large-scale deployment potential. Then we present two implementation examples on LPWA networks. By analyzing the field-test results, we identify several chall…
Investigating Low Level Protocols for Wireless Body Sensor Networks
2016
The rapid development of medical sensors has increased the interest in Wireless Body Area Network (WBAN) applications where physiological data from the human body and its environment is gathered, monitored, and analyzed to take the proper measures. In WBANs, it is essential to design MAC protocols that ensure adequate Quality of Service (QoS) such as low delay and high scalability. This paper investigates Medium Access Control (MAC) protocols used in WBAN, and compares their performance in a high traffic environment. Such scenario can be induced in case of emergency for example, where physiological data collected from all sensors on human body should be sent simultaneously to take appropria…
Unified System for Processing Real and Simulated Data in the ATLAS Experiment
2015
The physics goals of the next Large Hadron Collider run include high precision tests of the Standard Model and searches for new physics. These goals require detailed comparison of data with computational models simulating the expected data behavior. To highlight the role which modeling and simulation plays in future scientific discovery, we report on use cases and experience with a unified system built to process both real and simulated data of growing volume and variety.
The IceProd framework: distributed data processing for the IceCube neutrino observatory
2015
IceCube is a one-gigaton instrument located at the geographic South Pole, designed to detect cosmic neutrinos, identify the particle nature of dark matter, and study high-energy neutrinos themselves. Simulation of the IceCube detector and processing of data require a significant amount of computational resources. This paper presents the first detailed description of IceProd, a lightweight distributed management system designed to meet these requirements. It is driven by a central database in order to manage mass production of simulations and analysis of data produced by the IceCube detector. IceProd runs as a separate layer on top of other middleware and can take advantage of a variety of c…