6533b829fe1ef96bd128a29a

RESEARCH PRODUCT

A configurable rule based classful token bucket filter network request scheduler for the lustre file system

Xi LiYingjin QianShuichi IharaAndré BrinkmannJürgen KaiserLingfang ZengTim Süß

subject

File systemComputer scienceQuality of service020206 networking & telecommunications020207 software engineeringDenial-of-service attackRule-based system02 engineering and technologycomputer.software_genreObject storageRemote procedure callServer0202 electrical engineering electronic engineering information engineeringOperating systemLustre (file system)computer

description

HPC file systems today work in a best-effort manner where individual applications can flood the file system with requests, effectively leading to a denial of service for all other tasks. This paper presents a classful Token Bucket Filter (TBF) policy for the Lustre file system. The TBF enforces Remote Procedure Call (RPC) rate limitations based on (potentially complex) Quality of Service (QoS) rules. The QoS rules are enforced in Lustre's Object Storage Servers, where each request is assigned to an automatically created QoS class.The proposed QoS implementation for Lustre enables various features for each class including the support for high-priority and real-time requests even under heavy load and the utilization of spare bandwidth by less important tasks under light load. The framework also enables dependent rules to change a job's RPC rate even at very small timescales. Furthermore, we propose a Global Rate Limiting (GRL) algorithm to enforce system-wide RPC rate limitations.

yearjournalcountryeditionlanguage
2017-11-12Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
https://doi.org/10.1145/3126908.3126932