Scalable Parallel I/O in the Exa-scale Era

Hou, Kaiyuan

doi:https://doi.org/10.21985/n2-tnt1-hj14

Work

Scalable Parallel I/O in the Exa-scale Era

Public

Download PDF

Download All Files (.zip)

The speed of the storage device has long lagged behind the computation speed of processors.As a result, the I/O performance of storage systems in a supercomputer fails to keep up with its computational power. This gap continues to widen in modern supercomputers. On future exascale supercomputers, this issue can worsen to the extent that I/O operations become a bottleneck to many applications. In this dissertation, we experiment with three ideas to improve I/O performance and scalability on supercomputers.One of them is to try to utilize new types of storage hardware being introduced in modern supercomputers. They bridge the performance gap between the primary storage system and compute nodes, but also complicate the I/O stack. Another is to compress the data to ease the demand on I/O bandwidth. Compression reduces the bandwidth required to store the same amount of information but makes the data more difficult to access and manage. The other is to improve the I/O pattern to utilize I/O resources more efficiently. Large and contiguous I/O requests can generally be handled more efficiently; however, it takes computation and memory resources to reorganize the data into the desired pattern. We present four projects that study the practicality of the three ideas for future exascale HPC environments on various applications.The first one tries to utilize the burst buffer to perform I/O aggregation that combines and reorders I/O requests so it can be handled more efficiently by the file system. The second one is an experiment on the idea of compressing checkpoint data to reduce I/O time, and file system workload. The third one introduces a framework we designed to enable efficient parallel I/O on compressed variables in classic NetCDF files. The final one presents an HDF5 VOL we developed that stores datasets in a log-based storage layout that results in efficient I/O patterns. We show that the ideas mentioned above are effective and provide a decent solution to scalable I/O in the exascale era.

Creator