Article

Server-directed collective I/O in Panda

Authors:

M. WinslettAuthors Info & Claims

Supercomputing '95: Proceedings of the 1995 ACM/IEEE conference on Supercomputing

Pages 57 - es

https://doi.org/10.1145/224170.224371

Published: 08 December 1995 Publication History

Abstract

We present the architecture and implementation results for Panda 2.0, a library for input and output of multidimensional arrays on parallel and sequential platforms. Panda achieves remarkable performance levels on the IBM SP2, showing excellent scalability as data size increases and as the number of nodes increases, and provides throughputs close to the full capacity of the AIX file system on the SP2 we used. We argue that this good performance can be traced to Panda's use of server-directed i/o (a logical-level version of disk-directed i/o [Kotz94b]) to perform array i/o using sequential disk reads and writes, a very high level interface for collective i/o requests, and built-in facilities for arbitrary rearrangements of arrays during i/o. Other advantages of Panda's approach are ease of use, easy application portability, and a reliance on commodity system software.

Supplementary Material

PS File (a57-seamons.ps)

Download
358.69 KB

References

[1]

R. Bennett, K. Bryant, A. Sussman, R. Das, and J. Saltz, Jovian: A framework for optimizing parallel I/O, Proceedings of the 1994 Scalable Parallel Libraries Conference, pages 10-20. IEEE Computer Society Press, October 1994.]]

[2]

R. Bordawekar, J. Miguel del Rosario, and A. Choudary, Design and Evaluation of Primitives for Parallel I/O, Proceedings of Supercomputing '93, pages 452-461, 1993.]]

Digital Library

[3]

P. Brezany, T. Mueck, and E. Schikuta, Language, compiler and parallel database support for I/O intensive applications, Proceedings of the High Performance Computing and Networking 1995 Europe Conference, Milano, Italy, May 1995, Springer-Verlag.]]

Digital Library

[4]

P. F. Corbett and D. G. Feitelson, Vesta file system programmer's reference. Technical Report Research Report RC 19898 (88058), IBM T.J. Watson Research Center, Yorktown Heights, NY 10598, October 1994. Version 1.01.]]

[5]

P. Corbett, D. Feitelson, Y. Hsu, J. Prost, M. Snir, S. Fineberg, B. Nitzberg, B. Traversat, and P. Wong. MPI-IO: A Parallel File I/O Interface for MPI, Technical Report NAS-95-002, NASA Ames Research Center, January 1995.]]

[6]

J. M. del Rosario, M. Harry, A. Choudhary, The Design of VIP-FS: A Virtual, Parallel File System for High Performance Parallel and Distributed Computing, Technical Report SCCS-628, NPAC, Syracuse, NY, May 1994.]]

[7]

N. Galbreath, W. Gropp, and D. Levine, Applications-Driven Parallel I/O, Proceedings of Supercomputing '93, pages 462-471, 1993.]]

Digital Library

[8]

J. F. Karpovich, A. S. Grimshaw, J. C. French, Extensible File Systems (ELFS): An Object-Oriented Approach to High Performance File I/O, Proceedings of the International Conference on Object-Oriented Programming, Systems, Languages, and Applications, Portland OR, August 1994.]]

Digital Library

[9]

D. Kotz, Throughput of Existing Multiprocessor File Systems (An Informal Study), Dartmouth PCS-TR93-190, 1993.]]

Digital Library

[10]

D. Kotz, and N. Nieuwejaar, Dynamic file-access characteristics of a production parallel scientific workload. In Proceedings of Supercomputing '94, pages 640-649, November 1994.]]

Digital Library

[11]

D. Kotz, Disk-Directed I/O for MIMD Multiprocessors, First Symposium on Operating Systems Design and Implementation, November 1994.]]

Digital Library

[12]

D. Kotz, Expanding Potential for Disk-Directed I/O. Dartmouth PCS-TR95-254, submitted to SPDP '95.]]

Digital Library

[13]

D. Kotz, Disk-Directed I/O for Out-of-Core Computation. Dartmouth TR PCS-TR95-251, submitted to HPDC '95.]]

Digital Library

[14]

B. Pasquale, and G. Polyzos, Dynamic I/O Characterization of I/O intensive Scientific Applications, Technical Report No. CS94-364, University of California, San Diego, April 1994.]]

[15]

P. Pierce, A Concurrent File System for a Highly Parallel Mass Storage Subsystem, Proceedings of the 4th Conference on Hypercube Computers and Applications, Monterey, March 1989., pp. 155-160.]]

[16]

A. Purakayastha, C. Ellis, D. Kotz, N. Nieuwejaar, and M. Best, Characterizing Parallel File-Access Patterns on a Large-Scale Multiprocessor, Duke University Technical Report CS-1994-33, October 1994.]]

Digital Library

[17]

K. E. Seamons and M. Winslett, Physical Schemas for Large Multidimensional Arrays in Scientific Computing Applications, Proceedings of the 7th International Working Conference on Scientific and Statistical Database Management, Charlottesville, Virginia, September 1994.]]

Digital Library

[18]

K. E. Seamons and M. Winslett, An Efficient Abstract Interface for Multidimensional Array I/O, Proceedings of Supercomputing '94, Washington D.C., November 1994.]]

Digital Library

[19]

E. Seligman, and A. Beguelin, High-Level Fault Tolerance in Distributed Programs, School of Computer Science, Carnegie Mellon University, USA. Technical report CMU-CS-94-223, December 1994.]]

Digital Library

Cited By

Liao JYin DPeng X(2018)Block I/O Scheduling on Storage Servers of Distributed File SystemsJournal of Grid Computing10.1007/s10723-017-9423-116:2(299-316)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10723-017-9423-1
Son SSehrish SLiao WOldfield RChoudhary A(2017)Reducing I/O variability using dynamic I/O path characterization in petascale storage systemsThe Journal of Supercomputing10.1007/s11227-016-1904-773:5(2069-2097)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1007/s11227-016-1904-7
Nieplocha JFoster IKendall R(2016)ChemIo: High Performance Parallel I/o for Computational Chemistry ApplicationsThe International Journal of High Performance Computing Applications10.1177/10943420980120030412:3(345-363)Online publication date: 26-Jul-2016
https://doi.org/10.1177/109434209801200304
Show More Cited By

Index Terms

Recommendations

Panda: Fast access to persistent arrays using high level interfaces and server directed input/output
Persistent Array Access Using Server-Directed I/O
SSDBM '96: Proceedings of the Eighth International Conference on Scientific and Statistical Database Management

Large multidimensional arrays are a common data type in high-performance scientific applications. Without special techniques for handling access to these arrays, I/O can easily become a large fraction of execution time for applications using these ...
Server directed file domain allocation for noncontiguous file access
CCGRID '04: Proceedings of the 2004 IEEE International Symposium on Cluster Computing and the Grid

Parallel I/O systems frequently use collective I/O to improve the performance of applications that have small and noncontiguous data access patterns. The performance of a collective I/O implementation depends on how well its I/O libraries and parallel ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

Supercomputing '95: Proceedings of the 1995 ACM/IEEE conference on Supercomputing

December 1995

875 pages

ISBN:0897918169

DOI:10.1145/224170

Chairman:
Sid Karin
San Diego Supercomputer Center, San Diego, CA

Copyright � 1995 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 December 1995

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SC '95

Sponsor:

SIGARCH
IEEE-CS

SC '95: International Conference for High Performance Computing, Networking, Storage and Analysis

December 4 - 8, 1995

California, San Diego, USA

Acceptance Rates

Supercomputing '95 Paper Acceptance Rate 69 of 241 submissions, 29%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

103
Total Citations
View Citations
157
Total Downloads

Downloads (Last 12 months)4
Downloads (Last 6 weeks)3

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Liao JYin DPeng X(2018)Block I/O Scheduling on Storage Servers of Distributed File SystemsJournal of Grid Computing10.1007/s10723-017-9423-116:2(299-316)Online publication date: 1-Jun-2018
https://dl.acm.org/doi/10.1007/s10723-017-9423-1
Son SSehrish SLiao WOldfield RChoudhary A(2017)Reducing I/O variability using dynamic I/O path characterization in petascale storage systemsThe Journal of Supercomputing10.1007/s11227-016-1904-773:5(2069-2097)Online publication date: 1-May-2017
https://dl.acm.org/doi/10.1007/s11227-016-1904-7
Nieplocha JFoster IKendall R(2016)ChemIo: High Performance Parallel I/o for Computational Chemistry ApplicationsThe International Journal of High Performance Computing Applications10.1177/10943420980120030412:3(345-363)Online publication date: 26-Jul-2016
https://doi.org/10.1177/109434209801200304
Garcia-Carballeira FCalderon ACarretero JFernandez JPerez J(2016)The Design of the Expand Parallel File SystemThe International Journal of High Performance Computing Applications10.1177/109434200301700100317:1(21-37)Online publication date: 26-Jul-2016
https://doi.org/10.1177/1094342003017001003
Kuperman YMoscovici ENider JLadelsky RGordon ATsafrir D(2016)Paravirtual Remote I/OACM SIGARCH Computer Architecture News10.1145/2980024.287237844:2(49-65)Online publication date: 25-Mar-2016
https://dl.acm.org/doi/10.1145/2980024.2872378
Kuperman YMoscovici ENider JLadelsky RGordon ATsafrir D(2016)Paravirtual Remote I/OACM SIGOPS Operating Systems Review10.1145/2954680.287237850:2(49-65)Online publication date: 25-Mar-2016
https://doi.org/10.1145/2954680.2872378
Kuperman YMoscovici ENider JLadelsky RGordon ATsafrir D(2016)Paravirtual Remote I/OACM SIGPLAN Notices10.1145/2954679.287237851:4(49-65)Online publication date: 25-Mar-2016
https://dl.acm.org/doi/10.1145/2954679.2872378
Kuperman YMoscovici ENider JLadelsky RGordon ATsafrir DConte TZhou Y(2016)Paravirtual Remote I/OProceedings of the Twenty-First International Conference on Architectural Support for Programming Languages and Operating Systems10.1145/2872362.2872378(49-65)Online publication date: 25-Mar-2016
https://dl.acm.org/doi/10.1145/2872362.2872378
Isaila FCarretero JRoss RVarela CCastro HBarrios C(2016)CLARISSEProceedings of the 16th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing10.1109/CCGrid.2016.24(346-355)Online publication date: 16-May-2016
https://dl.acm.org/doi/10.1109/CCGrid.2016.24
Chen YLu YAmritkar PThakur RZhuang Y(2015)Performance model-directed data sieving for high-performance I/OThe Journal of Supercomputing10.1007/s11227-014-1277-871:6(2066-2090)Online publication date: 1-Jun-2015
https://dl.acm.org/doi/10.1007/s11227-014-1277-8
Show More Cited By

View Options

View options

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents