What Is OrangeFS?
OrangeFS is a next-generation parallel file system based on PVFS for compute and storage clusters of the future. More specifically, it is a file system for:
- High-performance computing (HPC)
- Parallel clusters (many computers linked together, running as one)
- Customers needing a standard, open source, easily installable and highly scalable storage solution
- Cutting-edge research for academic and government initiatives
- Commodity network and business applications (at a rapidly increasing pace)
- Customers seeking an enterprise solution, including commercial-grade services from the experts who develop the software
What do OrangeFS and PVFS currently offer?
All PVFS requests involve objects, which hold file data, file and directory metadata, directory entries, or symbolic links. Every object’s unique handle, along with other metadata, is contained in a set of key/value pairs. Each object also includes a bytestream to hold actual file data.
One analogy to use for object-based storage is the difference between parking your own car and letting a valet do it for you (you never need to know where the car is). In PVFS, the metadata server is your valet.
Separation of Data and Metadata
After accessing one of the Metadata Servers once for a file’s location, a PVFS Client can thereafter interface directly with the data servers. This eliminates a major bottleneck.
MPI programs can supply a description of the data based on MPI_Datatypes, which can describe complex non-contiguous patterns of data and ultimately enable MPI file views. This allows highly efficient access to file data for parallel applications.
Multiple Network Support
PVFS uses a networking layer named BMI which provides a non-blocking message interface designed specifically for file systems. BMI has multiple implementation modules for a number of different networks used in high-performance computing including, TCP/IP, Myrinet, Infiniband, and Portals.
Stateless (Lockless) Servers
PVFS servers do not share any state with each other or with clients. If a server crashes another can easily be restarted in its place. Updates are performed without using locks.
PVFS clients and servers run at the user level and kernel modifications are not needed. An optional kernel module allows PVFS to be mounted like any other file system, or programs can link directly to a user interface such as MPI-IO or a Posix-like interface. All this makes PVFS easier to install and less prone to causing system crashes.
The PVFS interface integrates at the system level. Its similarities with the Linux VFS make it easy to implement as a mountable file system, but are equally adaptable to user-level interfaces such as MPI-IO or Posix-like interfaces. Exposure to many features of its underlying file system allows other interfaces to take advantage of them.
Scalable, Portable, Flexible
Disk performance of better than 1 GB/sec has been achieved on Linux clusters using standard IDE hard disks. PVFS brings state-of-the-art parallel I/O concepts to production parallel systems. It is designed to scale to petabytes of storage and provide access rates at 100s of GB/sec.
PVFS relaxes POSIX consistency semantics where necessary to improve stability and performance. Cluster file systems that enforce POSIX consistency require stateful clients with locking subsystems, reducing the stability of the system in the face of failures. These systems can be difficult to maintain due to overhead of lock management.
Optimized MPI-IO Support
PVFS is designed to support a number of access models, from collective I/O to independent I/O as well as non-contiguous and structured access patterns. PVFS provides an object-based, stateless client interface, leading to optimizations for metadata operations within MPI-IO.
PVFS operates on a wide variety of systems, including IA32, IA64, Opteron, PowerPC, Alpha, and MIPS. It is easily integrated into local or shared storage configurations, and provides support for high-end networking fabrics, such as Infiniband and Myrinet.
PVFS is easy to deploy and manage. It builds out-of-the-box on a wide-variety of Linux distributions and has only a few required dependencies. PVFS builds and runs directly on your Linux installation and requires no kernel patches or specific kernel versions.
PVFS is developed by a multi-institution team of parallel I/O, networking and storage experts. It embodies the expertise of designers who have worked for over a decade in the field of parallel I/O. PVFS continues to be used as a platform for active research in the parallel I/O field, the code is designed to be easy to augment for research purposes.
What has the OrangeFS team been up to?
OrangeFS uses server-to-server collective communication to improve the scalability of metadata operations. OrangsFS is implementing distributed directories to make directory operations more scalable. Another project is developing the ability to search metadata as an alternative to traditional path look-up for SSD support.
Small, Unaligned Accesses
The OrangeFS team is developing middleware-driven caching on the client side, including configurable semantics that provide a tradeoff between performance and consistency management.
Hardware failover provides invaluable services for large storage systems. Some installations want additional redundancy, and smaller installations want redundancy without the high cost of hardware solutions. The OrangeFS team is developing configurable redundancy and fail-over mechanisms into the file system. These allow different files to have different levels of redundancy as required by the application and also allow FS redundancy to be turned off for maximum performance. Currently you can replicate the files when they are marked immutable.
Secure Access Control
The OrangeFS team has developed an access control method based on signed credentials and capabilities that is designed to work with federated authentication mechanisms and yet maintain high-performance characteristics. Current research is focusing on flexible access control schemas beyond simple userID-based permissions.
The OrangeFS project is working to be production-ready, which includes improvements to the documentation base for all of PVFS.
OrangeFS provides a Windows client to allow Windows systems to seamlessly connect to the file system. Client applications can use files on OrangeFS in the same way as local files.