XSEDE Gateways Resources
XSEDE supports and allocates resources on systems that provide gateway hosting services. Each system has slightly different capabilities and strengths as regards gateway hosting.
Each system's capabilities for supporting Gateways, and accepting and processing jobs from gateways, are described below. All XSEDE gateway resources offer community account support. Community accounts allow multiple users with a common XSEDE login to access software applications.
Gateway Support Features
Community Accounts: Gateways execute programs and access data through community accounts. Community accounts eliminate the need for gateway users to have their own accounts on XSEDE, allow gateway providers to cleanly separate their person research accounts from accounts that support gateways, and allow XSEDE Service Providers to implement optional limitations. For more about Community Accounts, see Science Gateways for Developers and Operators.
Gateway Server and Web Service Hosting: Gateway providers can use the indicated resources to directly run web servers and other gateway software on Virtual Machines hosted on by the XSEDE resource.
Database Hosting: Gateway providers can host scientific databases and similar database management systems such as iRODS on the indicated XSEDE resource.
Virtual Clusters: Gateway providers can run their own dedicated virtual clusters for executing their users jobs; they do not need to go through a shared queue with non-community account users.
Data Collections Storage: This feature provides gateways with large-scale, online and nearline storage for rapid access to community data sets.
Globus Data Services: All XSEDE resources support Globus Transfer, which provides an efficient mechanism for transferring large data sets. The indicated resources in Table 1. (*) also support Globus Sharing services, which allow gateways to map their user accounts to community storage areas.
Support Features per Resource
Almost all XSEDE compute resources provide gateway support features as detailed in Table 1.. Here we profile Bridges, Comet , Jetstream and Wrangler.
- Data collection hosting for gateways
- Database hosting
- Virtual hosting of gateway services
- Web service hosting
Comet provides specialized infrastructure, services, and support for science gateways. Gateway virtual front end servers (1-2 cores, 16GB of memory, 256 GB of disk space) are available to host infrastructure components such as web servers, database servers, data movers, and associated network services (with one public IP address).
Comet has specific policies that help support gateways including:
- Allocation sizes greater than the standard allocation limit of 10 million SUs.
- Restricted job sizes of 1,728 cores or less, which contributes to overall lower wait times - something that gateway users find particularly attractive.
- Support for shared node jobs, which allows for multiple jobs on the same node. This can reduce allocation charges and improve throughput.
- Support for long running jobs, which is important for some applications that do not have built-in restart.
Comet also provides options for a flexible software stack (different from the production cluster) via two approaches:
- Singularity containers that can package system software and libraries, applications, and scientific workflows.
- Virtual cluster technology with near bare metal access to nodes allowing research groups/communities to setup the operating system, application software, and services as needed.
Jetstream, led by the IU Pervasive Technology Institute and TACC, provides virtual machines that may be customized for Gateway use. Gateway developers will want to use the Jetstream API interface that allows IP address reservations. All of the Gateway Support Attributes including: Community Accounts, Data Collection Hosting for Gateways, Virtual Hosting of Gateways, Database Hosting Web Service Hosting are available on Jetstream. Jetstream only provides the framework for all of these services and gateway developers are responsible for building and maintaining these services.
Jetstream has specific policies that help support gateways including:
- Jetstream is the only XSEDE resource that is designed specifically as a cloud platform. It utilizes the core OpenStack projects, so Gateways built on Jetstream will be compatible with other OpenStack clouds.
- Jetstream is planned to be in operation until at least the summer of 2020, meaning that investment in a gateway on Jetstream now means years of use!
- Jetstream has no fixed limit on allocation sizes for community allocations.
- Jetstream offers VM-based hosting of Gateway services and because it is a cloud, it offers dynamic creation of VMs to perform calculations for the Gateway. You can use Docker Swarm, Apache Mesos, Kubernetes or whatever your favorite orchestration tool is to dynamically set up workers to perform calculations on behalf of your gateway.
- Jetstream has pre-built VMs to support many tasks, including pre-built VMs for the XSEDE National Integration Toolkit (XNIT). This means that hundreds of application tools are available for your use as part of a virtual cluster supporting your gateway.
- Database and data collection hosting is possible through creating volumes or leveraging Jetstream object storage.
- Jetstream already hosts a number of web services for gateways. Developers can create instances in multiple regions to further increase availability.
- Static IP addresses can be reserved through the use of OpenStack APIs on Jetstream. OpenStack Horizon is also available as a management tool for capabilities intentionally not available in the Jetstream Portal.
The Wrangler system provides large scale storage, database (MySQL, Postgres/PostGIS, MongoDB), and data management (iRODS) services that may be useful to science gateway providers. These may be integrated with Jetstream VM hosting capabilities to support novel systems; Jetstream and Wrangler hardware are co-located at IU and TACC. Wrangler's capabilities are fully described on the TACC web site and Wrangler User Guide.
|Resource / Feature||Community Accounts||Gateway Server and Web Service Hosting||Database Hosting||Virtual Clusters||Data Collections Storage||Globus Data Services|
|Open Science Grid||X|