Towards a Leadership-Class Computing Facility - Phase 1 (NSF 17-558) and Potential Collaborative Efforts with XSEDE

This solicitation presents opportunities for a new level of collaboration between XSEDE and NSF "Track 1" resources. The potential exists for Leadership-Class Computing Facility (LCCF) proposers to leverage existing services operated by XSEDE and to engage in collaborative efforts to develop, deliver, and support services that may be unique to the LCCF proposer's platform.

Managing Conflicts of interest and Communications with XSEDE

To avoid conflicts of interest with respect to the proposers and XSEDE staff, we will be providing to all interested proposers a list of items, efforts, activities, etc., that XSEDE is willing to commit to collaborate on with any potential proposer. Proposers will then be able to request a letter of commitment from the XSEDE PI that commits to collaborate on what is included in their proposal (standard commitment form letter) with the understanding that they select only from the menu of options provided.

If there are collaboration options not noted below which you are interested in exploring, XSEDE is willing to consider them, but, to be clear, in order to manage potential conflict of interest issues, XSEDE must offer any option to all potential proposers.

To further manage conflicts of interest, all potential proposers should direct their communications with XSEDE to the XSEDE PI (John Towns, All discussions with XSEDE staff should all be arranged via the XSEDE PI.

The XSEDE Federation and the Service Providers Forum

XSEDE coordinates and integrates the national cyberinfrastructure funded by the NSF while also reaching out to coordinate, integrate, provide support services, and be as inclusive as possible with the broader community. The extended organization created by the amalgamation of the XSEDE program and other organizations with which XSEDE collaborates is referred to as the XSEDE Federation, which includes many autonomous Service Providers. The XSEDE Federation includes those providers and consumers of services which will meet, to varying degrees, the requirements of interfaces with XSEDE and also will engage in the effective and sustained interactions required to develop and evolve those interfaces.

When a resource or service participates in XSEDE, the provider can coordinate and request services from and/or provide them to XSEDE. Communications between XSEDE and Service Providers are managed via the XD Service Providers (SP) Forum, a part of the broader XSEDE Federation including all of XSEDE's partners in various forms. LCCF proposers planning to work with XSEDE should also plan to become a member of the SP Forum. For more information, see Requesting Membership in the XSEDE Federation, the XSEDE Service Provider Software and Services Baseline, and the XSEDE Service Provider Checklist, which include more information on the Service Provider Levels and associated services available via XSEDE. More information regarding these options is outlined below.

Three Levels of Service Providers (SPs) are defined within the XSEDE Federation. Service Providers are classified as being at a specific Level by meeting a minimum set of conditions, described in detail in the Requesting Membership in the XSEDE Federation document. These Levels reflect the degree of coordination/integration between the Service Provider and XSEDE. Level 1 Service Providers are the most tightly coupled with XSEDE. Level 2 and Level 3 Service Providers are more loosely coupled with XSEDE. The XSEDE Software and Services Table for Service Providers document describes the software and services integration expected by XSEDE when participating in the XSEDE Federation as a Service Provider at various Levels.

Depending on their Level of participation, Service Providers commit to integrate with XSEDE in a variety of ways, such as: integration with the XSEDE User Portal; use of the XSEDE Resource Allocation System and XRAC allocations process; XSEDE Information Services; participating in XSEDE working groups on a periodic basis; and verifying their integration annually. The Service Provider Checklist document is updated periodically and reviewed with each Service Provider annually to keep up with the ever evolving XSEDE cyberinfrastructure environment. Once a Service Provider decides or is required to participate with XSEDE, the first contact is with the XSEDE Service Provider Coordinator.

Participating in XSEDE as a Level 1 Service Provider has an estimated start up effort of approximately 320 hours for meetings, software installation and configuration, and integration activities and then an annual maintenance and troubleshooting effort of approximately 160 hours. See the Service Provider Checklist document for the full list of integration components. Participating in XSEDE as a Level 2 Service Provider at a moderate level of integration (installation and integration with many but not all of XSEDE software components and integration activities) has a start up effort of approximately 160 hours and an annual maintenance and troubleshooting effort of approximately 80 hours. See the Service Provider Checklist document for the full list of integration components. Participating in XSEDE as a Level 3 Service Provider requires only filling out a Resource Description Repository (RDR) entry for the Service Provider resource and installation of the Information Services Publishing Framework (IPF) and has an estimated start up effort of 80 hours and an annual maintenance and troubleshooting estimate of 40 hours.

Potential Collaboration Areas

Below XSEDE has provided a set of options for LCCF proposers to consider. These have been organized in a way that we hope is amenable to considering how they might mesh into proposals under development. Again, other areas are possible, and interested potential proposers should discuss this with the XSEDE PI. In each case, we have worked to provide a crisp definition of the collaboration opportunity and any associated costs that proposers would need to budget to support.

Education, Training, Outreach, and Community Engagement

The Education and Student Programs within XSEDE's CEE Workforce Development area provides a continuum of learning resources and services designed to address the needs and requirements of researchers, educators, developers, integrators, and students utilizing advanced digital resources. The Education and Student Programs deliver these services via curricular materials, faculty enhancement, and student engagement. In the context of proposals in response to the LCCF solicitation, the XSEDE Education and Student Programs can:

  1. Provide curricular materials for introducing real HPC in science, math, and computer science classes,
  2. Offer faculty workshops for professional development enabling computational science and HPC fundamentals to be infused into curricula,
  3. Facilitate and support efforts to extend existing educational programs with computational science and HPC, and
  4. Engage students in furthering their HPC skills with authentic internship opportunities that include expert mentorship and access to XSEDE resources.

In support of this effort, LCCF proposers can choose to participate in a number of ways. If LCCF proposers wish to have educational materials developed for a new course and wish XSEDE CEE staff to assist them, 25% of an FTE should be allocated to support this effort. If LCCF proposers wish to coordinate with XSEDE educational staff and participate actively in CEE education activities, 10% of an FTE should be allocated.

The Training group in XSEDE's CEE Workforce Development area delivers training in a variety of formats, including in-person, webinars, multi-site hands-on workshops, and online, asynchronous tutorials. We develop learning assessments in the form of badges, conduct peer reviews of XSEDE training resources, help users to find the appropriate materials for their needs, and coordinate with training leads in other organizations to avoid duplication of effort. In the context of proposals in response to the LCCF solicitation, the XSEDE Training group can:

  1. Offer training to the wider community on a broad array of computing topics, in a variety of formats (which often applies to most high-end compute platforms),
  2. Facilitate workshop listing and registration on the XSEDE portal, when the event is open to the wider community,
  3. Provide use of training accounts, when appropriate,
  4. Facilitate access to an LCCF proposers' training materials by incorporating them into XSEDE's training material listings, and
  5. Develop introductory resource-specific materials.

In support of this effort, LCCF proposers should allocate within their budget 25% of an FTE to serve as the liaison with and participant in XSEDE's training team, ensuring that training activities are well tailored to the proposed LCCF architecture and proposed activities.

The Broadening Participation (and Student Programs) group in XSEDE works to engage underrepresented communities in Science, Technology, Engineering and Mathematics (STEM), particularly focusing on advanced computing skills. The group hosts a variety of programs, including on-site workshops at minority serving institutions, the annual Advanced Computing for Social Change Challenge, outreach efforts to engage underrepresented communities by attending conferences including SACNAS, National HBCU Week, K-16 Educational Justice, Grace Hopper Celebration of Women, CAHSI, Emerging Researchers National Conference, and Understanding Interventions. In the context of proposals in response to the LCCF solicitation, the XSEDE Broadening Participation and Student Programs group can:

  1. Collaborate to offer training workshops at minority serving and teaching institutions,
  2. Facilitate connections to a large group of contacts at minority serving institutions,
  3. Provide best practices developed within XSEDE for effective outreach to underrepresented communities, and
  4. Collaborate on additional Advanced Computing for Social Change programming and similar challenges.

In support of this effort, LCCF proposers should allocate within their budget 25% of an FTE to serve as the liaison with and participant in XSEDE's Broadening Participation group and participate in Broadening Participation and/or Diversity Forum calls where the broadening participation activities are coordinated. This level of support will ensure that the LCCF team is actively engaged in all CEE Broadening Participation activities and will ensure that new activities are developed in collaboration with the LCCF team.

The Campus Champions (CC) program, part of the XSEDE Campus Engagement program, works with research computing facilitators and other CI professionals at institutions nationwide, developing a community of practice among, so far, over 300 professionals at over 200 institutions, in every US state (plus 3 territories). In the context of proposals in response to the LCCF solicitation, the CC leadership will work closely with both the LCCF proposal leadership and the CCs, to:

  1. Help CCs to recognize computing-intensive and data-intensive investigations at their home institutions that would be appropriate for their LCCF proposal;
  2. Train CCs whose institutions have investigations appropriate for the proposed LCCF resource on the basics of using that resource;
  3. Provide such CCs with startup allocations to help their local researchers who have such investigations to get onto the system and port their code to it for testing and benchmarking purposes; and
  4. Help such local researchers to craft compelling resource allocation proposals. In support of this effort, LCCF proposers should allocate within their budget at least 25% of an FTE, to serve as the liaison with, and trainer of, these CCs.

Extended Collaborative Support Services (ECSS)

XSEDE provides in depth (up to 1 year) collaborations with researchers and ECSS consultants, subject to peer review and capacity of the ECSS consultants. ECSS staff have a wide range of skills, from optimizing code to integrating XSEDE resources into science gateways to delivering training and working with new communities to enhance their use of their proposed resources and services. ECSS staff require accounts and training, as early as possible, on all systems where support is offered. ECSS support is available to all researchers with allocations obtained via the XSEDE XRAC process. This would also be true of the LCCF system for any allocations that are made through the XRAC.

For proposal teams wishing to utilize ECSS support for allocations made outside the XRAC process, additional ECSS staff can be conscripted at the cost of $250k/FTE/year. In general, staff devote 25% time per year to an ECSS project. The LCCF team would work with ECSS management on staff assignments.

Resource Allocation Services

XSEDE's Resource Allocation Service (RAS) supports a range of mature and efficient services for members of the SP Forum at each of the three provider levels. Collaboration opportunities for LCCF proposers include managing allocations processes and NSF-approved policies for Startup, Education, and Research projects; accounting processes for tracking usage by users against allocations; and the ability for Service Providers to allow resource access via XSEDE's Single Sign-On Hub. Modifications to existing allocations, accounting, and identity management policies, procedures and capabilities can be explored through defined processes.

The solicitation indicates that 80% of the resource will be allocated via the PRAC process NSF has used with Blue Waters for allocating the bulk of that resource. LCCF proposers can leverage XSEDE-provided services both to manage the allocations made by NSF and support the allocations process and allocations management associated with the remaining 20% of the resource. To enable this, RAS supports the XSEDE Resource Allocation Service (XRAS), a hosted service that can provide allocation request submission, review, and administration capabilities for organizations that manage independent allocation processes. In addition, it can be used to manage allocations made via the PRAC process.

For any allocation made via the XRAC process, use of the XRAS service to support that process will come at no cost to the successful proposer, with the exception of any substantive changes necessary to the current process and procedures to accommodate any unusual needs. In this latter case (anticipated to be unusual), some additional staffing support from the LCCF proposer will be necessary to support those needs in the process.

For LCCF proposers that wish to use the XRAS service to support other allocations processes (tracking of PRAC awarded allocation, allocations process separate from the XRAC to allocate portions of the resource, etc.). LCCF proposers can make use of the XRAS service for their needs at a cost of $10,000 per year (inflated at a rate of 2.75% annually) for basic XRAS support. This is for clients that can work with the existing XRAS implementation and do not have additional features or customization necessary. If there are features that do not currently exist in XRAS, XSEDE staff will need to work with the LCCF awardee to determine the scope of work and develop a separate contract for this development effort.

Infrastructure Services and Integration Support

XSEDE installs, connects, maintains, secures, and evolves an integrated cyberinfrastructure that incorporates a wide range of digital capabilities to support national scientific, engineering, and scholarly research efforts. Infrastructure and enterprise services are provided by XSEDE Operations, which focuses on cybersecurity, networking and data transfer, enterprise services, and providing an operations center for prompt frontline user support and initial issue ticket management.

As noted above in the XSEDE Federation and the Service Providers Forum section, when a Service Provider participates in XSEDE, the Service Provider can coordinate, request, and/or provide resources and/or services from and/or to XSEDE. The resources and/or services can be both traditional or new, novel, and innovative resources and/or services. XSEDE is making available to LCCF proposers services which they may leverage to support their proposed resources and/or services. These include:

XSEDE Operations Center (XOC) and Service Request/Ticket system (RT): XSEDE operates a 24x7 XOC that provides front line user assistance. This provides timely and accurate assistance to the XSEDE community for a wide variety of user issues, and continuously monitors and provides front line troubleshooting for XSEDE user-facing systems and services. XSEDE also operates a service request/ticket system based on RT, where user and staff issues can be routed as service request tickets. Issues can be turned in by tickets via the XSEDE website, via email to, or by calling the XOC and having the operations staff enter the issue into XSEDE's RT. This gives the capability to track issues to resolution and keep all the information about the progress towards issue resolution. XSEDE uses RT, also known as Request Tracker from Best Practical Solutions. An LCCF proposer can leverage this capability to provide frontline/helpdesk services to support all of their users, regardless of how they are allocated, for $200k/year (escalated at 2.75% annually). This includes both staffing support and support for the RT system. If the LCCF awardee opts to allocate 10% or more of their resource via the XRAC (and thus also join the SP Forum), this will be offered at a lower cost of $150k/year (escalated at 2.75% annually). Some Service Providers also use RT locally, and there is an available capability to route tickets between RT systems.

XSEDEnet: XSEDEnet is the private point-to-point network provided by Internet2's Advanced Layer 2 Services (AL2S) platform. This network allows high performance and integrated connectivity between XSEDE Service Providers. Service Providers of all levels participate in XSEDEnet, usually using the Internet2 AL2S that most universities and research centers already have access to with their regional Internet service providers. If the proposers' regional internet service provider already includes Internet2's AL2S as part of their services, this service will likely be free except for the local networking devices that are required to connect to a regional provider. If AL2S is not provided free to the proposer, contact Internet2 to determine the costs associated with obtaining AL2S services from Internet2. domain: XSEDE operates and maintains the domain and any Service Provider can participate in the domain. Usually XSEDE allocated Service Providers have access to {site} to list resources associated with XSEDE in the domain space. The {site} domain can even be delegated to the sites local networking group to maintain the DNS space. This service is available to all Levels of Service Providers. The XSEDE Data Transfer Services group operates, manages and coordinates the domain and can delegate DNS zones as they deem appropriate. Typically, tightly integrated Service Providers (Level 1 Service Providers) are allowed to manage their own DNS zones in the space.

XSEDE Security Infrastructure: Security is an important area to closely coordinate with XSEDE especially for Level 1 and Level 2 allocated Service Providers. Incidents have been minimal in the last seven years, but as Service Providers are tightly integrated with XSEDE, the risk exists that an account compromise at one Service Provider could lead to the spread of the compromise to another XSEDE participating site. XSEDE has multiple authentication services that an SP Forum members can leverage, such as XSEDE's Kerberos, Certificate Authority (providing certificate credentials), CILogon, OAuth, and Duo two factor authentication. Also, XSEDE's cybersecurity team has access to the Qualys security scanning tool paid for by XSEDE for scanning XSEDE resources, which can include resources provided by SP Forum members. This service is used to scan all XSEDE Enterprise Services, wherever they are hosted, and is provided to Service Providers to scan canonical images of XSEDE or PRAC allocated resources. For example, Service Providers with allocations through XRAC, could request that one login node, a DTN, and a publicly accessible compute node is scanned regularly, though XSEDE would not be able to scan all compute nodes or any private hosts. LCCF proposers who wish to take advantage of this as member of the SP Forum should plan to budget 5% FTE of effort on their security team to support these scans. Also, XSEDE Operations Cybersecurity coordinates incident response across XSEDE and its SP Forum Service Provider partners. A security incident becomes an XSEDE security incident when the incident spreads across multiple SP Forum Service Provider resources or if XSEDE enterprise services are somehow impacted. XSEDE security incident coordination is provided free to SP Forum members by XSEDE.

SIngle Single-On Hub: XSEDE's Single Sign On (SSO) login hub,, is a single point-of-entry to the XSEDE cyberinfrastructure. Upon logging into the hub with an XSEDE User Portal (XUP) username, password and Duo two-factor authentication, a 12 hour proxy certificate is automatically generated for a user, allowing the user to access XSEDE resources via GSISSH for the duration of the proxy. GSISSH is further enhanced to have shortcuts to XSEDE compute resources where one can login without the need for the resource-specific username and password. The XSEDE SSO hub accepts standard SSH incoming connections and does not allow use of SSH keys. LCCF proposers can have their users set up to use the XSEDE SSO hub free of charge, as long as the LCCF proposer becomes a member of the SP Forum, those users have XSEDE User Portal accounts, and the LCCF proposer sets up the appropriate capabilities on their resource integrated with XSEDE to facilitate SSO access. This involves approximate 40-80 hours of staff time to work with XSEDE to establish initial integration depending on the XSEDE Service Provider integration level. If the LCCF proposer is an unallocated (via the XRAC) Level 2 or Level 3 SP, there will be one-time, incremental cost of $10,000 to XSEDE that they must bear to integrate with XSEDE and implement this capability.

Coordination meetings: XSEDE has a number of integration and community information meetings. These include the biweekly SP Forum meeting, the XSEDE Campus Champions meeting, the Service Provider Software monthly meetings, the weekly XSEDE Service Provider cybersecurity coordination meetings, the as-needed cybersecurity incident response meetings, the monthly XRAS account management meetings, meetings for participation with XCI (XSEDE Cyberinfrastructure Integration team) in defining and developing the future cyberinfrastructure, and the XSEDEnet participants meeting. Fully integrated Service Providers have approximately 12 hours of meetings a month for participation as a Level 1 allocated resources in XSEDE. Level 2 and Level 3 participants will have lower participation costs per month.

Community driven software requirements and capabilities: XSEDE offers the research community a way to build a shared understanding of driving use cases in areas such as allocations, account management, authentication, authorization, security, remote login, batch computing, data capabilities, and community building, among many other things. This shared understanding is achieved through a transparent and lightweight engineering process with public tools and information resources. Using these processes and tools, users, software and software based service providers, and infrastructure operators can achieve public transparency from driving use cases through production capabilities.

If an LCCF proposer chose to collaborate with XSEDE in this area it could provide its user, staff, and the software provider community with a consistent public understanding of LCCF's important use cases and which ones are shared between the LCCF proposer, XSEDE, and campuses. This would result in the ability to leverage common software and software based services or provide interoperability across these infrastructures. Building this shared understanding of driving use cases would require approximately 1 person-month of LCCF effort in the LCCF personnel budget, spread out over several months. There would be no XSEDE effort that would require support from the LCCF proposer.

Shared or interoperable software and software based services: As noted above, some capabilities are generally well understood and have been listed as explicit offering from XSEDE to LCCF proposers. Building on a shared understanding of the driving use cases, the LCCF awardee and XSEDE could achieve greater interoperability and further enable users to leverage all their available distributed infrastructure, including personal systems, local/campus resources, XSEDE resources, and the LCCF resource. The effort required for LCCF and XSEDE to share software solutions or implement interoperable solutions will depend on those use cases and available implementations and cannot be predicted. Whether an LCCF proposer is interested in sharing software or services with XSEDE, or wishes to be interoperable with XSEDE and campus resources, the proposer should appropriately plan staff time for the level of sharing or interoperability they desire.

Incorporating XSEDE Collaborative Efforts into Proposer Budgets

As noted above, potential conflict of interest issues make this process rather complicated. XSEDE is taking here is to provide opportunities for collaboration while being blind to the specific plans of particular LCCF proposers. This issue has been raised with NSF via both the XSEDE project cognizant program officer and the LCCF solicitation cognizant program officer. LCCF proposers interested in leveraging and collaborating with XSEDE are encouraged to contact the LCCF cognizant program officer to discuss this issue and obtain guidance.

Key Points
Collaborating with XSEDE on NSF 17-558 Towards a Leadership-Class Computing Facility – Phase 1
In order to mitigate any potential conflicts of interest, all potential proposers should direct their XSEDE communications to John Towns, the XSEDE PI
Contact Information