XSEDE ALLOCATION REQUESTS Open Submission, Guidelines, Resource and Policy Changes

Posted by Ken Hackworth on 03/16/2017 14:23 UTC

XSEDE is now accepting Research Allocation Requests for the allocation period, July 1, 2017 to June 30, 2018. The submission period is from March 15, 2017 thru April 15, 2017. Please review the new XSEDE systems and important policy changes (see below) before you submit your allocation request through the XSEDE User Portal
A recent change to the submission of proposals is that the Allocations proposal submission system(XRAS) will force submissions to adhere to the uploaded document page limits, these can be found at

NEW XSEDE Resources:
See the Resource Catalog for a list of XSEDE compute, visualization and storage resources, and more details on the new systems (

  • The Texas Advanced Computing Center(TACC) introduces their new resource Stampede 2. Stampede 2 will enter full production in the Fall 2017 as the 18 petaflop national resource that builds on the successes of the original Stampede system it replaces. The first phase of the Stampede 2 rollout features the second generation of processors based on Intel’s Many Integrated Core (MIC) architecture. These 4,200 Knights Landing (KNL) nodes represent a radical break with the first generation Knights Corner (KNC) MIC coprocessor. Unlike the legacy KNC, a Stampede KNL is not a coprocessor: each 68-core KNL is a stand-alone, self-booting processor that is the sole processor in its node. Phase 2 will add approximately 50% of additional compute power to the system as a whole by introducing new nodes equipped with a future Intel processor. When fully deployed, Stampede 2 will deliver twice the performance of the original Stampede system. Please note that Stampede 2 is allocated in service units (SU)s. An SU is defined as 1 wall-clock node hour not core hours!

Starting this submission period both the Pittsburgh Supercomputing Center(PSC) and the San Diego Supercomputer Center(SDSC) will be allocating their GPU compute resources separately from their standard compute nodes, please see details below:

  • SDSC’s Comet GPU has 36 general purpose GPU nodes, with 2 Tesla K80 GPU graphics cards per node, each with 2 GK210 GPUs (144 GPUs in total). Each GPU node also features 2 Intel Haswell processors of the same design and performance as the standard compute nodes (described separately). The GPU nodes are integrated into the Comet resource and available through the SLURM scheduler for either dedicated or shared node jobs (i.e., a user can run on 1 or more GPUs/node and will be charged accordingly). Like the Comet standard compute nodes, the GPU nodes feature a local SSD which can be specified as a scratch resource during job execution – in many cases using SSD’s can alleviate I/O bottlenecks associated with using the shared Lustre parallel file system.
    • Comet’s GPUs are a specialized resource that performs well for certain classes of algorithms and applications. There is a large and growing base of community codes that have been optimized for GPUs including those in molecular dynamics, and machine learning. GPU-enabled applications on Comet include: Amber, Gromacs, BEAST, OpenMM, TensorFlow, and NAMD.
  • PSC introduces Bridges GPU, a newly allocatable resource within Bridges that features 32 NVIDIA Tesla K80 GPUs and 64 NVIDIA Tesla P100 GPUs. Bridges GPU complements Bridges’ Regular, Bridges Large, and its Pylon storage system to accelerate deep learning and a wide variety of application workloads. The 16 GPU nodes, each with 2 NVIDIA Tesla K80 GPU cards, 2 Intel Xeon CPUs (14 cores each), and 128GB of RAM and 32 GPU nodes, each with 2 NVIDIA Tesla P100 GPU cards, 2 Intel Xeon CPUs (16 cores each), and 128GB of RAM.
    • The PSC’s Bridges is a uniquely capable resource for empowering new research communities and bringing together HPC and Big Data. Bridges integrates a uniquely flexible, user-focused, data-centric software environment with very large shared memory, a high-performance interconnect, and rich file systems to empower new research communities, bring desktop convenience to HPC and drive complex workflows.
    • Bridges supports new communities through extensive interactivity, gateways, persistent databases and web servers, high productivity programming languages, and virtualization. The software environment is extremely robust, supporting enabling capabilities such as Python, R, and MATLAB on large-memory nodes, genome sequence assembly on nodes with up to 12TB of RAM, machine learning and especially deep learning, Spark and Hadoop, complex workflows, and web architectures to support gateways.

Storage Allocations: Continuing this submission period, access to XSEDE storage resources along with compute resources will need to be requested and justified, both in the XRAS application and the body of the proposal’s main document. The following XSEDE sites will be offering allocatable storage facilities, these are:

    • SDSC (Data Oasis)
    • TACC (Ranch)
    • TACC (Wrangler storage)
    • PSC (Pylon)
    • IU-TACC (Jetstream)

Storage needs have always been part of allocation requests, however, XSEDE will be enforcing the storage awards in unison with the storage sites. Please see (

Estimated Available Service Units/GB for upcoming meeting:
Indiana University/TACC (Jetstream) 5,000,000
LSU (SuperMIC) 6,500,000
Open Science Grid (OSG) 2,000,000
PSC Bridges(Regular Memory) 38,000,000
PSC Bridges (Large Memory) 700,000
PSC Bridges (Bridges GPU) TBD
PSC Persistent disk storage (Pylon) 2,000,000
SDSC Dell Cluster with Intel Haswell Processors (Comet) 80,000,00
SDSC Dell Cluster with Intel Haswell Processors (Comet GPU) TBD
SDSC Medium-term disk storage (Data Oasis) 300,000
Stanford Cray CS-Storm GPU Supercomputer(XStream) 500,000
TACC HP/NVIDIA Interactive Visualization and Data Analytics System (Maverick) 4,000,000
TACC Dell/Intel Knight’s Landing System (Stampede2 – Phase 1) 10,000,000 node hours
TACC Data Analytics System (Wrangler) 180,000 node hours
TACC Long-term Storage (Wrangler Storage) 500,000
TACC Long-term tape Archival Storage (Ranch) 2,000,000

Allocation Request Procedures:

  • In the past code performance and scaling was to be a section addressed in all research requests main document, this section seems to have been overlooked by many PIs in the recent quarterly research submission periods which has led to severe reductions or even complete rejection of both new and renewal requests. Continuing this quarterly submission period it will be mandatory to upload a scaling and code performance document detailing your code efficiency. Please see section 7.2 Review Criteria, of the Allocations Policy document (
  • Also, it has become mandatory to discuss/detail, in the main document, the disclosure of access to other cyberinfrastructure resources(e.g. NSF Blue Waters, DOE INCITE resources, local campus, …) should be detailed in the main document. Please see section 7.3 Review Criteria, of the Allocations Policy document( The failure to disclose access to these resources could lead to severe reductions or even complete rejection of both new and renewal requests. If there is no access to other cyberinfrastructure resources this should be made clear as well.
  • The XRAC review panel has asked that the PIs include the following: "The description of the computational methods must include explicit specification of the integration time step value, if relevant (e.g. Molecular Dynamics Simulations). If these details are not provided a 1 femtosecond (1fs) will be assumed with this information being used accordingly to evaluate the proposed computations."
  • All funding used to support the Research Plan of an XRAC Research Request must be reported in the Supporting Grants form in the XRAS submission. Reviewers use this information to assess whether the PI has enough support to accomplish the Research Plan, analyze data, prepare publications, etc.
  • Publications that have resulted from the use of XSEDE resources should be entered into your XSEDE portal profile which you will be able to attach to your Research submission.
  • Also note that it is expected that the scaling and code performance information is from the resource(s) being requested in the research request.

Policy Changes: Allocations Policy document(

  • Storage allocation requests for Archival Storage in conjunction with compute and visualization resources and/or Stand Alone Storage need to be requested explicitly both in your proposal (research proposals) and also in the resource section of XRAS.
  • Furthermore, the PI must describe the peer-reviewed science goal that the resource award will facilitate. These goals must match or be sub-goals of those described in the listed funding award for that year.
  • After the Panel Discussion of the XRAC meeting, the total Recommended Allocation is determined and compared to the total Available Allocation across all resources. Transfers of allocations may be made for projects that are more suitable for execution on other resources; transfers may also be made for projects that can take advantage of other resources, hence balancing the load. When the total Recommended considerably exceeds Available Allocations a reconciliation process adjusts all Recommended Allocations to remove oversubscription. This adjustment process reduces large allocations more than small ones and gives preference to NSF-funded projects or project portions. Under the direction of NSF, additional adjustments may be made to achieve a balanced portfolio of awards to diverse communities, geographic areas, and scientific domains.
  • Conflict of Interest (COI) policy will be strictly enforced for large proposals. For small requests, the PI/reviewer may participate in the respective meeting, but leave the room during the discussion of their proposal.
  • XRAC proposals for allocations request resources that represent a significant investment of the National Science Foundation. The XRAC review process therefore strives to be as rigorous as for equivalent NSF proposals.
  • The actual availability of resources is not considered in the review. Only the merit of the proposal is. Necessary reductions due to insufficient resources will be made after the merit review, under NSF guidelines, as described in Section 6.4.1.
  • 10% max advance on all research requests, as described in Section 3.5.4

Examples of well-written proposals:
For more information about writing a successful research proposal as well as examples of successful research allocation requests please see: (

If you would like to discuss your plans for submitting a research request please send email to the XSEDE Help Desk at Your questions will be forwarded to the appropriate XSEDE Staff for their assistance.

Ken Hackworth
XSEDE Resource Allocations Coordinator