The discussion forums in the XSEDE User Portal are for users to share experiences, questions, and comments with other users and XSEDE staff. Visitors are welcome to browse and search, but you must login to contribute to the forums. While XSEDE staff monitor the lists, XSEDE does not guarantee that questions will be answered. Please note that the forums are not a replacement for formal support or bug reporting procedures through the XSEDE Help Desk. You must be logged in to post to the user forums.

« Back

how to run MPI + OpenMP batch

Combination View Flat View Tree View
Threads [ Previous | Next ]
how to run MPI + OpenMP batch
openmp mpi
1/26/14 11:49 PM
Without complete docs on mpirun and ibrun, I still am not sure how to submit new jobs on trestle?

I have successfully run jobs under MPI with a script:
#!/bin/tcsh -xv
# qsub xxsede_multiple.csh >&! tp.log_all
#PBS -q normal
#PBS -A TG-PHY130022
#PBS -l nodes=4:ppn=30
#PBS -l walltime=25:00:00
#PBS -o xsede_output
#PBS -N cmi_eeg
mpirun_rsh -np 120 -hostfile $PBS_NODEFILE MV2_ENABLE_AFFINITY=0 {$PWD_WORK}/CMI_EEG/ {$PWD_WORK}/CMI_EEG/tasks

However, now, each of the 120 MPI processes/runs/fits (using my ASA optimization code) ALSO can run independently in parallel under OpenMP. I understand that each ASA run can spawn of up to 16 threads, e.g., 120x16 MPI+OpenMP processses could run together. How do I submit this? What changes need to be made in the above script? Is 512 the maximum that can be spawned at any time; if so, then I likely would need 4 separate aggregate runs of 30x16 each, etc.

Alternatively, I was advised to run batch scripts using ibrun. I assume that this means I would use ibrun INSTEAD of mpirun? If so, how do I pass the same parameters + #PBS settings as I have been using under mpirun?



I do see quite a bit of info on such hybrid runs on
but I would like some definitive answers on the use of my previous mpi script to expand into the hyrbrid runs on Tresles. So far, the info I have received does not seem to jive with the kinds of codes/examples given in the cornell tutorial.

I'd be just as happy to run a top-level MPI-C code that would call ASA-C codes, each ASA-C run set up as now to use OpenMP. I'd just like to get an example that will run on Trestles.

RE: how to run MPI + OpenMP batch
1/31/14 2:08 AM as a reply to Lester Ingber.

I expect that Trestles will be similar to Gordon, but not certain on that:


# 2 MPI processes x 16 threads/node = 2 nodes x 16 cores/node = 32
#PBS -l nodes=2:ppn=1:native
cd /oasis/scratch/$USER/temp_project
mpirun_rsh -np 2 -hostfile $PBS_NODEFILE OMP_NUMTHREADS=16 executable [command line args]

# 8 MPI processes x 4 threads/node = 2 nodes x 16 cores/node = 32
#PBS -l nodes=2:ppn=4:native
cd /oasis/scratch/$USER/temp_project
mpirun_rsh -np 8 -hostfile $PBS_NODEFILE OMP_NUMTHREADS=4 executable [command line args]

RE: how to run MPI + OpenMP batch
1/31/14 2:24 AM as a reply to Jeff Pummill.

Thanks for your reply. I got some concrete answers from Glenn Lockwood in XSEDE Support. He recommends splitting the total jobs into pieces. The final draft I have is attached in three short files {, xxsede_mpi_openmp_master.csh, xxsede_mpi_openmp_template.csh} contained in

Attachments: (1.9k)

RE: how to run MPI + OpenMP batch
1/31/14 3:02 AM as a reply to Lester Ingber.

That's great! Glad you're taken care of. And...thank you for providing information and reference files for others who might have the same question! We're attempting to breathe some life back into the forum, and contributions are very important emoticon

RE: how to run MPI + OpenMP batch
2/3/14 7:33 PM as a reply to Lester Ingber.
I just had a chance to test my new scripts. There are few corrections in the attached zipfile (containing three scripts) from my previous posting.

Attachments: (2.1k)

RE: how to run MPI + OpenMP batch
openmp gordon example scripts trestles
2/5/14 4:48 AM as a reply to Lester Ingber.
We're actually in the process of changing our documentation since the recent system upgrades to both Gordon and Trestles has changed the way hybrid jobs need to be submitted. In addition, Lester's question touches on another type of job which involves bundling a number of non-MPI jobs into a single job submission which our documentation does not cover. For the sake of posterity, I'll just clarify both aspects: running hybrid MPI/OpenMP jobs at SDSC, and bundling non-MPI jobs together.

Hybrid Jobs
We used to advise users to simply adjust their ppn request to reduce the number of MPI tasks run on each node, but that no longer works very well due to some changes we made when Gordon and Trestles were upgraded. To simplify the task of running these MPI+OpenMP jobs, we've implemented the ibrun command which is similar in spirit to TACC's command of the same name.

The point of ibrun is to provide a simple, consistent way for users to launch their MPI jobs without having to worry about all of the different implementation-specific tweaks that may be necessary to get the best performance. To run a job that uses both MPI and OpenMP, you would run your application (called ./my_hybrid_app) by issuing ibrun like this:

1#PBS -l nodes=4:ppn=16:native
2export OMP_NUM_THREADS=16
3ibrun -npernode 1 ./my_hybrid_app

This would launch one MPI process per node (-npernode 1), each with 16 OpenMP threads (OMP_NUM_THREADS=16). Since your job script requested 4 nodes, you would run a total of 4 MPI processes, each with 16 threads, and use a total of 64 cores.

Trestles has four 8-core processors per node, so you may want to run an MPI process on each processor, each with 8 OpenMP threads. The relevant parts of your submit script would then be

1#PBS -l nodes=2:ppn=32
3ibrun -npernode 4 ./my_hybrid_app

ibrun is smart enough to know how many nodes you want to use from your #PBS -l nodes=X:ppn=... request. However you should always request the maximum number of ppn (16 on Gordon, 32 on Trestles).

If your hybrid application uses pthreads instead of OpenMP, you can tell ibrun how many threads per process (tpp) you want to use:

1#PBS -l nodes=2:ppn=32
2ibrun -npernode 4 -tpp 8 ./my_hybrid_app

For hybrid jobs, you should always specify either OMP_NUM_THREADS or -tpp. If you specify neither, ibrun will assume you are using one thread per process. The resulting performance tweaks it applies may not be optimal.

Job Bundling
Since Gordon does not allow users to share compute nodes, people who have serial applications that only use one core are at a disadvantage since they get charged for all 16 cores per node regardless of if their job will only use one of them.

As a workaround, we provide a "job bundling" script that allows you to specify multiple "tasks" in a tasks file, submit a single job as an MPI job, and have those tasks automatically distributed across all of the CPU cores on all of your job's nodes and executed concurrently.

This job bundler can be found in our GitHub user scripts repository. In brief, you create a "tasks" file (literally called 'tasks' if you'd like) and enter a single invocation of your serial application (e.g., "./my_application < my_input1 > my_output1") per line. Then submit the associated submit script which launches the program, and the program interfaces with the MPI environment and distributes all of the commands in your tasks files across all nodes it receives.

Job Bundling with Multithreaded Applications
Lester's question combines the above two ways of running jobs. His individual tasks are multithreaded, but uses MPI. Thus, running bundled, multithreaded applications requires the use of ibrun and because the net effect works like a hybrid MPI+OpenMP job.

The .qsub files we initially provided for used mpirun_rsh and not ibrun, which meant these .qsub files wouldn't work for launching bundled multithreaded tasks. We've just added some more examples (in the bundler/OpenMP directory) that have the correct ibrun usage and illustrate how to bundle multithreaded tasks. Sorry for not providing these scripts earlier!

ibrun can be used for the serial (non-multithreaded) task bundling as well, but we kept mpirun_rsh in the original example scripts to be consistent with our User Guides. As I alluded above, we have a new version of the Gordon and Trestles User Guides in the works that discuss ibrun as I have done here, and it will be consistent with the sample scripts I put in the bundler/OpenMP directory.