The discussion forums in the XSEDE User Portal are for users to share experiences, questions, and comments with other users and XSEDE staff. Visitors are welcome to browse and search, but you must login to contribute to the forums. While XSEDE staff monitor the lists, XSEDE does not guarantee that questions will be answered. Please note that the forums are not a replacement for formal support or bug reporting procedures through the XSEDE Help Desk. You must be logged in to post to the user forums.

« Back to General Discussion

Non-MPI jobs on comet (ie data parallelized)

Combination View Flat View Tree View
Threads [ Previous | Next ]
Please offer advice, and possibly add to your comet:/share/apps/examples/ on how to run tasks that are parallelized by splitting data, not by using MPI or open-MPI message passing.

The examples at https://portal.xsede.org/sdsc-comet are for MPI or openMPI tasks.
Use of 'ibrun' or openmpi run methods are tricky for data-split 1 cpu tasks. This is what
I end up testing, but don't know if it is making efficient use of the node's 24 cpus, and it apparently won't work for 2+ nodes:

#! /bin/bash
## env prog=blastallc.sh prodb=prot.db protin=prot.aa datad=`pwd` sbatch srun_prog.sh
#SBATCH --job-name="blasta1"
#SBATCH --output="blasta.%j.%N.out"
#SBATCH --partition=compute
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=24
#SBATCH -t 39:55:00

export ncpu=24
if [ "X" = "X$datad" ]; then echo "ERR:datad=what?"; exit -1; fi
if [ "X" = "X$prog" ]; then echo "ERR:prog=what?"; exit -1; fi
cd $datad

## * ibrun is wacky like aprun/ bigred in that it calls $prog ncpu times !!
ibrun --npernode 1 -v $prog

# $prog is a shell script that forks out $ncpu tasks on $ncpu data parts, as in
i=0; while [ $i -lt $ncpu ]; do { bioapp --part $i &; i=$(($i+1)); } done

Data parallelization rather than MPI is the commonest method for bioinformatics and genomics apps, as the problems are data-bound and splitting data into N chunks, running single cpu tasks on each chunk works well. Only a few of the many bio-apps are MPI-aware. For example, comet has a module with an ancient 'mpiblast' which I would not recommend because it likely hasn't been updated to NCBI's current blast code in years. The above method is what I use for NCBI blast, splitting data to ncpu parts.

- Don Gilbert