easy_sbatch - Batch submitting Slurm jobs with script templates

Overview

easy_sbatch

easy_sbatch - Batch submitting Slurm jobs via script templates

Install

git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/

Quick start

  1. A simple command (no file or data given):

     $ easy_sbatch 'cat /etc/hostname'
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:50 easy_sbatch.15565347.err
     -rw-r--r-- 1 shenwei cqmu   9 Dec 27 21:50 easy_sbatch.15565347.out
     -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
    
  2. Handling multiple files and specifying a job name:

     $ easy_sbatch 'ls {}' *.fq.gz -J list
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_1.fq.gz
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_2.fq.gz
    
  3. From stdin:

     $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
    
  4. Processing paired-ends FASTQ files:

     $ ls read_1.fq.gz \
         | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
    
     $ cat pe.15565379-read_1.fq.gz.out
     seqtk mergepe read_1.fq.gz read_2.fq.gz
    

Default template

You can choose another one via the option -t/--template.

You can edit the value of an option, e.g., replace $partition with the default partition of your cluster. But note that the corresponding option from the command line will not take effect since no variable name is ready to replace.

$ cat ~/.easy_sbatch/default.slurm 
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error


cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2

$cmd

Usage

. .out]) -e ERROR, --error ERROR file for batch script's standard error (default: . .err]) -v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs https://github.com/shenwei356/easy_sbatch ">
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
                   [-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
                   command [files ...]

easy_sbatch - Batch submitting Slurm jobs with script template

https://github.com/shenwei356/easy_sbatch 

Slurm script template:

  - Default template is ~/.easy_qsub/default.slurm .
  - You can also choose another one via the option -t/--template.
  - You can edit the value of an option, e.g., replace "$partition"
    with the default partition of your cluster. But note that the
    corresponding option from the command line will not take effect
    since no variable name is ready to replace.

Supported replacement strings in command:

  - "{}" for the full path of a input file.
  - "{/}" for dirname.
  - "{%}" for basename.
  - "{^suffix}" for clipping $suffix.
  - "{%^suffix}" for clipping $suffix from basename.

The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.

Examples:

  1. No file or data given:
      $ easy_sbatch 'cat /etc/hostname'

  2. From position arguments:
      $ easy_sbatch 'ls {}' *.fq.gz

  3. From stdin:
      $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'

  4. Slurm script and its output files:
      $ easy_sbatch 'ls' -J hello
        -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:57 hello.15565039.err
        -rw-r--r-- 1 shenwei cqmu  85 Dec 27 20:57 hello.15565039.out
        -rw------- 1 shenwei cqmu 281 Dec 27 20:57 hello.15565039.slurm

  5. Processing paired-ends FASTQ files:
      $ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe

      $ cat pe.15565114.out 
        seqtk mergepe read_1.fq.gz read_2.fq.gz

positional arguments:
  command               command to submit
  files                 input files

optional arguments:
  -h, --help            show this help message and exit
  -lp, --local_p        run commands locally, parallelly
  -ls, --local_s        run commands locally, serially
  -J NAME, --name NAME  job name (default: easy_sbatch)
  -c NCPUS, --ncpus NCPUS
                        number of cpus required per task (default: 24)
  -m MEM, --mem MEM     memory (default: 24gb)
  -p PARTITION, --partition PARTITION
                        partition requested (default: kshdnormal)
  -w WALLTIME, --walltime WALLTIME
                        walltime (default: 30-00:00:00)
  -t TEMPLATE, --template TEMPLATE
                        script template (default: /home/shenwei/.easy_sbatch/default.slurm
  -O SCRIPT, --script SCRIPT
                        output script file
  -o OUTPUT, --output OUTPUT
                        file for batch script's standard output (default: 
      
       .
       
        .out])
  -e ERROR, --error ERROR
                        file for batch script's standard error (default: 
        
         .
         
          .err])
  -v, --verbose         verbosely print information. -vv for just printing command not creating scripts and submitting jobs

https://github.com/shenwei356/easy_sbatch


         
        
       
      

License

MIT License

History

This utility is modified from easy_qsub, which was written for PBS job scheduler.

Owner
Wei Shen
Bioinformatician, postdoc
Wei Shen
Graveyard is an attempt at open-source reimplementation of DraciDoupe.cz

Graveyard: Place for Dead (and Undead) Graveyard is an attempt at open-source reimplementation of DraciDoupe.cz (referred to as DDCZ in this text). De

DraciDoupe.cz 5 Mar 17, 2022
MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework

MuMMI Core is the underlying infrastructure and generalizable component of the MuMMI framework, which facilitates the coordination of massively parallel multiscale simulations.

4 Aug 17, 2022
Manually Install Python 2.7 pip without any problem !

Python2.7_install_pip Manually Install Python 2.7 pip without any problem ! Download installPip.py to your system and Run the code using this Command

Ali Jafari 1 Dec 09, 2021
edgetest is a tox-inspired python library that will loop through your project's dependencies, and check if your project is compatible with the latest version of each dependency

Bleeding edge dependency testing Full Documentation edgetest is a tox-inspired python library that will loop through your project's dependencies, and

Capital One 16 Dec 07, 2022
A collection of repositories used to realise various end-to-end high-level synthesis (HLS) flows centering around the CIRCT project.

circt-hls What is this?: A collection of repositories used to realise various end-to-end high-level synthesis (HLS) flows centering around the CIRCT p

29 Dec 14, 2022
chiarose(XCR) based on chia(XCH) source code fork, open source public chain

chia-rosechain 一个无耻的小活动 | A shameless little event 如果您喜欢这个项目,请点击star 将赠送您520朵玫瑰,可以去 facebook 留下您的(xcr)地址,和github用户名。 If you like this project, please

ddou123 376 Dec 14, 2022
RDFLib is a Python library for working with RDF, a simple yet powerful language for representing information.

RDFLib RDFLib is a pure Python package for working with RDF. RDFLib contains most things you need to work with RDF, including: parsers and serializers

RDFLib 1.8k Jan 02, 2023
Basic code and description for GoBigger challenge 2021.

GoBigger Challenge 2021 en / 中文 Challenge Description 2021.11.13 We are holding a competition —— Go-Bigger: Multi-Agent Decision Intelligence Challeng

OpenDILab 183 Dec 29, 2022
Blender Add-on That Provides Quick Access to Render Controls

Blender Render Buttons Blender Add-on That Provides Quick Access to Render Controls A Blender 3.0 compatablity update of Blender2.8x-RenderButton v0.0

Don Schnitzius 3 Oct 18, 2022
A web-based chat application that enables multiple users to interact with one another

A web-based chat application that enables multiple users to interact with one another, in the same chat room or different ones according to their choosing.

3 Apr 22, 2022
Free APN For Python

Free APN For Python

XENZI GANZZ 4 Apr 22, 2022
An open-source Python project series where beginners can contribute and practice coding.

Python Mini Projects A collection of easy Python small projects to help you improve your programming skills. Table Of Contents Aim Of The Project Cont

Leah Nguyen 491 Jan 04, 2023
Integration of Hotwire's Turbo library with Flask.

turbo-flask Integration of Hotwire's Turbo library with Flask, to allow you to create applications that look and feel like single-page apps without us

Miguel Grinberg 240 Jan 06, 2023
Find out where all films you want to watch are streaming

Just Watch Letterboxd Find out where all films you want to watch are streaming Ever wonder what films you want to watch are already on the streaming p

Jordan Oslislo 2 Feb 04, 2022
A curses based mpd client with basic functionality and album art.

Miniplayer A curses based mpd client with basic functionality and album art. After installation, the player can be opened from the terminal with minip

Tristan Ferrua 102 Dec 24, 2022
The fundamentals of Python!

The fundamentals of Python Author: Mohamed NIANG, Staff ML Scientist Presentation This repository contains notebooks on the fundamentals of Python. Th

Mohamed NIANG 1 Mar 15, 2022
News-app - This is a news web app for reading news from different sources and topics

News-app - This is a news web app for reading news from different sources and topics

1 Feb 02, 2022
Collapse a set of redundant kmers to use IUPAC degenerate bases

kmer-collapse Collapse a set of redundant kmers to use IUPAC degenerate bases Overview Given an input set of kmers, find the smallest set of kmers tha

Alex Reynolds 3 Jan 06, 2022
Open source stenotype engine

Plover Bringing stenography to everyone. Homepage Releases Wiki Blog Google Group Discord Chat About Installation Getting help Contributing Donations

Open Steno Project 2k Jan 09, 2023
Awesome open-source alternatives to SaaS

Awesome-oss-alternatives - Awesome list of open-source startup alternatives to well-known SaaS products

Runa Capital 12.7k Jan 03, 2023