easy_sbatch - Batch submitting Slurm jobs with script templates

Overview

easy_sbatch

easy_sbatch - Batch submitting Slurm jobs via script templates

Install

git clone https://github.com/shenwei356/easy_sbatch.git
mkdir -p ~/bin
cp easy_sbatch ~/bin/

Quick start

  1. A simple command (no file or data given):

     $ easy_sbatch 'cat /etc/hostname'
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:50 easy_sbatch.15565347.err
     -rw-r--r-- 1 shenwei cqmu   9 Dec 27 21:50 easy_sbatch.15565347.out
     -rw------- 1 shenwei cqmu 314 Dec 27 21:50 easy_sbatch.15565347.slurm
    
  2. Handling multiple files and specifying a job name:

     $ easy_sbatch 'ls {}' *.fq.gz -J list
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:54 list.15565371-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:54 list.15565371-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565371-read_1.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:55 list.15565373-read_2.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  13 Dec 27 21:55 list.15565373-read_2.fq.gz.out
     -rw------- 1 shenwei cqmu 317 Dec 27 21:54 list.15565373-read_2.fq.gz.slurm
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_1.fq.gz
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:04 read_2.fq.gz
    
  3. From stdin:

     $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'
    
  4. Processing paired-ends FASTQ files:

     $ ls read_1.fq.gz \
         | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe
    
     $ ls
     -rw-r--r-- 1 shenwei cqmu   0 Dec 27 21:56 pe.15565379-read_1.fq.gz.err
     -rw-r--r-- 1 shenwei cqmu  40 Dec 27 21:56 pe.15565379-read_1.fq.gz.out
     -rw------- 1 shenwei cqmu 340 Dec 27 21:56 pe.15565379-read_1.fq.gz.slurm
    
     $ cat pe.15565379-read_1.fq.gz.out
     seqtk mergepe read_1.fq.gz read_2.fq.gz
    

Default template

You can choose another one via the option -t/--template.

You can edit the value of an option, e.g., replace $partition with the default partition of your cluster. But note that the corresponding option from the command line will not take effect since no variable name is ready to replace.

$ cat ~/.easy_sbatch/default.slurm 
#!/bin/bash
#
#SBATCH --partition $partition
#SBATCH --job-name $name
#SBATCH --nodes=1
#SBATCH --cpus-per-task=$ncpus
#SBATCH --mem=$mem
#SBATCH --time=$walltime
#SBATCH --output=$output
#SBATCH --error=$error


cd $$SLURM_SUBMIT_DIR
# echo run on node: $$HOSTNAME >&2

$cmd

Usage

. .out]) -e ERROR, --error ERROR file for batch script's standard error (default: . .err]) -v, --verbose verbosely print information. -vv for just printing command not creating scripts and submitting jobs https://github.com/shenwei356/easy_sbatch ">
usage: easy_sbatch [-h] [-lp | -ls] [-J NAME] [-c NCPUS] [-m MEM] [-p PARTITION] [-w WALLTIME] [-t TEMPLATE]
                   [-O SCRIPT] [-o OUTPUT] [-e ERROR] [-v]
                   command [files ...]

easy_sbatch - Batch submitting Slurm jobs with script template

https://github.com/shenwei356/easy_sbatch 

Slurm script template:

  - Default template is ~/.easy_qsub/default.slurm .
  - You can also choose another one via the option -t/--template.
  - You can edit the value of an option, e.g., replace "$partition"
    with the default partition of your cluster. But note that the
    corresponding option from the command line will not take effect
    since no variable name is ready to replace.

Supported replacement strings in command:

  - "{}" for the full path of a input file.
  - "{/}" for dirname.
  - "{%}" for basename.
  - "{^suffix}" for clipping $suffix.
  - "{%^suffix}" for clipping $suffix from basename.

The generated Slurm scripts are saved in /tmp/easy_qsub-user/ by default,
unless the flag -O/--script is set. If jobs are submitted successfully,
the scripts will be moved to the current directory. If not, they will
be removed.

Examples:

  1. No file or data given:
      $ easy_sbatch 'cat /etc/hostname'

  2. From position arguments:
      $ easy_sbatch 'ls {}' *.fq.gz

  3. From stdin:
      $ ls *.fq.gz | easy_sbatch 'echo {/} {%} {%^.fq.gz}'

  4. Slurm script and its output files:
      $ easy_sbatch 'ls' -J hello
        -rw-r--r-- 1 shenwei cqmu   0 Dec 27 20:57 hello.15565039.err
        -rw-r--r-- 1 shenwei cqmu  85 Dec 27 20:57 hello.15565039.out
        -rw------- 1 shenwei cqmu 281 Dec 27 20:57 hello.15565039.slurm

  5. Processing paired-ends FASTQ files:
      $ ls read_1.fq.gz | easy_sbatch 'echo seqtk mergepe {} {^_1.fq.gz}_2.fq.gz' -J pe

      $ cat pe.15565114.out 
        seqtk mergepe read_1.fq.gz read_2.fq.gz

positional arguments:
  command               command to submit
  files                 input files

optional arguments:
  -h, --help            show this help message and exit
  -lp, --local_p        run commands locally, parallelly
  -ls, --local_s        run commands locally, serially
  -J NAME, --name NAME  job name (default: easy_sbatch)
  -c NCPUS, --ncpus NCPUS
                        number of cpus required per task (default: 24)
  -m MEM, --mem MEM     memory (default: 24gb)
  -p PARTITION, --partition PARTITION
                        partition requested (default: kshdnormal)
  -w WALLTIME, --walltime WALLTIME
                        walltime (default: 30-00:00:00)
  -t TEMPLATE, --template TEMPLATE
                        script template (default: /home/shenwei/.easy_sbatch/default.slurm
  -O SCRIPT, --script SCRIPT
                        output script file
  -o OUTPUT, --output OUTPUT
                        file for batch script's standard output (default: 
      
       .
       
        .out])
  -e ERROR, --error ERROR
                        file for batch script's standard error (default: 
        
         .
         
          .err])
  -v, --verbose         verbosely print information. -vv for just printing command not creating scripts and submitting jobs

https://github.com/shenwei356/easy_sbatch


         
        
       
      

License

MIT License

History

This utility is modified from easy_qsub, which was written for PBS job scheduler.

Owner
Wei Shen
Bioinformatician, postdoc
Wei Shen
Agora-token-helper - Some help tools for AgoraToken

Agora Token Helper Support AgoraToken version 001 - 006. But for security reason

A joke conlang with minimal semantics

SyntaxLang Reserved Defined Words Word Function fo Terminates a noun phrase or verb phrase tu Converts an adjective block or sentence to a noun to Ter

Leo Treloar 1 Dec 07, 2021
Um jogo para treinar COO em python

WAR DUCK Este joguinho bem simples tem como objetivo treinar um pouquinho de POO com python. Não é nada muito complexo mas da pra se divertir Como rod

Gabriel Jospin 3 Sep 19, 2021
It is Keqin Wang first project in CMU, trying to use DRL(PPO) to control a 5-dof manipulator to draw line in space.

5dof-robot-writing this project aim to use PPO control a 5 dof manipulator to draw lines in 3d space. Introduction to the files the pybullet environme

Keqin Wang 4 Aug 22, 2022
Battery conservation Python script for ubuntu to enable battery conservation mode at 60% 80% or 90%

Description Batteryconservation is a small python script wich creates an appindicator for ubuntu which can be used to enable / disable battery conserv

3 Jan 04, 2022
Platform Tree for Xiaomi Redmi Note 7/7S (lavender)

The Xiaomi Redmi Note 7 (codenamed "lavender") is a mid-range smartphone from Xiaomi announced in January 2019. Device specifications Device Xiaomi Re

MUHAMAD KHOIRON 2 Dec 20, 2021
Курс "Искусственный интеллект и машинное обучение"

Искусственный интеллект и машинное обучение О курсе Данный репозиторий содержит в себе сопроводительный учебный материал для курса "Искусственный инте

Dmitry Aladin 19 Dec 04, 2022
Attempt at creating organized collection of little handy snippets of code I'm receiving along the way

ChaosCode Attempt at creating organized collection of little handy snippets of code I'm receiving along the way I always considered coding and program

INFU 4 Nov 26, 2022
A timer for bird lovers, plays a random birdcall while displaying its image and info.

Birdcall Timer A timer for bird lovers. Siriema hatchling by Junior Peres Junior Background My partner needed a customizable timer for sitting and sta

Marcelo Sanches 1 Jul 08, 2022
Painel simples com consulta de cep,CNPJ,placa e ip

Painel mpm Um painel simples com consultas de IP, CNPJ, CEP, PLACA, TELEFONE, CPF e NOME Início 🌐 apt update && apt upgrade -y pkg i python git pip i

8 Feb 27, 2022
A Tool to validate domestic New Zealand vaccine passes

Vaccine Validator Tool to validate domestic New Zealand vaccine passes Create a new virtual environment: python3 -m venv ./venv Activate virtual envi

8 May 01, 2022
a simple functional programming language compiler written in python

Functional Programming Language A compiler for my small functional language. Written in python with SLY lexer/parser generator library. Requirements p

Ashkan Laei 3 Nov 05, 2021
This program generates automatically new folders containing old version of program

Automated Folder Versions Generator by Sergiy Grimoldi - V.0.0.2 This program generates automatically new folders containing old version of something

Sergiy Grimoldi 1 Dec 23, 2021
The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal and to generate far-field speech data using room impulse response data from BUT [email protected] Reverb Database.

Add_noise_and_rir_to_speech The purpose of this code base is to add a specified signal-to-noise ratio noise from MUSAN dataset to a pure speech signal

Yunqi Chen 7 Oct 30, 2022
Nimbus - Open Source Cloud Computing Software - 100% Apache2 licensed

⚠️ The Nimbus infrastructure project is no longer under development. ⚠️ For more information, please read the news announcement. If you are interested

Nimbus 194 Jun 30, 2022
Python script that automates the tasks involved in starting a new coding project

Auto Project Builder Automates the repetitive tasks while starting a new project Installation Use the REQUIREMENTS.txt file to install the dependencie

Prathap S S 1 Feb 03, 2022
A git extension for seeing your Cloud Build deployment

A git extension for seeing your Cloud Build deployment

Katie McLaughlin 13 May 10, 2022
Your E-Canteen that is convenient and accessible wherever you are in the campus

Food Web E-Canteen System Your E-Canteen that is convenient and accessible wherever you are in the campus. Table of Contents About The Project Contrib

Pudding 5 Jan 07, 2023
Terrible sudoku solver with spaghetti code and performance issues

SudokuSolver Terrible sudoku solver with spaghetti code and performance issues - if it's unable to figure out next step it will stop working, it never

Kamil Bizoń 1 Dec 05, 2021
Vita Specific Patches and Application for Doki Doki Literature Club (Steam Version) using Ren'Py PSVita

Doki-Doki-Literature-Club-Vita Vita Specific Patches and Application for Doki Doki Literature Club (Steam Version) using Ren'Py PSVita Contains: Modif

Jaylon Gowie 25 Dec 30, 2022