Reproducible research and reusable acyclic workflows in Python. Execute code on HPC systems as if you executed them on your personal computer!

Issue description

In the standalone backend on Unix systems, the os.system(command) used here

https://github.com/JoeriHermans/awflow/blob/1fcf255debfbc18d39a6b2baa387bbc85050209d/awflow/backends/standalone/executor.py#L53-L60

actually calls /bin/sh. For some OS, like Ubuntu, sh links to dash which does not support the scripting features required by conda activations. This results in runtime errors like

sh: 5: /home/username/miniconda3/envs/envname/etc/conda/activate.d/activate-binutils_linux-64.sh: Syntax error: "(" unexpected

Proposed solution

A solution would be to change the shell with which the commands are called. This is possible thanks to the subprocess package. A good default would be bash as almost all Unix systems use it.

    if node.tasks > 1:
        for task_index in range(node.tasks):
            task_command = command + ' ' + str(task_index)
            return_code = subprocess.call(task_command, shell=True, executable='/bin/bash')
    else:
        return_code = subprocess.call(command, shell=True, executable='/bin/bash')

One could also add a way to change this default. Additionally, wouldn't it be better to launch the tasks as background jobs for the standalone backend (simply add & at the end of the command) ?

$ pip install awflow==0.0.4
Collecting awflow==0.0.4
  Using cached awflow-0.0.4.tar.gz (19 kB)
    ERROR: Command errored out with exit status 1:
     command: /home/francois/awf/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ou4rxs3q/awflow/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ou4rxs3q/awflow/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-ou4rxs3q/awflow/pip-egg-info
         cwd: /tmp/pip-install-ou4rxs3q/awflow/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-ou4rxs3q/awflow/setup.py", line 54, in <module>
        'examples': _load_requirements('requirements_examples.txt')
      File "/tmp/pip-install-ou4rxs3q/awflow/setup.py", line 17, in _load_requirements
        with open(file_name, 'r') as file:
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements_examples.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Hi,

I tried submitting a few jobs with awflow but somehow each time I run it with slurm backend it never produces a pool.starmap and the process simply times out on cluster. `0 0 8196756 5.1g 85664 S 0.0 1.0 2:12.27 python 790517 rnath 20 0 7953388 5.0g 12020 S 0.0 1.0 0:01.66 python

790518 rnath 20 0 7953388 5.0g 12020 S 0.0 1.0 0:01.45 python

790519 rnath 20 0 7953388 5.0g 12020 S 0.0 1.0 0:01.76 python

790520 rnath 20 0 7953388 5.0g 12020 S 0.0 1.0 0:02.02 python

790521 rnath 20 0 7953388 5.0g 12020 S 0.0 1.0 0:01.99 python `

An example of what happens in the cluster where the processes are spawned but each process uses 0 % of the cpu slurmstepd: error: *** JOB 1933332 ON compute-04 CANCELLED AT 2022-04-08T19:33:26 DUE TO TIME LIMIT ***