Spline is a tool that is capable of running locally as well as part of well known pipelines like Jenkins (Jenkinsfile), Travis CI (.travis.yml) or similar ones.

Overview

Welcome to spline - the pipeline tool

PyPI version PyPI format PyPI versions PyPI license Read The Docs Coverage Status BCH compliance Known Vulnerabilities


Important note:

Since change in my job I didn't had the chance to continue on this project. My main new project is here https://github.com/thomas-lehmann-private/hyperion-task-processor which is planned to be a improved task processing tool; step by step those one will be capable to have more functionality across all platforms (powershell, batch, bash, JShell, Groovy, Kotlin, Docker, ... features depend on platform). It's fully written in Java. Stay tuned.


Table Of Content:
Motivation
Quickstart
Example
Matrix
The Model
The Pipeline
Pipeline Stages
Tasks
The Shell
Environment Variables
Docker Container
Hooks
Include
Event Logging
Tool: spline-loc
How to contact?
Links

Features:

  • Python support for 2.7.x, 3.4.x, 3.5.x, 3.6.x, PyPy and PyPy3
  • automatic schema validation for yaml file
  • matrix based pipeline with tags
  • ordered and parallel pipelines (matrix)
  • ordered and parallel tasks execution
  • pipeline stages (named groups)
  • shell script execution: inline and file
  • environment variables merged across each level: matrix, pipeline, stage, and tasks
  • support for model data (a dictionary of anything you need)
  • cleanup hook
  • filtered execution via tags
  • supporting Jinja templating in Bash scripts (also nested inside model)
  • support for Docker containers and Docker images
  • support for the Packer tool
  • execution time on each level: pipeline, stage, tasks and shell (event logging)
  • documentation here and also at read the docs
  • usable by Jenkinsfile as well as by a .travis.yml (or other pipelines).
  • dry run and debug support
  • support for Python scripts
  • support for task variables
  • support for conditional tasks
  • enabled for code reuse: !include statement

How to contact?

Interesting Links

Comments
  • Recursive render fails on raw/endraw

    Recursive render fails on raw/endraw

    Following problem:

    $ PYTHONPATH=$PWD scripts/spline --definition=examples/docker.yaml --tags=remove-docker-container
    2018-08-16 05:50:04,315 - spline.application - Running with Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
    2018-08-16 05:50:04,321 - spline.application - Running on platform Linux-4.9.0-6-amd64-x86_64-with-debian-9.4
    2018-08-16 05:50:04,321 - spline.application - Current cpu count is 4
    2018-08-16 05:50:04,321 - spline.application - Processing pipeline definition 'examples/docker.yaml'
    2018-08-16 05:50:04,339 - spline.application - Schema validation for 'examples/docker.yaml' succeeded
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Bash', Version(4.4.12)
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Docker', Version(17.12.1)
    2018-08-16 05:50:04,383 - spline.tools.version - Using tool 'Spline', Version(1.11)
    2018-08-16 05:50:04,383 - spline.components.stage - Processing pipeline stage 'example'
    2018-08-16 05:50:04,384 - spline.components.tasks - Processing group of tasks (parallel=no)
    2018-08-16 05:50:04,386 - spline.components.tasks - Processing Bash code: start
    2018-08-16 05:50:04,397 - spline.tools.filters - render(syntax error): unexpected '.'
    2018-08-16 05:50:04,398 - spline.components.tasks - Pipeline has failed: leaving as soon as possible!
    

    The yaml contains {% raw %} and {% endraw %} which seems to be rendered twice and second time the remaining string contains {{.ID}} which cannot be resolved.

    When I remove the recursive replace all is fine.

    bug 
    opened by Nachtfeuer 7
  • Allow renderable `with` content.

    Allow renderable `with` content.

    This patch permits the with option, of tasks, to contain Jinja2 renderable content. This is useful when the models section contains data to be reused across a number of tasks.

    opened by jbenden 5
  • spline-stash: stash functionality (spline-server, part one)

    spline-stash: stash functionality (spline-server, part one)

    On the path to #63 (brainstorming: spline-server) my idea is to start as soon as possible using written functionality which is currently #64 (InMemoryFiles). With this I would like to introduce a new tool spline-stash that can store a path (recursively) at the stash-server and of course I would like to get back the content (usually on another location).

    I still have to make my thoughts about the command line syntax but I'm a bit inspired by git stash of course. The server can run either locally or at a server; the only thing you would have to do is to install spline as usual and to run the server. For the moment no configuration is involved.

    For the server I feel pretty comfortable (the moment) using Falcon (https://falconframework.org). I never used it before except running the given demo.

    enhancement 
    opened by Nachtfeuer 3
  • Conditional tasks

    Conditional tasks

    Considering that the pipeline uploads/deploys something this usually will happen on concrete branches only. As an example in some projects using Git the branch master is used as main line. Each time a pull request gets merged there a successful build should provide artifacts somewhere. The next task snippet could be the way to do it:

    - shell:
        script: echo "upload"
        when: "{{ env.BRANCH_NAME }}" == "master"
    
    enhancement 
    opened by Nachtfeuer 3
  • Asynchronous Bash execution

    Asynchronous Bash execution

    When executing a Bash script the output of it is passed back the Python tool after completion which has two effects:

    • for the time of execution of one Bash you see no output
    • the logging of the lines of the output get wrong timestamps

    It's not really a bug since the code does work.

    enhancement 
    opened by Nachtfeuer 3
  • Recursively expand Jinja2 templates

    Recursively expand Jinja2 templates

    This patch introduces automatic expansion of Jinja2 templates, except the Docker set-up related templates.

    A direct benefit of this patch is one's pipeline configuration becomes less complex; with many Jinja2 render filters used everywhere becoming unnecessary.

    opened by jbenden 2
  • Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Following problem:

    FOO="\${}" spline
    ...
    2018-06-13 15:15:08,595 - spline.components.bash - Running script /tmp/pipeline-script-Kj8mAR.sh
    2018-06-13 15:15:08,612 - spline.components.tasks -  | /tmp/pipeline-script-Kj8mAR.sh: line 24: FOO=${}: bad substitution
    2018-06-13 15:15:08,612 - spline.components.bash - Exit code has been 1
    ...
    

    Simple pipeline demo for that issue (as sufficient):

    pipeline:
      - stage(demo):
          - tasks(ordered):
            - docker(container):
                script: |
                  echo "hello"
    
    bug 
    opened by Nachtfeuer 2
  • Put curly braces and quotes on variables for Docker mounts

    Put curly braces and quotes on variables for Docker mounts

    As I were told it seems that docker run --rm -v $PWD:/mnt/host -it centos:7 bash -c "ls /mnt/host" has failed on OSX while using ${PWD} should have been working. The docker container template is required to be updated for it.

    bug 
    opened by Nachtfeuer 2
  • docker(image) task: failed to add files to image

    docker(image) task: failed to add files to image

    Because final scripts runs on temporary path Dockerfile commands like ADD are not able to fetch content from the source path.

    It turns out that providing the source path wouldn't help because Docker doesn't allow access to path outside the context (relative to the path where the build process is running)

    bug 
    opened by Nachtfeuer 2
  • New task: ansible(simple)

    New task: ansible(simple)

    • embedded solution
    • generator for one environment (inventory, group_vars, playbook)
    • dev, qa, or prod via model
    • script section is playbook
    • no roles
    • updating the presentation
    enhancement 
    opened by Nachtfeuer 2
  • Adding ssh agent support for Docker container

    Adding ssh agent support for Docker container

    • Automatically organize mount and environment variable when ssh agent is active only.
    • Basic procedure: -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent
    enhancement 
    opened by Nachtfeuer 2
  • Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump wheel from 0.29.0 to 0.38.1

    Bump wheel from 0.29.0 to 0.38.1

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    astroid 1.6.6 requires lazy-object-proxy, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- low severity | 441/1000
    Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
    39.0.1 -> 65.5.1
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic


    Learn how to fix vulnerabilities with free interactive lessons:

    🦉 Regular Expression Denial of Service (ReDoS)

    opened by Nachtfeuer 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292150 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292151 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090584 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090586 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090587 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090588 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PYGMENTS-1088505 | Pygments:
    2.5.2 -> 2.7.4
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    🛠 Adjust project settings

    📚 Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
Releases(1.12)
  • 1.12(Oct 26, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    One feature only: labels for docker(container) task.

    Source code(tar.gz)
    Source code(zip)
  • 1.11(Sep 3, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • spline-loc tool (https://spline.readthedocs.io/en/latest/spline_loc.html)
    • bugfix for tool versions using spline on OSX
    • bugfix for escaping values of Bash variables
    • bugfix for closing of file handles (thanks to @jbenden)
    • bugfix for exec permission for user only (thanks to @jbenden)
    • extension to with field being renderable (thanks to @jbenden)
    Source code(tar.gz)
    Source code(zip)
  • 1.10.2(May 7, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • break pipeline when required tools are missing
    • code reuse: !include support for main yaml
    • exclude copying some path variables (HOME, PYTHONPATH, JAVAHOME, PATH, ...)
    • rendering support for task title
    • some minor bugfixes (including quotes and curly braces for $PWD)
    • documentation and presentation extended
    Source code(tar.gz)
    Source code(zip)
  • 1.9(Apr 8, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • new task: ansible(simple)
    • new task: packer
    • bugfix: variables field has not been available in templating (Jinja2) for Docker images
    • bugfix: path adjustment for Dockerfile to be in $PWD so that you can use ADD in Dockerfile for current source path
    • improvement: for ordered task execution enable immediate variable evaluation on next task
    • improvement: new optional field network for Docker container task
    • documentation as presentation updated (as usual)
    Source code(tar.gz)
    Source code(zip)
  • 1.8(Mar 11, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • Bash strict mode (--strict)
    • Temporary scripts path adjustable (--temporary-scripts-path)
    • Forwarding SSH agent into Docker container
    • Fixed exit code (variable) for Python tasks
    • Documentation extended for Python development: http://spline.readthedocs.io/en/latest/development.html
    • Internal: Testing wheel file installation after build
    Source code(tar.gz)
    Source code(zip)
  • 1.7.2(Feb 1, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    Following content:

    • HTML report generation (with auto refresh) Details: http://spline.readthedocs.io/en/latest/report.html
    • tasks variables (writing output of a task into a variable being known py whole pipeline) Details: http://spline.readthedocs.io/en/latest/tasks.html#variables-on-tasks
    • condition tasks (allowing jinja templating and some concrete simple conditions) Details: http://spline.readthedocs.io/en/latest/conditions.html
    • You now can leave out --definition when you use a pipeline.yaml (it's now default)
    • Contribution guide added
    Source code(tar.gz)
    Source code(zip)
  • 1.6.1(Jan 6, 2018)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New option --dry-run which does show all Bash script content instead of executing it
    • New option --debug which adds set -x to Bash script (verbose logging of Bash commands)
    • The tools now does log used cpu count
    • To some extend improved handling for unicode
    • Handling of Python 3.7 added to project (pipeline.yaml and examples/docker-image.yaml). Adding Python 3.7 to .travis.yml probably won't work since I detected issues with pylint and again with unicode. I tested following exact version: https://www.python.org/downloads/release/python-370a2/. Since it is called a "early developer preview" nothing to worry about yet.
    Source code(tar.gz)
    Source code(zip)
  • 1.5(Dec 27, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New feature: Python task; read here: http://spline.readthedocs.io/en/latest/python.html
    • Better Code 10/10
    • Support for Pypy (2.7.x compatible) and Pypy3 (3.5.x compatible)
    • Bandit analyzer added
    • README (with badges) now also on PyPI
    • Code Of Conduct added
    Source code(tar.gz)
    Source code(zip)
  • 1.4(Dec 22, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • code complexity of spline reduced (radon and flake8)
    • coverage now clearly above 90% (roughly 94%)
    • pipeline definition for spline itself added (docker tests will be skipped there)
    • new feature docker(image)
    Source code(tar.gz)
    Source code(zip)
  • 1.3(Dec 20, 2017)

    You can install it with following command:

    pip install spline --upgrade
    
    • added with field for shell and docker container allowing to run same task with different data
    • fixed asynchronous bash execution
    • improved schema by replacing quite nice pykwalify with a bit more Pythonic schema solution being more independent from file format.
    Source code(tar.gz)
    Source code(zip)
  • 1.2(Dec 10, 2017)

    You can install it with following command:

    pip install spline --upgrade
    

    One main task was to have unitests in place. The project is currently at 86% code coverage. Another main task has been to provide a realistic demo on how to build a project with spline + Python + tox.

    Please check https://github.com/Nachtfeuer/pipeline/milestone/3?closed=1 for details. In addition one problem with PIPELINE_BASH_FILE and availability of model and env inside Docker container.

    The documentation of the first example can be seen here:

    • https://github.com/Nachtfeuer/pipeline/blob/master/docs/example.rst
    • or here: http://spline.readthedocs.io/en/latest/example.html
    Source code(tar.gz)
    Source code(zip)
Owner
Thomas Lehmann
Highly interested in development, programming languages, infrastructure, processes and design. Favor C++, Java and Python.
Thomas Lehmann
This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust.

Demo BERT ONNX pipeline written in rust This demo showcase the use of onnxruntime-rs with a GPU on CUDA 11 to run Bert in a data pipeline with Rust. R

Xavier Tao 14 Dec 17, 2022
Memory efficient transducer loss computation

Introduction This project implements the optimization techniques proposed in Improving RNN Transducer Modeling for End-to-End Speech Recognition to re

Fangjun Kuang 51 Nov 25, 2022
Neural Nano-Optics for High-quality Thin Lens Imaging

Neural Nano-Optics for High-quality Thin Lens Imaging Project Page | Paper | Data Ethan Tseng, Shane Colburn, James Whitehead, Luocheng Huang, Seung-H

Ethan Tseng 39 Dec 05, 2022
Official implementation for the paper: "Multi-label Classification with Partial Annotations using Class-aware Selective Loss"

Multi-label Classification with Partial Annotations using Class-aware Selective Loss Paper | Pretrained models Official PyTorch Implementation Emanuel

99 Dec 27, 2022
GPU Accelerated Non-rigid ICP for surface registration

GPU Accelerated Non-rigid ICP for surface registration Introduction Preivous Non-rigid ICP algorithm is usually implemented on CPU, and needs to solve

Haozhe Wu 144 Jan 04, 2023
FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics

FusionNet_Pytorch FusionNet: A deep fully residual convolutional neural network for image segmentation in connectomics Requirements Pytorch 0.1.11 Pyt

Choi Gunho 102 Dec 13, 2022
Rayvens makes it possible for data scientists to access hundreds of data services within Ray with little effort.

Rayvens augments Ray with events. With Rayvens, Ray applications can subscribe to event streams, process and produce events. Rayvens leverages Apache

CodeFlare 32 Dec 25, 2022
Crowd-sourced Annotation of Human Motion.

Motion Annotation Tool Live: https://motion-annotation.humanoids.kit.edu Paper: The KIT Motion-Language Dataset Installation Start by installing all P

Matthias Plappert 4 May 25, 2020
CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing

CapsuleVOS This is the code for the ICCV 2019 paper CapsuleVOS: Semi-Supervised Video Object Segmentation Using Capsule Routing. Arxiv Link: https://a

53 Oct 27, 2022
Pytorch implementations of Bayes By Backprop, MC Dropout, SGLD, the Local Reparametrization Trick, KF-Laplace, SG-HMC and more

Bayesian Neural Networks Pytorch implementations for the following approximate inference methods: Bayes by Backprop Bayes by Backprop + Local Reparame

1.4k Jan 07, 2023
PyTorch implementation(s) of various ResNet models from Twitch streams.

pytorch-resnet-twitch PyTorch implementation(s) of various ResNet models from Twitch streams. Status: ResNet50 currently not working. Will update in n

Daniel Bourke 3 Jan 11, 2022
BASH - Biomechanical Animated Skinned Human

We developed a method animating a statistical 3D human model for biomechanical analysis to increase accessibility for non-experts, like patients, athletes, or designers.

Machine Learning and Data Analytics Lab FAU 66 Nov 19, 2022
A multi-entity Transformer for multi-agent spatiotemporal modeling.

baller2vec This is the repository for the paper: Michael A. Alcorn and Anh Nguyen. baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotempor

Michael A. Alcorn 56 Nov 15, 2022
Code for the Convolutional Vision Transformer (ConViT)

ConViT : Vision Transformers with Convolutional Inductive Biases This repository contains PyTorch code for ConViT. It builds on code from the Data-Eff

Facebook Research 418 Jan 06, 2023
SOLOv2 on onnx & tensorRT

SOLOv2.tensorRT: NOTE: code based on WXinlong/SOLO add support to TensorRT inference onnxruntime tensorRT full_dims and dynamic shape postprocess with

47 Nov 26, 2022
Our solution for SSN Invente 2021's Hackathon

Our solution for SSN Invente 2021's Hackathon. To help maitain godowns in a pristine and safe condition using raspberry pi.

1 Jan 12, 2022
Implementing Vision Transformer (ViT) in PyTorch

Lightning-Hydra-Template A clean and scalable template to kickstart your deep learning project 🚀 ⚡ 🔥 Click on Use this template to initialize new re

2 Dec 24, 2021
Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation, CVPR 2018

Learning Pixel-level Semantic Affinity with Image-level Supervision This code is deprecated. Please see https://github.com/jiwoon-ahn/irn instead. Int

Jiwoon Ahn 337 Dec 15, 2022
Implementation of the "Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos" paper.

Point 4D Transformer Networks for Spatio-Temporal Modeling in Point Cloud Videos Introduction Point cloud videos exhibit irregularities and lack of or

Hehe Fan 101 Dec 29, 2022
Task-related Saliency Network For Few-shot learning

Task-related Saliency Network For Few-shot learning This is an official implementation in Tensorflow of TRSN. Abstract An essential cue of human wisdo

1 Nov 18, 2021