Spline is a tool that is capable of running locally as well as part of well known pipelines like Jenkins (Jenkinsfile), Travis CI (.travis.yml) or similar ones.

Overview

Welcome to spline - the pipeline tool

PyPI version PyPI format PyPI versions PyPI license Read The Docs Coverage Status BCH compliance Known Vulnerabilities


Important note:

Since change in my job I didn't had the chance to continue on this project. My main new project is here https://github.com/thomas-lehmann-private/hyperion-task-processor which is planned to be a improved task processing tool; step by step those one will be capable to have more functionality across all platforms (powershell, batch, bash, JShell, Groovy, Kotlin, Docker, ... features depend on platform). It's fully written in Java. Stay tuned.


Table Of Content:
Motivation
Quickstart
Example
Matrix
The Model
The Pipeline
Pipeline Stages
Tasks
The Shell
Environment Variables
Docker Container
Hooks
Include
Event Logging
Tool: spline-loc
How to contact?
Links

Features:

  • Python support for 2.7.x, 3.4.x, 3.5.x, 3.6.x, PyPy and PyPy3
  • automatic schema validation for yaml file
  • matrix based pipeline with tags
  • ordered and parallel pipelines (matrix)
  • ordered and parallel tasks execution
  • pipeline stages (named groups)
  • shell script execution: inline and file
  • environment variables merged across each level: matrix, pipeline, stage, and tasks
  • support for model data (a dictionary of anything you need)
  • cleanup hook
  • filtered execution via tags
  • supporting Jinja templating in Bash scripts (also nested inside model)
  • support for Docker containers and Docker images
  • support for the Packer tool
  • execution time on each level: pipeline, stage, tasks and shell (event logging)
  • documentation here and also at read the docs
  • usable by Jenkinsfile as well as by a .travis.yml (or other pipelines).
  • dry run and debug support
  • support for Python scripts
  • support for task variables
  • support for conditional tasks
  • enabled for code reuse: !include statement

How to contact?

Interesting Links

Comments
  • Recursive render fails on raw/endraw

    Recursive render fails on raw/endraw

    Following problem:

    $ PYTHONPATH=$PWD scripts/spline --definition=examples/docker.yaml --tags=remove-docker-container
    2018-08-16 05:50:04,315 - spline.application - Running with Python 2.7.13 (default, Nov 24 2017, 17:33:09) [GCC 6.3.0 20170516]
    2018-08-16 05:50:04,321 - spline.application - Running on platform Linux-4.9.0-6-amd64-x86_64-with-debian-9.4
    2018-08-16 05:50:04,321 - spline.application - Current cpu count is 4
    2018-08-16 05:50:04,321 - spline.application - Processing pipeline definition 'examples/docker.yaml'
    2018-08-16 05:50:04,339 - spline.application - Schema validation for 'examples/docker.yaml' succeeded
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Bash', Version(4.4.12)
    2018-08-16 05:50:04,382 - spline.tools.version - Using tool 'Docker', Version(17.12.1)
    2018-08-16 05:50:04,383 - spline.tools.version - Using tool 'Spline', Version(1.11)
    2018-08-16 05:50:04,383 - spline.components.stage - Processing pipeline stage 'example'
    2018-08-16 05:50:04,384 - spline.components.tasks - Processing group of tasks (parallel=no)
    2018-08-16 05:50:04,386 - spline.components.tasks - Processing Bash code: start
    2018-08-16 05:50:04,397 - spline.tools.filters - render(syntax error): unexpected '.'
    2018-08-16 05:50:04,398 - spline.components.tasks - Pipeline has failed: leaving as soon as possible!
    

    The yaml contains {% raw %} and {% endraw %} which seems to be rendered twice and second time the remaining string contains {{.ID}} which cannot be resolved.

    When I remove the recursive replace all is fine.

    bug 
    opened by Nachtfeuer 7
  • Allow renderable `with` content.

    Allow renderable `with` content.

    This patch permits the with option, of tasks, to contain Jinja2 renderable content. This is useful when the models section contains data to be reused across a number of tasks.

    opened by jbenden 5
  • spline-stash: stash functionality (spline-server, part one)

    spline-stash: stash functionality (spline-server, part one)

    On the path to #63 (brainstorming: spline-server) my idea is to start as soon as possible using written functionality which is currently #64 (InMemoryFiles). With this I would like to introduce a new tool spline-stash that can store a path (recursively) at the stash-server and of course I would like to get back the content (usually on another location).

    I still have to make my thoughts about the command line syntax but I'm a bit inspired by git stash of course. The server can run either locally or at a server; the only thing you would have to do is to install spline as usual and to run the server. For the moment no configuration is involved.

    For the server I feel pretty comfortable (the moment) using Falcon (https://falconframework.org). I never used it before except running the given demo.

    enhancement 
    opened by Nachtfeuer 3
  • Conditional tasks

    Conditional tasks

    Considering that the pipeline uploads/deploys something this usually will happen on concrete branches only. As an example in some projects using Git the branch master is used as main line. Each time a pull request gets merged there a successful build should provide artifacts somewhere. The next task snippet could be the way to do it:

    - shell:
        script: echo "upload"
        when: "{{ env.BRANCH_NAME }}" == "master"
    
    enhancement 
    opened by Nachtfeuer 3
  • Asynchronous Bash execution

    Asynchronous Bash execution

    When executing a Bash script the output of it is passed back the Python tool after completion which has two effects:

    • for the time of execution of one Bash you see no output
    • the logging of the lines of the output get wrong timestamps

    It's not really a bug since the code does work.

    enhancement 
    opened by Nachtfeuer 3
  • Recursively expand Jinja2 templates

    Recursively expand Jinja2 templates

    This patch introduces automatic expansion of Jinja2 templates, except the Docker set-up related templates.

    A direct benefit of this patch is one's pipeline configuration becomes less complex; with many Jinja2 render filters used everywhere becoming unnecessary.

    opened by jbenden 2
  • Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Require usage of single quotes for docker -e parameter (otherwise subsitutions do happen)

    Following problem:

    FOO="\${}" spline
    ...
    2018-06-13 15:15:08,595 - spline.components.bash - Running script /tmp/pipeline-script-Kj8mAR.sh
    2018-06-13 15:15:08,612 - spline.components.tasks -  | /tmp/pipeline-script-Kj8mAR.sh: line 24: FOO=${}: bad substitution
    2018-06-13 15:15:08,612 - spline.components.bash - Exit code has been 1
    ...
    

    Simple pipeline demo for that issue (as sufficient):

    pipeline:
      - stage(demo):
          - tasks(ordered):
            - docker(container):
                script: |
                  echo "hello"
    
    bug 
    opened by Nachtfeuer 2
  • Put curly braces and quotes on variables for Docker mounts

    Put curly braces and quotes on variables for Docker mounts

    As I were told it seems that docker run --rm -v $PWD:/mnt/host -it centos:7 bash -c "ls /mnt/host" has failed on OSX while using ${PWD} should have been working. The docker container template is required to be updated for it.

    bug 
    opened by Nachtfeuer 2
  • docker(image) task: failed to add files to image

    docker(image) task: failed to add files to image

    Because final scripts runs on temporary path Dockerfile commands like ADD are not able to fetch content from the source path.

    It turns out that providing the source path wouldn't help because Docker doesn't allow access to path outside the context (relative to the path where the build process is running)

    bug 
    opened by Nachtfeuer 2
  • New task: ansible(simple)

    New task: ansible(simple)

    • embedded solution
    • generator for one environment (inventory, group_vars, playbook)
    • dev, qa, or prod via model
    • script section is playbook
    • no roles
    • updating the presentation
    enhancement 
    opened by Nachtfeuer 2
  • Adding ssh agent support for Docker container

    Adding ssh agent support for Docker container

    • Automatically organize mount and environment variable when ssh agent is active only.
    • Basic procedure: -v $SSH_AUTH_SOCK:/ssh-agent --env SSH_AUTH_SOCK=/ssh-agent
    enhancement 
    opened by Nachtfeuer 2
  • Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bump wheel from 0.29.0 to 0.38.1 in /examples/python/primes

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump wheel from 0.29.0 to 0.38.1

    Bump wheel from 0.29.0 to 0.38.1

    Bumps wheel from 0.29.0 to 0.38.1.

    Changelog

    Sourced from wheel's changelog.

    Release Notes

    UNRELEASED

    • Updated vendored packaging to 22.0

    0.38.4 (2022-11-09)

    • Fixed PKG-INFO conversion in bdist_wheel mangling UTF-8 header values in METADATA (PR by Anderson Bravalheri)

    0.38.3 (2022-11-08)

    • Fixed install failure when used with --no-binary, reported on Ubuntu 20.04, by removing setup_requires from setup.cfg

    0.38.2 (2022-11-05)

    • Fixed regression introduced in v0.38.1 which broke parsing of wheel file names with multiple platform tags

    0.38.1 (2022-11-04)

    • Removed install dependency on setuptools
    • The future-proof fix in 0.36.0 for converting PyPy's SOABI into a abi tag was faulty. Fixed so that future changes in the SOABI will not change the tag.

    0.38.0 (2022-10-21)

    • Dropped support for Python < 3.7
    • Updated vendored packaging to 21.3
    • Replaced all uses of distutils with setuptools
    • The handling of license_files (including glob patterns and default values) is now delegated to setuptools>=57.0.0 (#466). The package dependencies were updated to reflect this change.
    • Fixed potential DoS attack via the WHEEL_INFO_RE regular expression
    • Fixed ValueError: ZIP does not support timestamps before 1980 when using SOURCE_DATE_EPOCH=0 or when on-disk timestamps are earlier than 1980-01-01. Such timestamps are now changed to the minimum value before packaging.

    0.37.1 (2021-12-22)

    • Fixed wheel pack duplicating the WHEEL contents when the build number has changed (#415)
    • Fixed parsing of file names containing commas in RECORD (PR by Hood Chatham)

    0.37.0 (2021-08-09)

    • Added official Python 3.10 support
    • Updated vendored packaging library to v20.9

    ... (truncated)

    Commits
    • 6f1608d Created a new release
    • cf8f5ef Moved news item from PR #484 to its proper place
    • 9ec2016 Removed install dependency on setuptools (#483)
    • 747e1f6 Fixed PyPy SOABI parsing (#484)
    • 7627548 [pre-commit.ci] pre-commit autoupdate (#480)
    • 7b9e8e1 Test on Python 3.11 final
    • a04dfef Updated the pypi-publish action
    • 94bb62c Fixed docs not building due to code style changes
    • d635664 Updated the codecov action to the latest version
    • fcb94cd Updated version to match the release
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    [Snyk] Security upgrade setuptools from 39.0.1 to 65.5.1

    This PR was automatically created by Snyk using the credentials of a real user.


    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    astroid 1.6.6 requires lazy-object-proxy, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- low severity | 441/1000
    Why? Recently disclosed, Has a fix available, CVSS 3.1 | Regular Expression Denial of Service (ReDoS)
    SNYK-PYTHON-SETUPTOOLS-3113904 | setuptools:
    39.0.1 -> 65.5.1
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the affected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic


    Learn how to fix vulnerabilities with free interactive lessons:

    πŸ¦‰ Regular Expression Denial of Service (ReDoS)

    opened by Nachtfeuer 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292150 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Out-of-bounds Read
    SNYK-PYTHON-PILLOW-1292151 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    [Snyk] Security upgrade Pillow from 6.2.2 to 8.2.0

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090584 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090586 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090587 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PILLOW-1090588 | Pillow:
    6.2.2 -> 8.2.0
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
  • [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    [Snyk] Security upgrade Pygments from 2.5.2 to 2.7.4

    Snyk has created this PR to fix one or more vulnerable packages in the `pip` dependencies of this project.

    Changes included in this PR

    • Changes to the following files to upgrade the vulnerable dependencies to a fixed version:
      • requirements.txt
    ⚠️ Warning
    sphinxcontrib-inheritance 0.9.0 has requirement Sphinx<=1.4.9, but you have Sphinx 1.6.5.
    Sphinx 1.6.5 requires Pygments, which is not installed.
    
    

    Vulnerabilities that will be fixed

    By pinning:

    Severity | Priority Score (*) | Issue | Upgrade | Breaking Change | Exploit Maturity :-------------------------:|-------------------------|:-------------------------|:-------------------------|:-------------------------|:------------------------- high severity | 661/1000
    Why? Recently disclosed, Has a fix available, CVSS 7.5 | Denial of Service (DoS)
    SNYK-PYTHON-PYGMENTS-1088505 | Pygments:
    2.5.2 -> 2.7.4
    | No | No Known Exploit

    (*) Note that the real score may have changed since the PR was raised.

    Some vulnerabilities couldn't be fully fixed and so Snyk will still find them when the project is tested again. This may be because the vulnerability existed within more than one direct dependency, but not all of the effected dependencies could be upgraded.

    Check the changes in this PR to ensure they won't cause issues with your project.


    Note: You are seeing this because you or someone else with access to this repository has authorized Snyk to open fix PRs.

    For more information: 🧐 View latest project report

    πŸ›  Adjust project settings

    πŸ“š Read more about Snyk's upgrade and patch logic

    opened by snyk-bot 0
Releases(1.12)
  • 1.12(Oct 26, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    One feature only: labels for docker(container) task.

    Source code(tar.gz)
    Source code(zip)
  • 1.11(Sep 3, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • spline-loc tool (https://spline.readthedocs.io/en/latest/spline_loc.html)
    • bugfix for tool versions using spline on OSX
    • bugfix for escaping values of Bash variables
    • bugfix for closing of file handles (thanks to @jbenden)
    • bugfix for exec permission for user only (thanks to @jbenden)
    • extension to with field being renderable (thanks to @jbenden)
    Source code(tar.gz)
    Source code(zip)
  • 1.10.2(May 7, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • break pipeline when required tools are missing
    • code reuse: !include support for main yaml
    • exclude copying some path variables (HOME, PYTHONPATH, JAVAHOME, PATH, ...)
    • rendering support for task title
    • some minor bugfixes (including quotes and curly braces for $PWD)
    • documentation and presentation extended
    Source code(tar.gz)
    Source code(zip)
  • 1.9(Apr 8, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • new task: ansible(simple)
    • new task: packer
    • bugfix: variables field has not been available in templating (Jinja2) for Docker images
    • bugfix: path adjustment for Dockerfile to be in $PWD so that you can use ADD in Dockerfile for current source path
    • improvement: for ordered task execution enable immediate variable evaluation on next task
    • improvement: new optional field network for Docker container task
    • documentation as presentation updated (as usual)
    Source code(tar.gz)
    Source code(zip)
  • 1.8(Mar 11, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    
    • Bash strict mode (--strict)
    • Temporary scripts path adjustable (--temporary-scripts-path)
    • Forwarding SSH agent into Docker container
    • Fixed exit code (variable) for Python tasks
    • Documentation extended for Python development: http://spline.readthedocs.io/en/latest/development.html
    • Internal: Testing wheel file installation after build
    Source code(tar.gz)
    Source code(zip)
  • 1.7.2(Feb 1, 2018)

    You can install it with following command (for upgrade please add --upgrade):

    pip install spline
    

    Following content:

    • HTML report generation (with auto refresh) Details: http://spline.readthedocs.io/en/latest/report.html
    • tasks variables (writing output of a task into a variable being known py whole pipeline) Details: http://spline.readthedocs.io/en/latest/tasks.html#variables-on-tasks
    • condition tasks (allowing jinja templating and some concrete simple conditions) Details: http://spline.readthedocs.io/en/latest/conditions.html
    • You now can leave out --definition when you use a pipeline.yaml (it's now default)
    • Contribution guide added
    Source code(tar.gz)
    Source code(zip)
  • 1.6.1(Jan 6, 2018)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New option --dry-run which does show all Bash script content instead of executing it
    • New option --debug which adds set -x to Bash script (verbose logging of Bash commands)
    • The tools now does log used cpu count
    • To some extend improved handling for unicode
    • Handling of Python 3.7 added to project (pipeline.yaml and examples/docker-image.yaml). Adding Python 3.7 to .travis.yml probably won't work since I detected issues with pylint and again with unicode. I tested following exact version: https://www.python.org/downloads/release/python-370a2/. Since it is called a "early developer preview" nothing to worry about yet.
    Source code(tar.gz)
    Source code(zip)
  • 1.5(Dec 27, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • New feature: Python task; read here: http://spline.readthedocs.io/en/latest/python.html
    • Better Code 10/10
    • Support for Pypy (2.7.x compatible) and Pypy3 (3.5.x compatible)
    • Bandit analyzer added
    • README (with badges) now also on PyPI
    • Code Of Conduct added
    Source code(tar.gz)
    Source code(zip)
  • 1.4(Dec 22, 2017)

    You can install/upgrade it with following command:

    pip install spline [--upgrade]
    

    Following content:

    • code complexity of spline reduced (radon and flake8)
    • coverage now clearly above 90% (roughly 94%)
    • pipeline definition for spline itself added (docker tests will be skipped there)
    • new feature docker(image)
    Source code(tar.gz)
    Source code(zip)
  • 1.3(Dec 20, 2017)

    You can install it with following command:

    pip install spline --upgrade
    
    • added with field for shell and docker container allowing to run same task with different data
    • fixed asynchronous bash execution
    • improved schema by replacing quite nice pykwalify with a bit more Pythonic schema solution being more independent from file format.
    Source code(tar.gz)
    Source code(zip)
  • 1.2(Dec 10, 2017)

    You can install it with following command:

    pip install spline --upgrade
    

    One main task was to have unitests in place. The project is currently at 86% code coverage. Another main task has been to provide a realistic demo on how to build a project with spline + Python + tox.

    Please check https://github.com/Nachtfeuer/pipeline/milestone/3?closed=1 for details. In addition one problem with PIPELINE_BASH_FILE and availability of model and env inside Docker container.

    The documentation of the first example can be seen here:

    • https://github.com/Nachtfeuer/pipeline/blob/master/docs/example.rst
    • or here: http://spline.readthedocs.io/en/latest/example.html
    Source code(tar.gz)
    Source code(zip)
Owner
Thomas Lehmann
Highly interested in development, programming languages, infrastructure, processes and design. Favor C++, Java and Python.
Thomas Lehmann
Implementation of Kronecker Attention in Pytorch

Kronecker Attention Pytorch Implementation of Kronecker Attention in Pytorch. Results look less than stellar, but if someone found some context where

Phil Wang 16 May 06, 2022
An Industrial Grade Federated Learning Framework

DOC | Quick Start | δΈ­ζ–‡ FATE (Federated AI Technology Enabler) is an open-source project initiated by Webank's AI Department to provide a secure comput

Federated AI Ecosystem 4.8k Jan 09, 2023
Official Implementation for "ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement" https://arxiv.org/abs/2104.02699

ReStyle: A Residual-Based StyleGAN Encoder via Iterative Refinement Recently, the power of unconditional image synthesis has significantly advanced th

967 Jan 04, 2023
Code repository for our paper regarding the L3D dataset.

The Large Labelled Logo Dataset (L3D): A Multipurpose and Hand-Labelled Continuously Growing Dataset Website: https://lhf-labs.github.io/tm-dataset Da

LHF Labs 9 Dec 14, 2022
Using BERT+Bi-LSTM+CRF

Chinese Medical Entity Recognition Based on BERT+Bi-LSTM+CRF Step 1 I share the dataset on my google drive, please download the whole 'CCKS_2019_Task1

Xiang WU 55 Dec 21, 2022
Attention-guided gan for synthesizing IR images

SI-AGAN Attention-guided gan for synthesizing IR images This repository contains the Tensorflow code for "Pedestrian Gender Recognition by Style Trans

1 Oct 25, 2021
SNIPS: Solving Noisy Inverse Problems Stochastically

SNIPS: Solving Noisy Inverse Problems Stochastically This repo contains the official implementation for the paper SNIPS: Solving Noisy Inverse Problem

Bahjat Kawar 35 Nov 09, 2022
Official implementation of "OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Temporal Association" in PyTorch.

openpifpaf Continuously tested on Linux, MacOS and Windows: New 2021 paper: OpenPifPaf: Composite Fields for Semantic Keypoint Detection and Spatio-Te

VITA lab at EPFL 50 Dec 29, 2022
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (CVPR 2021)

ASGNet The code is for the paper "Adaptive Prototype Learning and Allocation for Few-Shot Segmentation" (accepted to CVPR 2021) [arxiv] Overview data/

Gen Li 91 Dec 23, 2022
Models Supported: AlbUNet [18, 34, 50, 101, 152] (1D and 2D versions for Single and Multiclass Segmentation, Feature Extraction with supports for Deep Supervision and Guided Attention)

AlbUNet-1D-2D-Tensorflow-Keras This repository contains 1D and 2D Signal Segmentation Model Builder for AlbUNet and several of its variants developed

Sakib Mahmud 1 Nov 15, 2021
DeepStruc is a Conditional Variational Autoencoder which can predict the mono-metallic nanoparticle from a Pair Distribution Function.

ChemRxiv | [Paper] XXX DeepStruc Welcome to DeepStruc, a Deep Generative Model (DGM) that learns the relation between PDF and atomic structure and the

Emil Thyge Skaaning Kjær 13 Aug 01, 2022
Code for "NeRS: Neural Reflectance Surfaces for Sparse-View 3D Reconstruction in the Wild," in NeurIPS 2021

Code for Neural Reflectance Surfaces (NeRS) [arXiv] [Project Page] [Colab Demo] [Bibtex] This repo contains the code for NeRS: Neural Reflectance Surf

Jason Y. Zhang 234 Dec 30, 2022
Fusion-in-Decoder Distilling Knowledge from Reader to Retriever for Question Answering

This repository contains code for: Fusion-in-Decoder models Distilling Knowledge from Reader to Retriever Dependencies Python 3 PyTorch (currently tes

Meta Research 323 Dec 19, 2022
A python library for implementing a recommender system

python-recsys A python library for implementing a recommender system. Installation Dependencies python-recsys is build on top of Divisi2, with csc-pys

Oscar Celma 1.5k Dec 17, 2022
Website for D2C paper

D2C This is the repository that contains source code for the D2C Website. If you find D2C useful for your work please cite: @article{sinha2021d2c au

1 Oct 21, 2021
Repository for the Bias Benchmark for QA dataset.

BBQ Repository for the Bias Benchmark for QA dataset. Authors: Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Tho

MLΒ² AT CILVR 18 Nov 18, 2022
Code for "Universal inference meets random projections: a scalable test for log-concavity"

How to use this repository This repository contains code to replicate the results of "Universal inference meets random projections: a scalable test fo

Robin Dunn 0 Nov 21, 2021
Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness

Imbalanced Gradients: A Subtle Cause of Overestimated Adversarial Robustness Code for Paper "Imbalanced Gradients: A Subtle Cause of Overestimated Adv

Hanxun Huang 11 Nov 30, 2022
Code for approximate graph reduction techniques for cardinality-based DSFM, from paper

SparseCard Code for approximate graph reduction techniques for cardinality-based DSFM, from paper "Approximate Decomposable Submodular Function Minimi

Nate Veldt 1 Nov 25, 2022
Boosted neural network for tabular data

XBNet - Xtremely Boosted Network Boosted neural network for tabular data XBNet is an open source project which is built with PyTorch which tries to co

Tushar Sarkar 175 Jan 04, 2023