Metadata-Version: 2.1
Name: spire-pipeline
Version: 0.6.1
Summary: Run software pipelines using YAML files
Home-page: https://github.com/lamyj/spire/
Author: Julien Lamy
Author-email: lamy@unistra.fr
License: MIT
Description: # Welcome to Spire
        
        Spire is a pipeline runner based on [YAML](https://en.wikipedia.org/wiki/YAML) files and [Ninja](https://ninja-build.org/). The pipeline descriptions (i.e. the list of steps to run) are stored in [Jinja](http://jinja.pocoo.org/)-templated YAML files which allow both readability and modularity. Using Ninja as a backend ensures fast and correct dependency handling.
        
        ## Quickstart
        
        Install Ninja, following [official instructions](https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages).
        
        From the source directory, use `pip` to ensure that dependencies are installed along with Spire:
        ```
        pip install spire-pipeline
        ```
        
        A pipeline description is a templated YAML file containing a list of `steps`, each step having the following fields:
        * `id`: unique name to reference this step from other steps
        * `inputs`: files that must be present for the step to execute
        * `outputs`: files that will be generated by the step
        * `commands`: a list of commands (optional for `phony` rules, see below)
        * `phony`: flag indicating whether the target is phony i.e. does not produce any output (optional, defaults to `false`)
        
        The `id` and `outputs`, and `inputs` fields are mandatory for all steps. For non-`phony` rules, the `commands` is optional. The `inputs`, `outputs` and `commands` may be written either as lists or single values. 
        
        The YAML content can be templated using Jinja, with extra features:
        * Filters to convert to YAML (`foo|yaml`) or to JSON (`foo|json`)
        * Function to perform globbing in the directory where the build will be executed (`glob("*.c")`)
        * The `basename` and `dirname` functions from the Python module `os.path`
        * The current build directory (`build_directory`)
        * The absolute path to the directory containing the current pipeline (`pipeline_directory`)
        * References to the outputs or inputs of the current step or of another step, specified by its identifier (`{{ inputs("other_step_id") }}`, `{{ outputs("other_step_id") }}`). These references are always list.
        * Functions to load data from a file in CSV, JSON or YAML format (respectively `load_csv`, `load_json`, `load_yaml`).
        
        The following example demonstrates most of those features. The first step looks for `.c` files and compile them to `.o`; the second step links them together to build an executable.
        
        ```yaml
        steps:
          - id: compile
            inputs: {{ glob("*.c") }}
            outputs: {{ inputs("compile")|replace(".c", ".o") }}
            commands: gcc -c {{ inputs("compile")|join(" ") }}
          
          - id: link
            inputs: {{ outputs("compile") }}
            outputs: foo
            commands: 
              - gcc -o {{ outputs("link")[0] }} {{ inputs("link")|join(" ") }}
              - strip {{ outputs("link")[0] }}
        ```
        
        If you use Atom and the [atom-jinja2](https://atom.io/packages/atom-jinja2) package, save it as `.yml.j2` for better syntax highlighting.
        
        The runner takes two sets of options: everything before a litteral `--` is passed to the parser, everything after is passed to the build engine. The parser expects the file name containing the pipeline description (mandatory), and a list of variable definitions that will be fed to Jinja, in the form `key=value`.
        
        The following willl parse `build.yml.j2` with the variable `root` containing the current working directory, and will then build the target `foo`.
        
        ```bash
        spire build.yml.j2 root=$(pwd) -- foo
        ```
        
        ## Design
        
        The input data is a text file containing a YAML-templated description of the pipeline steps. Three entities will process this data in order to run the pipeline:
        * Parser: run the template engine (e.g. globbing, root directory, number of slots for parallel jobs, etc.) and handle cross-step references
        * Generator: convert the rendered description to build-engine specific file
        * Build engine: run the build itself with user-specified options
        
        Under this design, multiple build engines and their corresponding generators may be implemented; the build engine of choice is Ninja, due to its ability to handle multiple outputs per step and to its parallel build features.
        
Keywords: pipeline,workflow,task execution
Platform: UNKNOWN
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: Intended Audience :: Science/Research
Classifier: Topic :: Software Development :: Build Tools
Classifier: Topic :: Scientific/Engineering
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 2
Classifier: Programming Language :: Python :: 3
Description-Content-Type: text/markdown
