src/index.js

Build Status

A framework for creating and running data pipelines. Data pipelines have stages, which are made of an arbitrary number of tasks. Stages and tasks are run in serial: once a task completes, the next task in the pipeline is executed.

Pipelines take a data object as input, and each task operates on the object in some way.

For an example of the Pipeliner in action, checkout the nxus-static-site module.

Installation

> npm install @nxus/pipeliner --save

Usage

Step 1: Define a pipeline

app.get('pipeliner').pipeline('my-pipeline')

Step 1a: Define stages

By default, every pipeline is pre-configured with three stages: 'collect', 'process', 'generate'. However, you can define your own stages:

app.get('pipliner').stages('my-pipeline', ['stage1', 'stage2', 'stage3'])

Step 2: Define tasks

A task is a javascript function that accepts any objects passed into the pipeline when it is run.

let myTask = (word) => {
  word.toUpperCase();
}

app.get('pipeliner').task('my-pipeline', 'process', myTask)

Step 3: Run a pipeline

Once all the tasks for a pipeline have been defined, the last step is to run the pipeline.

app.get('pipeliner').run('my-pipeline', someData)

API


Pipeliner

src/index.js

Examples

let myTask = (data) => {
  data.word.toUpperCase()
}
let data = {word: 'hello'}

let pipeliner = app.get('pipeliner')

pipeliner.pipeline('capitalize')
pipeliner.task('capitalize', 'process', myTask)
pipeliner.run('capitalize', data).then(() => {
  console.log('data') // {word: 'HELLO'}
})

Instance members

#getPipeline(pipeline)

Returns a specific pipeline

getPipeline(pipeline)

src/index.js

Returns a specific pipeline

Parameters

  • string pipeline :

    The name of a pipeline to return.

Returns

object :

The pipeline object.

#getPipelines

Returns all pipelines which have been defined

getPipelines

src/index.js

Returns all pipelines which have been defined

Returns

object :

A hash of the pipelines.

#pipeline(pipeline)

Create a new pipeline.

pipeline(pipeline)

src/index.js

Create a new pipeline.

Parameters

  • string pipeline :

    The name of the pipeline to create

#run(pipeline, args)

Runs the specified pipeline, passing the arguments to each task.

run(pipeline, args)

src/index.js

Runs the specified pipeline, passing the arguments to each task.

Parameters

  • string pipeline :

    the pipeline to run

  • ... object args :

    Arguments to pass to the pipeline tasks

Returns

Promise :

A promise that is executed when the pipeline completes.

#stages(stages, pipeline)

Define stages for a pipeline

stages(stages, pipeline)

src/index.js

Define stages for a pipeline

Parameters

  • array stages :

    An array of strings, each string being the name of a stage. Stages are executed in the order in the array.

  • string pipeline :

    The name of the pipeline to use the stages.

Returns

[ type ] :

[description]

#task(pipeline, stage, job)

Defintes a task for a pipeline and a stage.

task(pipeline, stage, job)

src/index.js

Defintes a task for a pipeline and a stage.

Parameters

  • string pipeline :

    The name of the pipeline

  • string stage :

    The name of the string

  • function job :

    A function which accepts data