Pipeline Designer Tips

The Control Hub Pipeline Designer is closely based on the Data Collector pipeline configuration canvas. Some functional differences are described below:

Authoring Data Collector

When configuring a pipeline or pipeline fragment in Pipeline Designer, you select the authoring Data Collector to use. You can use the system Data Collector provided with Control Hub for exploration and light development. To perform explicit validation or data preview, use one of your registered Data Collectors as the authoring Data Collector.

Choose a registered Data Collector that is the same version as the Data Collectors that you intend to use to run the pipeline in production. Using a different Data Collector version can result in developing a pipeline that is invalid for production Data Collectors.

For example, if you use a more recent version of Data Collector for development than for production, you might include a stage, stage library, or stage functionality does not exist in the production Data Collector.

The registered Data Collector must meet certain requirements to be used as the authoring Data Collector. For more information, see Authoring Data Collectors.

Create a Pipeline or Fragment

When you create a pipeline or pipeline fragment, you specify whether the pipeline will run on Data Collector or SDC Edge. For pipelines, you can start with a blank canvas or an existing template.

Pipeline Designer provides several Data Collector and SDC Edge templates as sample pipelines. You can use them to familiarize yourself with Pipeline Designer and pipeline designs or you can use them as a basis for pipeline development.

Edit a Published Pipeline or Fragment

When viewing a published pipeline or pipeline fragment, Pipeline Designer displays the pipeline or fragment in read only mode. The mode displays above the canvas as shown:

To edit a published pipeline or fragment, click the Edit icon: . The pipeline or fragment then enters Edit mode.

Select Multiple Stages

When editing a pipeline or pipeline fragment, you can select multiple stages in the pipeline canvas and then move or delete the selected stages. To select multiple stages in the canvas, select the Shift key and then click each stage.

Update Stage Libraries for the Pipeline

When editing a pipeline or pipeline fragment, you can use the Update Stage Libraries dialog box to update the stage libraries for multiple stages in the pipeline or fragment.

This allows you to update all necessary stage libraries at one time when you change the authoring Data Collector for the pipeline or fragment. When preferred, you can also change stage libraries individually by editing each stage.

The stage libraries that display for each stage depends on the authoring Data Collector selected for the pipeline or fragment. For example, if the authoring Data Collector has the MapR 5.0 and 5.1 stage libraries installed, then these are the stage libraries that display for a MapR FS destination or MapR FS File Metadata executor.

To update multiple stage libraries at one time, click the More icon (), then select Update Stage Libraries.

The Update Stage Libraries dialog box displays the stage name and type for each stage in the pipeline or fragment. On the right is the corresponding list of stage libraries for the stage that are available on the authoring Data Collector.

Update the stage libraries as needed, then click Update to save your changes.

Work with Versions

When you have multiple versions of a pipeline or pipeline fragment, Pipeline Designer indicates the version of the pipeline or fragment that you are viewing. You can click the pipeline or fragment version to select a different version to view, as follows:

For more information about working with versions, see Version History.

Publish a Pipeline or Fragment

When you have completed work on a pipeline or fragment, you publish the pipeline or fragment. Publish a pipeline to create and run jobs based on the pipeline. Publish a fragment to make it available for testing or use in pipelines.

Use the Publish icon to publish a valid pipeline or fragment: . Enter a commit message stating what changed in this version so that you can track the commit history of the pipeline or fragment.

After you publish a pipeline, it enters read only mode and can be used to create a job. After you publish a fragment, it enters read only mode and can be included in pipelines.

Create a Job

After you publish a pipeline, you can create a job.

You can create a job using the Create Job icon in Pipeline Designer: . Or, you can create a job from the Jobs view.

Data Preview Requirement

You can preview data when the pipeline uses a registered Data Collector as the authoring Data Collector.

If the pipeline uses the system Data Collector or a selected registered Data Collector that is not accessible, the Preview Pipeline icon () is disabled.

Validation Requirement

You can perform explicit validation when the pipeline uses a registered Data Collector as the authoring Data Collector.

Use the Validate icon to perform explicit validation: .