Data Preview

Data Preview Overview

You can preview data to help build or fine-tune a pipeline. When using Control Hub, you can also use data preview when developing pipeline fragments.

You can use data preview with complete or incomplete pipelines and fragments. And you can choose from several options to provide source data for the preview.

When you preview data, source data passes through the pipeline or fragment, allowing you to review how the data passes and changes through each stage. You can edit stage properties and run the preview again to see how your changes affect the data. You can also edit preview data to test and tune the pipeline logic.

You can preview data for one stage at a time or for a group of stages. You can also view the data in list or table view, and refresh the preview data.

Data Preview Availability

You can preview complete and incomplete pipelines and Control Hub pipeline fragments. The Data Preview icon becomes active when data preview is available.

You can preview data under the following conditions:
  • The authoring Data Collector is an available registered Data Collector.
  • All stages in the pipeline are connected
  • All required properties are defined
Tip: Stage configuration does not have to be accurate or complete to preview data. After you connect all stages, you can enable data preview by entering any valid value for required properties.

Source Data for Data Preview

You can use the following types of data for a data preview:
  • Data from the origin - Use available data from the origin.
  • Data from the test origin - Use data from the test origin configured in the pipeline or fragment properties.

Writing to Destinations

As a tool for development, data preview does not write data to destinations by default.

If you like, you can configure the preview to write data to destinations. We advise against writing preview data to production destinations.

Notes

Keep the following notes in mind when previewing your data:
  • Date, datetime, and time data - Data preview displays date, datetime, and time data using the default format of the browser locale. For example, if the browser uses the en_US locale, preview displays dates using the following format: MMM d, y h:mm:ss a.

    Data preview displays date, datetime, and time data using the time zone that you select in the preview configuration. By default, data preview displays data using the browser time zone.

  • Oracle CDC Client pipelines - When previewing a pipeline that uses the Oracle CDC Client origin, data preview might time out before connecting to the origin system. When this occurs, try increasing the timeout to 120,000 milliseconds to allow the origin time to connect.
  • Whole file data format - When previewing a pipeline that processes whole file data, data preview displays only one record.

Preview Codes

In Preview mode, Pipeline Designer displays different colors for different types of data. Pipeline Designer uses other codes and formatting to highlight changed fields.

The following table describes the color and asterisk coding:
Preview Code Description
Black values Date data
Blue values Numeric data
Green values String data
Red values Boolean data
Asterisk Records that include edited field values
Red italic field labels Fields that contain edited data
Light red background Fields removed by a stage
Italic values Edited data
Green stage First stage in a multiple-stage preview
Red stage Last stage in a multiple-stage preview

Previewing a Single Stage

You can preview data for a single stage. In the Preview panel, you can review the values for each record to determine if the stage transforms data as expected.

  1. Above the pipeline canvas, click the Preview icon: .
    If the Preview icon is disabled, check the Validation Errors list for unconnected stages and required properties that are not defined.
  2. In the Preview Configuration dialog box, configure the following properties:
    Preview Property Description
    Preview Source Source data for the preview:
    • Configured Source - Provides data from the origin system.
    • Test Origin - Provides data from the test origin configured for the pipeline.
    Time Zone Time zone used to display date, datetime, or time data.

    Default is the browser time zone.
    Preview Batch Size Number of records to use in the preview. Honors values up to the Data Collector preview batch size.

    Default is 10. The Data Collector default is 10.
    Preview Timeout Milliseconds to wait for preview data. Use to limit the time data preview waits for data to arrive at the origin. Relevant for transient origins only.
    Write to Destinations and Executors Determines whether the preview passes data to destinations or executors.

    By default, does not pass data to destination or executor stages.
    Execute Pipeline Lifecycle Events Triggers the generation of any appropriate pipeline events, typically the Start event. If the event is configured to be used, event consumption is also triggered.
    Show Record/Field Header Displays record header attributes and field attributes when in List view. Attributes do not display in Table view.
    Show Field Type Displays the data type for fields in List view. Field types do not display in Table view.
    Remember the Configuration Stores the current preview configuration for use every time you request a preview for this pipeline.

    After you run data preview, you can change this option in the Preview panel by selecting the Preview Configuration icon () and clearing the option. The change takes effect the next time you run data preview.
  3. Click Run Preview.
    The Preview panel highlights the origin stage and displays preview data in list view. Since this is the origin of the pipeline, no input data displays.

    To view preview data in table view, click the Table View icon: .

  4. To view data for the next stage, select the stage in the pipeline canvas.
  5. To refresh the preview, click Reload Preview.
    Refreshing the preview provides a new set of data.
  6. To exit data preview, click Close Preview.

Previewing Multiple Stages

You can preview data for a group of linked stages within a pipeline.

When you preview multiple stages, you select the first stage and the last stage in the group. The Preview panel then displays the output data of the first stage in the group and the input data of the last stage in the group.

In the Preview panel, you can review the values for each record to determine if the group of stages transforms data as expected.

  1. Above the pipeline canvas, click the Preview icon: .
    If the Preview icon is disabled, check the Validation Errors list for unconnected stages and required properties that are not defined.
  2. In the Preview Configuration dialog box, configure the following properties:
    Preview Property Description
    Preview Source Source data for the preview:
    • Configured Source - Provides data from the origin system.
    • Test Origin - Provides data from the test origin configured for the pipeline.
    Time Zone Time zone used to display date, datetime, or time data.

    Default is the browser time zone.
    Preview Batch Size Number of records to use in the preview. Honors values up to the Data Collector preview batch size.

    Default is 10. The Data Collector default is 10.
    Preview Timeout Milliseconds to wait for preview data. Use to limit the time data preview waits for data to arrive at the origin. Relevant for transient origins only.
    Write to Destinations and Executors Determines whether the preview passes data to destinations or executors.

    By default, does not pass data to destination or executor stages.
    Execute Pipeline Lifecycle Events Triggers the generation of any appropriate pipeline events, typically the Start event. If the event is configured to be used, event consumption is also triggered.
    Show Record/Field Header Displays record header attributes and field attributes when in List view. Attributes do not display in Table view.
    Show Field Type Displays the data type for fields in List view. Field types do not display in Table view.
    Remember the Configuration Stores the current preview configuration for use every time you request a preview for this pipeline.

    After you run data preview, you can change this option in the Preview panel by selecting the Preview Configuration icon () and clearing the option. The change takes effect the next time you run data preview.
  3. Click Run Preview.
    The Preview panel highlights the origin stage and displays preview data in list view. Since this is the origin of the pipeline, no input data displays.

    To view preview data in table view, click the Table View icon: .

  4. To preview multiple stages, click Multiple.
    The preview canvas highlights the first stage and the last stage, as follows:

    The Preview panel displays the input and output data for the selected group of stages. You can review the details of each record.

  5. To change the first stage in the group, select the current first stage and then select the desired stage.
    For example, let's assume that you are previewing the pipeline displayed in the image above. To change the first stage, select Directory 1, the current first stage, and then select the desired first stage, such as Field Masker 1.
  6. To change the last stage in the group, select the current last stage and then select the desired stage.
  7. To refresh the preview, click Reload Preview.
    Refreshing the preview provides a new set of data.
  8. To exit data preview, click Close Preview.

Editing Preview Data

You can edit preview data to view how a stage or group of stages processes the changed data. Edit preview data to test for data conditions that might not appear in the preview data set.

For example, when the stage filters integer data based on an expression, you might change the input data to test positive and negative integer values, as well as zero.

You can edit preview data in the following locations:
  • The output data column for an origin.
  • The input data column for processors.

When you edit preview data, you can pass the changed data through the pipeline, or you can revert your changes to return to the original data.

  1. To change field values, in the Output Data column of an origin or the Input Data column for all other stages, click the value that you want to change and enter a new value.
    You can edit values for any input data.
  2. To process changed data, click Run With Changes.
    This runs the data preview with the current set of data and stage configuration.
    In the Input Data column, records with changed values display with an asterisk and the changed values are highlighted. The Output Data column displays the results of processing. You can change and process preview data as often as necessary.
  3. To refresh the preview, click Reload Preview.
    Refreshing the preview provides a new set of data.
  4. To revert changes to data, click Revert Data Changes.

Editing Properties

In data preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the expression in an Expression Evaluator to see how the expression alters data.

When you edit properties, you can test the change with the existing preview data or you can refresh the preview data.

When changing properties in the origin, refresh the preview data to test your changes. Refreshing the preview data allows the Data Collector to use the latest origin properties to process preview data instead of using the cached data.
Note: Unlike changes to data, you cannot automatically revert property changes. Manually revert any changes that you do not want to preserve.
  1. To edit stage properties while in data preview, select the stage you want to edit and click the Stage Configuration icon: .
  2. Change properties as needed.
  3. To test properties changed in the origin, click Reload Preview.
    This refreshes the preview data. Based on the origin type, it might use the same data or a new set of data with the updated origin properties.
    To test properties in any non-origin stage using the same set of data, click Run With Changes.
  4. If you want to revert your change, manually change the property back.