Basic Tutorial

When you configure a pipeline, you need to decide what to do with error records. You can discard them or - more productively - write them to file, another pipeline, or to Kafka.

1. If you aren't already, log in to the Data Collector.
2. From the Home page or Getting Started page, click Create New Pipeline.
   Tip: To get to the Home page, click the Home icon.
3. In the New Pipeline window, enter a pipeline title and optional description, and choose to run the pipeline on Data Collector.
4. Click Save.
   An empty canvas displays:

   

   Note a few helpful areas:

   Number	Name	Description
   1	Pipeline Creation Help Bar	Indicates that the origin is missing and offers a list of origins to choose from. If the Pipeline Creation Help Bar does not display, you might have disabled it. To enable it, in the top right corner of the Data Collector window, click the Help > Settings. Clear the Hide Pipeline Creation Help Bar option.
   2	Issues icon	Click to display the list of pipeline issues found by implicit validation.
   3	Stage library panel	Displays a list of available origins by default.
   4	Stage menu	Changes the stages that display in the stage library. Displays all stages by default.
   5	Properties panel / Preview panel / Monitor panel	When you configure a pipeline, the Properties panel displays the properties of the pipeline or selected stage. You can resize, minimize and maximize the panel. When you preview data, the Preview panel displays the input and output data for the selected stage or group of stages. When you monitor a running pipeline, the Monitor panel displays real-time metrics and statistics. Note: Some icons and options might not display in the UI. The items that display are based on the task that you are performing and roles assigned to your user account.
   	Stage Library icon	Toggles the display of the Stage Library panel.

5. In the Properties panel, click the Error Records tab. And for the Error Records property, select Write to File.
   This writes error records to a file so you can deal with error records without having to stop the pipeline.
6. Click the Error Records - Write to File tab and configure the following properties.

   Use the defaults for properties that aren't listed:

   Write to File Property	Description
   Directory	Directory for error record files. Enter the directory that you set up for the tutorial. We recommended: /<base directory>/tutorial/error Note: To prevent validation errors, the directory must already exist.
   Files Prefix	This defines a prefix for error record files. By default, the files are prefixed with "SDC" and an expression that returns the Data Collector ID, but that's more than we need here. Delete the default and enter the following prefix: err_
   Max File Size	For the tutorial, reduce the files size to something more manageable, such as 5 or 1 MB.

The origin represents the incoming data for the pipeline. When you configure the origin, you define how to connect to the origin system, the type of data to be processed, and other properties specific to the origin.

1. To add the stage to the canvas, from the Pipeline Creation Help Bar, click Select Origin > Directory. Or, in the Stage Library panel, click the Directory origin: .
   The origin displays in the canvas and the Properties panel displays the properties for the stage.
2. In the Properties panel, click the Files tab and configure the following properties.

   Use the defaults for properties that aren't listed:

   Directory Property	Value
   Files Directory	Directory where you saved the sample file. Enter an absolute path. We recommended: /<base directory>/tutorial/origin.
   File Name Pattern	The Directory origin processes only the files in the directory that match the file name pattern. The tutorial sample file name is nyc_taxi_data.csv. Since the file is the only file in the directory, you can use something generic, like the asterisk wild card (*) or *.csv. If you had other .csv files in the directory that you didn't want to process, you might be more specific, like this: nyc_taxi*.csv. Or if you want to process files with prefixes for other cities, you might use *taxi*.csv.
   Read Order	This determines the read order when the directory includes multiple files. You can read based on the last-modified timestamp or file name. Because it's simpler, let's use Last Modified Timestamp.

3. Click the Data Formats tab, and configure the following properties.

   Use the defaults for properties that aren't listed:

   Delimited Property	Description
   Data Format	The data in the sample file is delimited, so select Delimited.
   Delimiter Format Type	Since the sample file is a standard CSV file, use the default: Default CSV (ignores empty lines).
   Header Line	The sample file includes a header, so select With Header Line.
   Root Field Type	This property determines how the Data Collector processes delimited data. Use the default List-Map. This allows you to use standard functions to process delimited data. With the List root field type, you need to use delimited data functions.

To become more familiar with the data set and gather some important details for pipeline configuration, let's preview the source data.

1. Above the pipeline canvas, click the Preview icon: .
   If the icon is not enabled, make sure you completed all the previous steps for the tutorial. If more than one issue displays in the list, correct any issue except Validation_0011.

   Validation_0011 just indicates that Directory origin isn't connected to anything yet and does not prevent data preview.

2. In the Preview Configuration dialog box, configure the following properties.

   Use the defaults for properties that aren't listed:

   Data Preview Property	Description
   Preview Source	Use the default Configured Source to use the sample source data.
   Write to Destinations and Executors	By default, this property is not selected. In general, you should keep this property clear to avoid writing data to destination systems or triggering tasks in destination systems.
   Show Field Type	By default, this property is selected. Keep it selected to see the data type of fields in the record.
   Remember the Configuration	Select to use these properties each time you run data preview. When you select this option, the UI enters data preview without showing this dialog box again.

3. Click Run Preview.
   The Preview Panel displays a list of 10 output records for Directory in list view. If you drill down a bit, you see that the each record displays an ordered list of field names and values with the data type of each field.

   Because the data is read from a file, all fields are strings:

   

   Note that in the Preview panel, the preview records display on the Records tab: . To review or change properties, use the Stage Configuration tab: . To change preview properties, use the Preview Configuration tab: .

4. For a full view of the data, click the Table View icon: .
   The Preview panel displays the first several columns of data.

   

5. To view all columns, click Show All.
   If you scroll to the right, you can verify that the credit_card field contains credit card numbers for the first three records. When you scroll back to the payment_type field, notice this corresponds to records where the payment type is "CRD".
6. Now that we have the information that we need, click the Close Preview icon: .

To route data to different streams for processing, we use the Stream Selector processor.

${record:value('/payment_type') == 'CRD'}

1. From the Pipeline Creation Help Bar, click Select Processor to Connect > Stream Selector. Or, in the Stage Library panel, select the Stream Selector processor () and connect the Directory origin to it.
2. On the General tab, click in the Required Fields text box.
   A list of available fields displays because you already performed data preview. It also displays when the pipeline is valid for data preview.
3. To discard records with no payment type information, select the following field: /payment_type.
   If a list does not appear, you can manually enter the field path: /payment_type.
4. To configure the Stream Selector condition, click the Conditions tab.
   A condition for the default stream displays. The default stream represents any records not captured by other conditions.
5. Click the Add icon: .
   A condition text box displays in the Property panel, and the corresponding output location appears on the stage in the canvas.
6. The following condition captures records where a credit card is used for payment. You can copy and paste the expression, but try typing it in to see how the expression completion feature helps you select the function and ensure valid syntax.

   ${record:value('/payment_type') == 'CRD'}

   All records that match this condition pass to the first output stream. All other records are captured by the default condition and passed through the second output stream.

Next, we'll evaluate credit card numbers to determine the credit card type. You can use an Expression Evaluator to do the same calculations, but with a short script, the Jython Evaluator is easier.

1. Add a Jython Evaluator processor to the canvas.

   If the Jython Evaluator processor isn’t listed in the stage library, you’ll need to install the Jython stage library first. By default, a full Data Collector installation includes the Jython stage library. The core installation does not include the Jython stage library.

   1. In the Data Collector top right toolbar, click the Package Manager icon:
      
   2. In the Package Manager search field, type “jy” so that the Package Manager displays the Jython stage library:
      
   3. Select the library, click the Install icon (), and then click Install to confirm your choice.
      Data Collector installs the external library and displays a message offering to restart Data Collector.
   4. Click Restart Data Collector.
   5. After Data Collector restarts, open the tutorial pipeline in the canvas and then add a Jython Evaluator processor to the canvas.
2. Connect the first output location of the Stream Selector to the Jython Evaluator.
   This routes records paid by credit card to the Jython Evaluator.
3. With the Jython Evaluator selected, in the Properties panel, click the Jython tab.
4. Use the default Batch by Batch record processing mode to process data in batches, instead of record by record.
5. In the Script text box, review the information in the comments, then delete it. Paste in the following script:

   try: 
     for record in records:
       cc = record.value['credit_card']
       if cc == '':
         error.write(record, "Payment type was CRD, but credit card was null")
         continue
   
       cc_type = ''
       if cc.startswith('4'):
         cc_type = 'Visa'
       elif cc.startswith(('51','52','53','54','55')):
         cc_type = 'MasterCard'
       elif cc.startswith(('34','37')):
         cc_type = 'AMEX'
       elif cc.startswith(('300','301','302','303','304','305','36','38')):
         cc_type = 'Diners Club'
       elif cc.startswith(('6011','65')):
         cc_type = 'Discover'
       elif cc.startswith(('2131','1800','35')):
         cc_type = 'JCB'
       else:
         cc_type = 'Other'
   
       record.value['credit_card_type'] = cc_type
   
       output.write(record)
   except Exception as e:
     error.write(record, e.message)

   Note: Incorrect indentation can cause Jython validation errors. For best results, copy the script from the online help. Copying the script from the PDF can result in incorrect indentation.

   To launch context-sensitive help, click the Help icon in the Properties panel. Then in the table of contents, scroll down to find the Data Collector Tutorial chapter > Basic Tutorial > Use Jython for Card Typing.

Now let's prevent sensitive information from reaching internal databases by using a Field Masker to mask the credit card numbers.

(.*)([0-9]{4})

1. Add a Field Masker processor to the canvas and connect the Jython Evaluator to it.
2. In the Properties panel, click the Mask tab.
3. Click in the Fields to Mask field. Scroll through the list of fields and select the field that represents credit card data: /credit_card.
   A list of fields displays when the pipeline is valid for data preview. If the list does not display, you can enter the field path manually.
4. To use a regular expression as a mask and display the last 4 digits of the credit card number, configure the rest of the properties as follows:

   Field Masker Property	Configuration
   Mask Type	Regular Expression
   Regular Expression	(.*)([0-9]{4})
   Groups to Show	2

The Data Collector can write data to many destinations. The Local FS destination writes to files in a local file system.

1. Add a Local FS destination to the canvas and connect the Field Masker to it.
2. Click the Output Files tab and configure the following properties.

   Use the defaults for properties that aren't listed:

   Local FS Property	Configuration
   Files Prefix	Defines a prefix for output file names. By default, the files are prefixed with "SDC" and an expression that returns the Data Collector ID, but that's more than we need here. Let's simplify and use "out_" instead.
   Directory Template	By default, the directory template includes datetime variables to create a directory structure for output files. This is intended for writing large volumes of data. Since we only have the sample file to process, we don't need the datetime variables. Go ahead and delete the default and enter the directory where you want the files to be written. We suggested: /<base directory>/tutorial/destination.
   Max File Size (MB)	For the tutorial, let's lower the file size to something manageable, like 5 or 1.

3. Click the Data Format tab, and configure the following properties.

   Use the defaults for properties that aren't listed:

   Delimited Property	Configuration
   Data Format	Delimited
   Header Line	With Header Line

The Jython Evaluator script added a new field to the credit payments branch. To ensure all records have the same structure, we'll use the Expression Evaluator to add the same field to the non-credit branch.

1. Click the Preview icon.
2. In the pipeline, click the Jython Evaluator to view the output of that processor.
3. Expand the first output record, scroll down, and notice the new field highlighted in green: /credit_card_type.
   
4. Click Close Preview.
5. Add an Expression Evaluator processor to the canvas and connect the second, default stream of the Stream Selector to it.
6. Click the Expressions tab.
7. Configure the following Field Expression properties:

   Output Field	Expression
   /credit_card_type	n/a

   This creates a credit_card_type field that indicates the information is not applicable.
   Since we're using "n/a" as a constant for the expression, we don't need to use the dollar sign and brackets for the expression. But if we wanted to use them, we could define them as ${'credit_card_type'} and ${'N/A'}.
8. Link the Expression Evaluator to the Local FS destination.
   This streams the data from this branch to the destination, merging data from both branches:

   

Now before we run the basic pipeline, let's add a data rule and alert. Data rules are user-defined rules used to inspect data moving between two stages. They are a powerful way to look for outliers and anomalous data.

1. Between the Stream Selector and the Jython Evaluator, select the link or Data Inspection icon: .
   The Data Rules tab displays in the Preview panel.
2. Click Add.
3. In the Data Rule dialog box, configure the following properties.

   Use the defaults for properties that aren't listed:

   Data Rule Property	Description
   Label	Missing Card Numbers
   Condition	${record:value("/credit_card") == ""}
   Sampling Percentage	35
   Alert Text	At least 10 missing credit card numbers!
   Threshold Value	10

   This creates an alert that goes off after finding ten records that don't include credit card numbers in the credit card payment stream.

   Note: With a larger data set, a smaller sampling percentage and higher threshold value might be appropriate, but we'll use these numbers for the purposes of the tutorial.
4. Click Save.
   The rule displays in the data rule list. And the Data Inspection icon darkens to show that a data rule is configured for the stream.
5. To enable the data rule and alert, click Active.
   Notice the Data Inspection icon becomes a darker grey to show that a data rule is active on the stream.