To set up and deploy a Data Collector engine in your corporate network, you create environments and deployments in Control Hub.

Pipelines and most pipeline stages include advanced options with default values that should work in most cases. By default, each pipeline and stage hides the advanced options. Advanced options can include individual properties or complete tabs.

Runtime values are values that you define outside of the pipeline and use for stage and pipeline properties. You can change the values for each pipeline run without having to edit the pipeline.

Many stages can use SSL/TLS encryption to securely connect to the external system.

When you use an origin to read log data, you define the format of the log files to be read.

You can use the whole file data format to transfer entire files from an origin system to a destination system. With the whole file data format, you can transfer any type of file.

An origin stage represents the source for the pipeline. You can use a single origin stage in a pipeline.

This solution describes how to convert Avro files to the columnar format, Parquet.

This solution describes how to configure a Drift Synchronization Solution for Hive pipeline to automatically refresh the Impala metadata cache each time changes occur in the Hive metastore.

This solution describes how to design a pipeline that writes output files to a destination, moves the files to a different location, and then changes the permissions for the files.

This solution describes how to design a pipeline that stops automatically after it finishes processing all available data.

This solution describes how to offload data from relational database tables to Hadoop.

This solution describes how to design a pipeline to send email notifications at different moments during pipeline processing.

This solution describes how to design a pipeline that preserves an audit trail of pipeline and stage events that occur.

You can use several solutions to load data into a Delta Lake table on Databricks.

The Drift Synchronization Solution for Hive detects drift in incoming data and updates corresponding Hive tables.

The Drift Synchronization Solution for PostgreSQL detects drift in incoming data and automatically creates or alters corresponding PostgreSQL tables as needed before the data is written.

A microservice pipeline is a pipeline that creates a fine-grained service to perform a specific task.

Data preview displays different colors for different types of data. Preview also uses other codes and formatting to highlight changed fields.

After running preview for a pipeline, you can view the input and output schema for each stage on the Schema tab in the pipeline properties panel. The schema includes each field path and data type.

You can preview data for a group of linked stages within a pipeline.

You can edit preview data to view how a stage or group of stages processes the changed data. Edit preview data to test for data conditions that might not appear in the preview data set.

In data preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the expression in an Expression Evaluator to see how the expression alters data.

The basic tutorial creates a pipeline that reads a file from a directory, processes the data in two branches, and writes all data to a file system. You'll use data preview to help configure the pipeline, and you'll create a data alert and run the pipeline.

The extended tutorial builds on the basic tutorial, using an additional set of stages to perform some data transformations and write to the Trash development destination. We'll also use data preview to test stage configuration.