StreamSets TransformerTM is a design and execution engine that runs data processing pipelines on Apache Spark, an open-source cluster-computing framework. Because Transformer pipelines run on Spark deployed on a cluster, the pipelines can perform transformations that require heavy processing on the entire data set in batch or streaming mode.

Transformer functions as a Spark client that launches distributed Spark applications.

Transformer can run pipelines in batch mode. A batch pipeline processes all available data in a single batch, and then stops.

Transformer can run pipelines in streaming mode. A streaming pipeline maintains connections to origin systems and processes data as it becomes available. The pipeline runs continuously until you manually stop it.

Transformer can work with Apache Spark that runs locally on a single machine or that runs on a cluster. The steps that you use to install Transformer depend on your Spark installation.

A Transformer pipeline describes the flow of data from origin systems to destination systems and defines how to transform the data along the way.

For users already familiar with StreamSets Data Collector pipelines, here's how Transformer pipelines are similar... and different.

Transformer pipelines can run in batch or streaming mode.

Local pipelines run on the local Spark installation on the Transformer machine.

Cluster pipelines run on a cluster, where Spark distributes the processing across nodes in the cluster.

Cluster pipelines run on a cluster, where Spark distributes the processing across nodes in the cluster.

When you create a pipeline, you can define extra Spark configuration properties that determine how the pipeline runs on Spark. Transformer passes the configuration properties to Spark when it launches the Spark application.

When you start a pipeline, StreamSets Transformer launches a Spark application. Spark runs the application just as it runs any other application, splitting the pipeline data into partitions and performing operations on the partitions in parallel.

You can configure most origins and processors to cache data. You might enable caching when a stage passes data to more than one downstream stage.

To look up data in a Transformer pipeline, use an additional origin in the pipeline to read the lookup data, then use a Join processor to join the lookup data with the primary pipeline data.

You can configure Transformer to run a pipeline in ludicrous mode.

Transformer includes certain new features and stages with the Technology Preview designation. Technology Preview functionality is available for use in development and testing, but is not meant for use in production.

Configure a pipeline to define the flow of data. After you configure the pipeline, you can start the pipeline.

An origin stage represents the source for the pipeline. You can use a single origin stage in a pipeline. Or, you can use multiple origin stages in a pipeline and then join the origins using the Join processor.

When reading delimited or JSON data, you can configure an origin to use a custom schema to process the data. By default, origins infer the schema from the data.

You can preview data to help build or fine-tune a pipeline. You can preview complete or incomplete pipelines.

In Preview mode, Transformer displays different colors for different types of data. Transformer uses other codes and formatting to highlight changed fields.

When previewing data for a processor, you can preview both the input and the output data. You can display the output records in the order that matches the input records or in the order produced by the processor.

When running preview, you can edit stage properties to see how the changes affect preview data. For example, you might edit the condition in a Stream Selector processor to see how the condition alters which records pass to the different output streams.

When Transformer runs a pipeline, you can view real-time statistics about the pipeline.

When you monitor a pipeline, you can view real-time summary statistics for the pipeline and for stages in the pipeline.

As you monitor a pipeline, you can also access the Spark web UI for the application launched for the pipeline. Use the Spark web UI to monitor the Spark jobs executed for the launched application, just as you monitor any other Spark application.

You can view the run history of a pipeline when you configure or monitor a pipeline. View the run history from either the Summary or History tab.