Dataflow Triggers

Use the origin and processors that you need, like any other pipeline. And then, configure Hadoop FS to write Avro data to HDFS.

The Hadoop FS destination generates events each time it closes a file. This is perfect because we want to convert files to Parquet only after they are fully written.

On the General tab of the destination, select the Produce Events property.

With this property selected, the event output stream becomes available and Hadoop FS generates an event record each time it closes an output file. The event record includes the file path for the closed file.

Now, each time the MapReduce executor receives an event, it triggers the jobs that you configure it to run.

In the MapReduce executor, configure the MapReduce configuration details and select the Avro to Parquet job type. On the Avro Conversion tab, configure the file input and output directory information and related properties:

The Input Avro File property default, ${record:value('/filepath')}, runs the job on the file specified in the Hadoop FS file closure event record.

Then, on the Avro to Parquet tab, optionally configure advanced Parquet properties.

On the General tab, select the Produce Events property.

Now, the event output stream becomes available, and Hive Metastore destination generates an event record every time it updates the Hive metastore. The event record contains the name of the table that was created or updated.

With this property selected the event output stream becomes available, and Hadoop FS generates an event record every time it closes a file:

`${file:pathElement(record:value('/filepath'), -3)}`.`${file:pathElement(record:value('/filepath'), -2)}`

So when Hadoop FS completes a file, it writes the path of the written file in the filepath field, such as users/logs/server1weblog.txt. And the expression above properly interprets the database and table name as: logs.server1weblog.

In the Hive Query executor, configure the Hive configuration details on the Hive tab. If you have any trouble configuring the URL, see the Impala driver information in our Ask StreamSets post.

invalidate metadata ${record:value('/table')}

This query refreshes the Impala cache for the specified table. And the table is either the table in the Hive Metastore event record that was just updated or the table where Hadoop FS wrote a file.

Here's the final pipeline:

With these new event streams, each time the Hive Metastore destination creates a table, partition or column, and each time the Hadoop FS destination completes writing a file, the destinations generate event records. When the Hive Query executor receives an event record, it runs the Invalidate Metadata command so Impala can update its cache with the new information. Done!

On the General tab of the JDBC Query Consumer origin, select the Produce Events property.

The event output stream becomes available:

The JDBC Query Consumer generates several types of events: query success, query failure, and no-more-data. We know this because you checked the Event Record section of the JDBC Query Consumer documentation. Every event-generating stage has event details in a similar section.

The query success and failure events can be useful, so you might use a Stream Selector to route those records to a separate event stream. But let's say we don't care about those events, we just want the no-more-data event to pass to the Pipeline Finisher executor.

At this point, all events that the origin generates come to the executor. Since the JDBC Query Consumer origin generates multiple event types, this setup might cause the executor to stop the pipeline too soon.

With a precondition, only records that meet the specified condition can enter the stage.

We know that each event record includes the event type in the sdc.event.type record header attribute. So to ensure that only no-more-data events enter the stage, we can use the following expression in the precondition:

${record:eventType() == 'no-more-data'}

So here's the Pipeline Finisher:

We use the JDBC Multitable Consumer in the following simple pipeline, but your pipeline can be as complex as needed.

Now, when the origin completes processing all data, it passes a no-more-data event to the Pipeline Finisher. And after all pipeline tasks are complete, the executor stops the pipeline.

To do this, on the General tab, you configure the Start Event property, selecting the Hive Query executor as follows:

Notice, a Start Event - Hive Query tab now displays. This is because the executors for pipeline start and stop events do not display in the pipeline canvas - you configure the selected executor as part of the pipeline properties.

Also note that you can pass each type of pipeline event to one executor or to another pipeline for more complex processing. For more information about pipeline events, see Pipeline Event Generation.

You configure the connection properties as needed, then specify the query to use. In this case, you can use the following query, filling in the table name:

TRUNCATE TABLE IF EXISTS <table name>

Also, select Stop on Query Failure. This ensures that the pipeline stops and avoids performing any processing when the executor cannot complete the truncate query. The properties should look like this:

With this configuration, when you start the pipeline, the Hive Query executor truncates the specified table before data processing begins. And when the truncate completes successfully, the pipeline begins processing.

Configure the pipeline to pass the pipeline stop event to the HDFS File Metadata executor as follows:

With these configurations in place, when you start the pipeline, the Hive Query executor truncates the table specified in the query, then pipeline processing begins. When the JDBC Multitable Consumer completes processing all available data, it passes a no-more-data event to the Pipeline Finisher executor.

The Pipeline Finisher executor allows the pipeline stop event to trigger the HDFS File Metadata executor to create the empty file, then brings the pipeline to a graceful stop. Batch job complete!

On the General tab of the origin, select the Produce Events property.

Now, the event output stream becomes available. Note that the JDBC Query Consumer generates several types of events. You know this because you checked the Event Record section of the JDBC Query Consumer documentation. Every event-generating stage has event details in a similar section.

Of course, the Jython Evaluator will only generate events if your script is set up to do so. But you also need to enable the Produce Events property on the General tab.

1. The query failure event has the jdbc-query-failure event type, so on the Email tab, you use the following condition:

   ${record:eventType() == 'jdbc-query-failure'}

2. All of the email properties allow expressions, so for the email subject, you might include the pipeline name as follows:

   Query failure in ${pipeline:title()}!

3. When you compose the email body, you might use additional expressions to include information about the pipeline and information included in the event record in the email.

   Remember, the Event Record documentation lists all header attributes and fields in the event record, so you can refer to it when configuring the email. For more information about using expressions in email, see Using Expressions.

   For this email, you might include the following information:

   Pipeline ${pipeline:title()} encountered an error. 
   
   At ${time:millisecondsToDateTime(record:eventCreation() * 1000)}, the JDBC Query
   Consumer failed to complete the following query: ${record:value('/query')}
   
   Only the following number of rows were processed: ${record:value('/row-count')} 

The email configuration looks like this:

Since you want to send an email for each event that the Jython Evaluator generates, for the condition, you can use the event type defined in the script. Let's say it's "invalidTransaction". As with the first email, you can include additional information about the pipeline and data from the event record in the email body, as follows:

On the General tab, select the Produce Events property

Now the event output stream becomes available, and the destination generates an event each time it closes a file. For this destination, each event record includes fields for the file name, file path, and size of the closed file.

You could be done right there, but you want to include the time of the event in the record, so you know exactly when the Hadoop FS destination closed a file.

${record:attribute('sdc.event.creation_timestamp')}

The resulting pipeline looks like this:

Note that event creation time is expressed as an epoch or Unix timestamp, such as 1477698601031. And record header attributes provide data as strings.

Stage-generated event records differ from stage to stage. For a description of stage events, see "Event Record" in the documentation for the event-generating stage. For a description of pipeline events, see Pipeline Event Records.

For a list of logical event generation and executor pairings, see Logical Pairings.

For example, you might add an Expression Evaluator to add the event generation time to an event record before writing it to a destination. Or, you might use a Stream Selector to route different types of event records to different executors.