JavaScript Evaluator

Record by Record

The processor calls the script for each record. The processor passes the record to the script as a map and processes each record individually.

The script does not require error handling logic. Error records are passed to the processor for error handling. The processor handles error records based on the On Record Error property.

Use this mode to avoid including error handling logic in the code. Since this mode calls the script for each record, pipeline performance will be negatively affected.

Batch by Batch

The processor calls the script for each batch. The processor passes the batch to the script as an array and processes the batch at one time.

Include error handling logic in the script. Without error handling logic, a single error record sends the entire batch to the processor for error handling. The processor handles error records based on the On Record Error property.

Use this mode to improve performance by processing a batch of data at a time.

state

An object to store information between invocations of the init, main, and destroy scripts. A state is a map object that includes a collection of key/value pairs. You can use the state object to cache data such as lookups, counters, or a connection to an external system.

The state object functions much like a member variable:

• The information is transient and is lost when the pipeline stops or restarts.
• The state object is available only for the instance of the processor stage it is defined in. If the pipeline executes in cluster mode, the state object is not shared across nodes.

The same instance of the state object is available to all three scripts. For example, you might use the init script to open a connection to a database and then store a reference to that connection in the state object. In the main script, you can access the open connection using the state object. Then in the destroy script, you can close the connection using the state object.

Warning: The state object is best used for a fixed or static set of data. Adding to the cache on every record or batch can quickly consume the memory allocated to Data Collector and cause out of memory exceptions.

log

An object to write messages to the log. Includes four methods: info(), warn(), debug(), and trace().

The signature of the four methods is as follows:

(message-template, arguments...) 

The message template can have positional variables denoted by curly brackets: { }. The arguments are replaced in the message template curly brackets in positional manner, i.e., this is the first argument in the first { } occurrence, and so on.

output

An object that writes the record to the output batch. Includes a write(Record) method.

error

An object that passes error records to the processor for error handling. Includes a write(Record, message) method.

sdcFunctions

An object that runs functions that evaluate or modify data. Includes the following methods:

• getFieldNull(Record, 'field path') - Method that checks if a field is assigned a constant such as NULL_INTEGER or NULL_STRING.
• createRecord('recordId') - Method that creates a new record with the specified fields and values. The recordId is a string that uniquely identifies the record. It should include enough information to track down the record source.
• createMap(boolean listMap) - Method that creates a map for use as a field in a record. Pass true to create a list-map field, or false to create a map field.
• createEvent(String type, int version) - Method that creates a new event record with the specified event type and version. Be sure to enable event generation in the stage before implementing event methods.
• toEvent(Record) - Method that sends an event record to the Event output stream. Be sure to enable event generation in the stage before implementing event methods.

Data type of null values

The processor can associate null values with a data type. For example, if the JavaScript Evaluator assigns a null value to an Integer field, the field is returned to the pipeline as an integer with a null value.

Use constants in the JavaScript code to create a new field of a specific data type with a null value. For example, you can create a new String field with a null value by assigning the type constant NULL_STRING to the field as follows:

records[i].value.new_field = NULL_STRING

Date fields

Use the String data type to create a new field to store a date with a specific format. For example, the following sample code creates a new String field that stores the current date using the format YYYY-MM-dd:

 // Define a date object to record the current date
var date = new Date()

for(var i = 0; i < records.length; i++) {
  try {
    // Create a string field to store the current date with the specified format
    records[i].value.date = date.getFullYear()+ "-" + date.getMonth() + "-" + date.getDate()
    
    // Write record to the processor output
    output.write(records[i]);
  } catch (e) {
    // Send record to error
    error.write(records[i], e);
  }
}

Values retain their original type

Values retain their original type regardless of whether the processor modifies the value.

1. On the General tab, select the Produce Events property.

   This enables the event output stream for use.

2. In the script, include both of the following scripting objects:
   • sdcFunctions.createEvent(String type, int version) - Creates an event record with the specified event type and version number. You can create a new event type or use an existing event type. Existing event types are documented in other event-generating stages.

     The event record contains no record fields. Generate record fields as needed.

   • sdcFunctions.toEvent(Record) - Use to pass events to the event output stream.

• record.<header name> - Use to return the value of a read-only header attribute.
• record.attributes - Use to return a map of custom record header attributes, or to create or update a specific record header attribute.

input_stream = record.value['fileRef'].getInputStream()
input_stream.read()

input_stream.close()

var <class name> = Java.type('<package>.<class name>');

var DigestSHA3 = Java.type('org.bouncycastle.jcajce.provider.digest.SHA3.DigestSHA3');