The Base64 Field Decoder decodes Base64 encoded data to binary data. Use the processor to decode Base64 encoded data before evaluating data in the field.
The Base64 Field Encoder encodes binary data using Base64. Use the processor to encode binary data that must be sent over channels that expect ASCII data.
The Data Parser processor allows you to parse supported data formats embedded in a field. You can parse NetFlow embedded in a byte array field or syslog messages embedded in a string field.
The Databricks ML Evaluator processor uses a machine learning model exported with Databricks ML Model Export to generate evaluations, scoring, or classifications of data. This processor is a Technology Preview feature. It is not meant for use in production.
The Encrypt and Decrypt Fields processor encrypts or decrypts field values.
The Field Hasher uses an algorithm to encode data. Use Field Hasher to encode highly-sensitive data. For example, you might use Field Hasher to encode social security or credit card numbers.
The Field Masker masks string values based on the selected mask type. You can use variable-length, fixed-length, custom, or regular expression masks. Custom masks can reveal part of the string value.
The Field Merger merges one or more fields in a record to a different location in the record. Use only for records with a list or map structure.
The Field Order processor orders fields in a map or list-map field and outputs the fields into a list-map or list root field.
Use the Field Renamer to rename fields in a record. You can specify individual fields to rename or use regular expressions to rename sets of fields.
The Field Replacer replaces values in fields with nulls or with new values. Use the Field Replacer to update values or to replace invalid values.
The Field Splitter splits string data based on a regular expression and passes the separated data to new fields. Use the Field Splitter to split complex string values into logical components.
The Field Type Converter processor converts the data types of fields to compatible data types. You might use the processor to convert the data types of fields before performing calculations. You can also use the processor to change the scale of decimal data.
The HBase Lookup processor performs key-value lookups in HBase and passes the lookup values to fields. Use the HBase Lookup to enrich records with additional data.
The HTTP Router processor passes records to data streams based on the HTTP method and URL path in the record header attributes.
The JDBC Lookup processor uses a JDBC connection to perform lookups in a database table and pass the lookup values to fields. Use the JDBC Lookup to enrich records with additional data.
The Kudu Lookup processor performs lookups in a Kudu table and passes the lookup values to fields. Use the Kudu Lookup to enrich records with additional data.
The MLeap Evaluator processor uses a machine learning model stored in an MLeap bundle to generate evaluations, scoring, or classifications of data. This processor is a Technology Preview feature. It is not meant for use in production.
The MongoDB Lookup processor performs lookups in MongoDB and passes all values from the returned document to a new list-map field in the record. Use the MongoDB Lookup processor to enrich records with additional data.
The PMML Evaluator processor uses a machine learning model stored in the Predictive Model Markup Language (PMML) format to generate predictions or classifications of data. This processor is a Technology Preview feature. It is not meant for use in production.
The Record Deduplicator evaluates records for duplicate data and routes data to two streams - one for unique records and one for duplicate records. Use the Record Deduplicator to discard duplicate data or route duplicate data through different processing logic.
The Redis Lookup processor performs key-value lookups in Redis and passes the lookup values to fields. Use the Redis Lookup to enrich records with additional data.
The Salesforce Lookup processor performs lookups in a Salesforce object and passes the lookup values to fields. Use the Salesforce Lookup to enrich records with additional data.
The Spark Evaluator performs custom processing within a pipeline based on a Spark application that you develop.
The Static Lookup processor performs lookups of key-value pairs that are stored in local memory and passes the lookup values to fields. Use the Static Lookup to store String values in memory that the pipeline can look up at runtime to enrich records with additional data.
The Stream Selector passes data to streams based on conditions. Define a condition for each stream of data that you want to create. The Stream Selector uses a default stream to pass records that do not match user-defined conditions.
The TensorFlow Evaluator processor uses a TensorFlow machine learning model to generate predictions or classifications of data. The TensorFlow Evaluator processor is a Technology Preview feature. It is not meant for use in production.
The Whole File Transformer processor transforms fully written Avro files to highly efficient, columnar Parquet files. Use the Whole File Transformer in a pipeline that reads Avro files as whole files and writes the transformed Parquet files as whole files.