Hadoop FS Standalone

For example, suppose you have the following folders:

/hr/employees/CA/SFO
/hr/employees/CA/SJC
/hr/employees/CA/LAX
/hr/employees/WA/SEA

When defining the directory, you can include wildcards to have the origin read files from different combinations of folders. The following table shows some possibilities:

Last Modified Timestamp

The Hadoop FS Standalone origin can read files in ascending order based on the last modified timestamp associated with the file. When the origin reads from a secondary location - not the directory where the files are created and written - the last-modified timestamp should be when the file is moved to the directory to be processed.

Tip: Avoid moving files using commands that preserve the existing timestamp, such as cp -p. Preserving the existing timestamp can be problematic in some cases, such as moving files across time zones.

When ordering based on timestamp, any files with the same timestamp are read in lexicographically ascending order based on the file names.

For example, when reading files with the log*.json file name pattern, the origin reads the following files in the following order:

File Name	Last Modified
log-1.json	APR 24 2016 14:03:35
log-903.json	APR 24 2016 14:05:03
log-2.json	APR 24 2016 14:45:11
log-3.json	APR 24 2016 14:45:11

Notice, the log-2.json and log-3.json files have identical timestamps, and so are processed in lexicographically ascending order based on their file names.

Lexicographically Ascending File Names

The Hadoop FS Standalone origin can read files in lexicographically ascending order based on file names. Note that lexicographically ascending order reads the numbers 1 through 11 as follows:

1, 10, 11, 2, 3, 4... 9

For example, when reading files with the web*.log file name pattern, the Hadoop FS Standalone origin reads the following files in the following order:

web-1.log
web-10.log
web-11.log
web-2.log
web-3.log
web-4.log
web-5.log
web-6.log
web-7.log
web-8.log
web-9.log

To read these files in logical and lexicographically ascending order, you might add leading zeros to the file naming convention as follows:

web-0001.log
web-0002.log
web-0003.log
...
web-0009.log
web-0010.log
web-0011.log

For example, you configure the Hadoop FS Standalone origin to read from the /logs/ file directory, select the Last Modified Timestamp read order, and enable reading from subdirectories. The Hadoop FS Standalone origin reads the following files in the following order based on timestamp, even though the files are written to different subdirectories.

To read from Azure Data Lake Storage, configure the Hadoop FS Standalone origin as follows:

1. Configure Azure credentials in one of the following ways:
   • If the Azure credentials are defined in the HDFS configuration file core-site.xml, configure access to the file.
     1. On the Hadoop FS tab, configure the Hadoop FS Configuration Directory property to point to the directory that includes the file.

     For information about defining the Azure credentials in the core-site.xml file, see the Hadoop documentation.

   • If the credentials are not defined in the core-site.xml file, use Hadoop FS configuration properties to pass Azure credentials:
     1. On the Hadoop FS tab, click the Add icon to add new Hadoop FS configuration properties, for the required credentials.

        You can use simple or bulk edit mode to add configuration properties.

     2. Enter the property names and values for the credential provider that you want to use.
        To connect using Azure Active Directory refresh tokens, configure the following properties:

        • fs.adl.oauth2.access.token.provider.type - Set to RefreshToken.
        • fs.adl.oauth2.client.id - Set to the application ID.
        • fs.adl.oauth2.refresh.token - Set to an OAuth2 refresh token generated through Azure Active Directory. For information on generating a refresh token, see the Microsoft documentation.

        To connect using Azure Active Directory Service Principal, configure the following properties:

        • fs.adl.oauth2.access.token.provider.type - Set to ClientCredential.
        • fs.adl.oauth2.refresh.url - Set to the OAuth 2 token endpoint URL for Azure Active Directory.
        • fs.adl.oauth2.client.id - Set to the application ID.
        • fs.adl.oauth2.credential - Set to a generated application key.

        For tips on locating and generating the required information in the Azure portal, see the Hadoop documentation.

2. In the origin, on the Hadoop FS tab, configure the Hadoop FS URI property using the following structure:

   adl://<account name>.azuredatalakestore.net/

   In the URI, <account name> is the name of the Azure Data Lake Storage account.

To read from HDInsight, configure the Hadoop FS Standalone origin as follows:

1. On the General tab of the Hadoop FS Standalone origin, for the Stage Library property, select the HDP stage library version 2.4 or later.
2. Configure Azure credentials in one of the following ways:
   • If the Azure credentials are defined in the HDFS configuration file core-site.xml, configure the origin to access the file.
     1. On the Hadoop FS tab, configure the Hadoop FS Configuration Directory property to point to the directory that includes the file.
   • If the credentials are not defined in the core-site.xml file, use a Hadoop FS configuration property to pass the Azure credentials:
     1. In the origin, on the Hadoop FS tab, click the Add icon to add a new Hadoop FS configuration property.

        You can use simple or bulk edit mode to add configuration properties.

     2. Enter the following property name, using the Azure storage account name for <storage account name>:

        fs.azure.account.key.<storage account name>.blob.core.windows.net

        For example, if the storage account name is sdchd, then enter the following name for the property:

        fs.azure.account.key.sdchd.blob.core.windows.net

        Tip: You can find the Azure storage account name on the Access Keys page in the Microsoft Azure portal. To view the page in the Microsoft Azure portal, click All Resources > Storage Account > Access Keys. A page like the following appears, with the storage account name and access keys:

        

     3. For the value of the Hadoop FS Configuration property, enter an access key value for the Azure storage account. You can use any valid key.
        Tip: The account key value also displays on the Access Keys page. For example, on the image above, you could use either the key1 or key2 value.
3. In the origin, on the Hadoop FS tab, configure the Hadoop FS URI property using the following structure:

   <wasb[s]>://<container name>@<storage account name>.blob.core.windows.net/<path to files>

   In the URI, <container name> is the Azure container name. And <storage account name> is the same Azure storage account name that you used for the Hadoop FS configuration property.

   For example, for a sdc-hd container in a storage account named sdchd, with all files in a files directory, you would define the Hadoop FS URI as follows:

   wasbs://sdc-hd@sdchd.blob.core.windows.net/files

   Tip: You can find the container name and storage account name on the Essentials page in the Microsoft Azure portal. For a standard storage account, in the Microsoft Azure portal, click All Resources > Storage Account > Overview > Blobs. For a blob storage account, click All Resources > Storage Account > Overview.

   A page like the following displays with the container name and storage account name:

   

   Though the host name for the Hadoop FS URI is <storage account name>.blob.core.windows.net, you can alternatively use the host name of the Azure blob service endpoint as the hostname for the Hadoop FS URI.

The Hadoop FS Standalone origin creates the following record header attributes:

• avroSchema - When processing Avro data, provides the Avro schema.
• baseDir - Base directory containing the file where the record originated.

• filename - Provides the name of the file where the record originated.
• file - Provides the file path and file name where the record originated.
• mtime - Provides the last-modified time for the file.
• offset - Provides the file offset in bytes. The file offset is the location in the file where the record originated.

• atime - Provides the last accessed time.
• isDirectory - Indicates if the file is a directory.
• isSymbolicLink - Indicates if the file is a symbolic link.
• size - Provides the file size.
• owner - Provides the file owner.
• group - Provides the group associated with the file.
• blocksize - Provides the block size of the file.
• replication - Provides the replication of the file.
• isEncrypted - Indicates if the file is encrypted.

Hadoop FS Standalone events can be used in any logical way. For example:

• With the Pipeline Finisher executor to stop the pipeline and transition the pipeline to a Finished state when the origin completes processing available data.

  When you restart a pipeline stopped by the Pipeline Finisher executor, the origin continues processing from the last-saved offset unless you reset the origin.

  For an example, see Case Study: Stop the Pipeline.

• With the Email executor to send a custom email after receiving an event.

  For an example, see Case Study: Sending Email.

• With a destination to store event information.

  For an example, see Case Study: Event Storage.

When a record is larger than the specified limit, the Hadoop FS Standalone origin processes the source file based on the stage error handling:

Discard

The origin discards the record and all remaining records in the file, and then continues processing the next file.

Send to Error

With a buffer limit error, the origin cannot send the record to the pipeline for error handling because it is unable to fully process the record.

Instead, the origin creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays in the pipeline history and displays as an alert when you monitor the pipeline.

If an error directory is configured for the stage, the origin moves the file to the error directory and continues processing the next file.

Stop Pipeline

The origin stops the pipeline and creates a message stating that a buffer overrun error occurred. The message includes the file and offset where the buffer overrun error occurred. The information displays as an alert and in the pipeline history.

Note: You can also check the Data Collector log file for error details.

You can configure the Hadoop FS Standalone origin to use individual HDFS properties or HDFS configuration files:

HDFS configuration files

You can use the following HDFS configuration files with the Hadoop FS Standalone origin:

• core-site.xml
• hdfs-site.xml

To use HDFS configuration files:

1. Store the files or a symlink to the files in the Data Collector resources directory.
2. In the Hadoop FS Standalone origin, configure the Hadoop FS Configuration Directory property to specify the location of the files.

Note: For a Cloudera Manager installation, Data Collector automatically creates a symlink to the files named hadoop-conf. Enter hadoop-conf for the location of the files in the Hadoop FS Standalone origin.

Individual properties

You can configure individual HDFS properties in the origin. To add an HDFS property, you specify the exact property name and the value. The Hadoop FS Standalone origin does not validate the property names or values.

Note: Individual properties override properties defined in the HDFS configuration file.

To configure a user in the Hadoop FS Standalone origin to read from HDFS, perform the following tasks:

1. On Hadoop, configure the user as a proxy user and authorize the user to impersonate a Hadoop user.

   For more information, see the Hadoop documentation.

2. In the Hadoop FS Standalone origin, on the Hadoop FS tab, configure the HDFS User property.

The Hadoop FS Standalone origin processes data differently based on the data format. The origin processes the following types of data:

Avro

Generates a record for every Avro record. The origin includes the Avro schema in the avroSchema record header attribute. It also includes a precision and scale field attribute for each Decimal field.

The origin expects each file to contain the Avro schema and uses the schema to process the Avro data.

The origin reads files compressed by Avro-supported compression codecs without requiring additional configuration.

Delimited

Generates a record for each delimited line. You can use the following delimited format types:

• Default CSV - File that includes comma-separated values. Ignores empty lines in the file.
• RFC4180 CSV - Comma-separated file that strictly follows RFC4180 guidelines.
• MS Excel CSV - Microsoft Excel comma-separated file.
• MySQL CSV - MySQL comma-separated file.
• Tab-Separated Values - File that includes tab-separated values.
• PostgreSQL CSV - PostgreSQL comma-separated file.
• PostgreSQL Text - PostgreSQL text file.
• Custom - File that uses user-defined delimiter, escape, and quote characters.
• Multi Character Delimited - File that uses multiple user-defined characters to delimit fields and lines, and single user-defined escape and quote characters.

You can use a list or list-map root field type for delimited data, and optionally include field names from a header line, when available. For more information about the root field types, see Delimited Data Root Field Type.

When using a header line, you can enable handling records with additional columns. The additional columns are named using a custom prefix and integers in sequential increasing order, such as _extra_1, _extra_2. When you disallow additional columns, records that include additional columns are sent to error.

You can also replace a string constant with null values.

When a record exceeds the maximum record length defined for the stage, the origin cannot continue reading the file. Records already read from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

JSON

Generates a record for each JSON object. You can process JSON files that include multiple JSON objects or a single JSON array.

When an object exceeds the maximum object length defined for the origin, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Log

Generates a record for every log line.

When a line exceeds the user-defined maximum line length, the origin truncates longer lines.

You can include the processed log line as a field in the record. If the log line is truncated, and you request the log line in the record, the origin includes the truncated line.

You can define the log format or type to be read.

Protobuf

Generates a record for every protobuf message.

Protobuf messages must match the specified message type and be described in the descriptor file.

When the data for a record exceeds 1 MB, the origin cannot continue processing data in the file. The origin handles the file based on file error handling properties and continues reading the next file.

For information about generating the descriptor file, see Protobuf Data Format Prerequisites.

SDC Record

Generates a record for every record. Use to process records generated by a Data Collector pipeline using the SDC Record data format.

For error records, the origin provides the original record as read from the origin in the original pipeline, as well as error information that you can use to correct the record.

When processing error records, the origin expects the error file names and contents as generated by the original pipeline.

Text

Generates a record for each line of text or for each section of text based on a custom delimiter.

When a line or section exceeds the maximum line length defined for the origin, the origin truncates it. The origin adds a boolean field named Truncated to indicate if the line was truncated.

For more information about processing text with a custom delimiter, see Text Data Format with Custom Delimiters.

Whole File

Streams whole files from the origin system to the destination system. You can specify a transfer rate or use all available resources to perform the transfer.

The origin generates two fields: one for a file reference and one for file information. For more information, see Whole File Data Format.

XML

Generates records based on a user-defined delimiter element. Use an XML element directly under the root element or define a simplified XPath expression. If you do not define a delimiter element, the origin treats the XML file as a single record.

Generated records include XML attributes and namespace declarations as fields in the record by default. You can configure the stage to include them in the record as field attributes.

You can include XPath information for each parsed XML element and XML attribute in field attributes. This also places each namespace in an xmlns record header attribute.

Note: Field attributes and record header attributes are written to destination systems automatically only when you use the SDC RPC data format in destinations. For more information about working with field attributes and record header attributes, and how to include them in records, see Field Attributes and Record Header Attributes.

When a record exceeds the user-defined maximum record length, the origin cannot continue processing data in the file. Records already processed from the file are passed to the pipeline. The behavior of the origin is then based on the error handling configured for the stage:

• Discard - The origin continues processing with the next file, leaving the partially-processed file in the directory.
• To Error - The origin continues processing with the next file. If a post-processing error directory is configured for the stage, the origin moves the partially-processed file to the error directory. Otherwise, it leaves the file in the directory.
• Stop Pipeline - The origin stops the pipeline.

Use the XML data format to process valid XML documents. For more information about XML processing, see Reading and Processing XML Data.

Tip: If you want to process invalid XML documents, you can try using the text data format with custom delimiters. For more information, see Processing XML Data with Custom Delimiters.