WebSocket Server
The WebSocket Server origin is a multithreaded origin that listens on a WebSocket endpoint and processes the contents of all authorized WebSocket client requests. Use the WebSocket Server origin to read high volumes of WebSocket client requests using multiple threads.
The WebSocket Server origin can use multiple threads to enable parallel processing of data from multiple WebSocket clients.
Before you configure the origin, perform additional steps to configure the WebSocket clients.
When you configure the WebSocket Server origin, you specify the maximum number of concurrent requests to determine how many threads to use. You define the listening port, application ID, and the maximum message size. You can also configure SSL/TLS properties, including default transport protocols and cipher suites.
Prerequisites
Before you run a pipeline with the WebSocket Server origin, complete the following prerequisites to configure the WebSocket clients.
Send Data to the Listening Port
Configure the WebSocket clients to send data to the WebSocket Server listening port.
When you configure the WebSocket Server origin, you define a listening port number where the origin listens for data. To pass data to the pipeline, configure each WebSocket client to send data to a URL that includes the listening port number.
<ws | wss>://<sdc_hostname>:<listening_port>/The URL includes the following components:
- <ws | wss> - Use wss for secure WebSocket connections over HTTPS.
- <sdc_hostname> - The Data Collector host name.
- <listening_port> - The port number where the origin listens for data.
For example: wss://localhost:8080/
Include the Application ID in Requests
Configure the WebSocket clients to include the WebSocket Server application ID in each request.
When you configure the WebSocket Server origin, you define an application ID that is used to pass requests to the origin. All messages sent to the origin must include the application ID.
Include the application ID for each client request in one of the following ways:
- In request headers
- Add the following information to the request header for all WebSocket requests
that you want the origin to
process:
X-SDC-APPLICATION-ID: <application_ID> - In a query parameter in the URL
- If you cannot configure the client request headers - for example if the requests are generated by another system - then configure each WebSocket client to send data to a URL that includes the application ID in a query parameter.
Multithreaded Processing
The WebSocket Server origin performs parallel processing and enables the creation of a multithreaded pipeline.
The WebSocket Server origin uses multiple concurrent threads based on the Max Concurrent Requests property. Each thread connects to the origin system and creates a batch of data, and passes the batch to an available pipeline runner.
A pipeline runner is a sourceless pipeline instance - an instance of the pipeline that includes all of the processors and destinations in the pipeline and represents all pipeline processing after the origin. Each pipeline runner processes one batch at a time, just like a pipeline that runs on a single thread. When the flow of data slows, the pipeline runners wait idly until they are needed, generating an empty batch at regular intervals. You can configure the Runner Idle Time pipeline property specify the interval or to opt out of empty batch generation.
Multithreaded pipelines preserve the order of records within each batch, just like a single-threaded pipeline. But since batches are processed by different pipeline instances, the order that batches are written to destinations is not ensured.
For example, say you set the Max Concurrent Requests property to 5. When you start the pipeline, the origin creates five threads, and Data Collector creates a matching number of pipeline runners. Upon receiving data, the origin passes a batch to each of the pipeline runners for processing. In the batch, WebSocket Server includes only the WebSocket requests with the specified application ID.
At any given moment, the five pipeline runners can each process a batch, so this multithreaded pipeline processes up to five batches at a time. When incoming data slows, the pipeline runners sit idle, available for use as soon as the data flow increases.
For more information about multithreaded pipelines, see Multithreaded Pipeline Overview.
Data Formats
The WebSocket Server origin processes data differently based on the data format that you select. The origin processes the following types of data:
- Binary
- Generates a record with a single byte array field at the root of the record.
- Datagram
- Generates a record for every message. The origin can process collectd messages, NetFlow 5 and NetFlow 9 messages, and the
following types of syslog messages:
- RFC 5424 (https://tools.ietf.org/html/rfc5424)
- RFC 3164 (https://tools.ietf.org/html/rfc3164)
- Non-standard common messages, such as RFC 3339 dates with no version digit
- Delimited
- Generates a record for each delimited line. You can use the
following delimited format types:
- Default CSV - File that includes comma-separated values. Ignores empty lines in the file.
- RFC4180 CSV - Comma-separated file that strictly follows RFC4180 guidelines.
- MS Excel CSV - Microsoft Excel comma-separated file.
- MySQL CSV - MySQL comma-separated file.
- PostgreSQL CSV - PostgreSQL comma-separated file.
- PostgreSQL Text - PostgreSQL text file.
- Tab-Separated Values - File that includes tab-separated values.
- Custom - File that uses user-defined delimiter, escape, and quote characters.
- JSON
- Generates a record for each JSON object. You can process JSON files that include multiple JSON objects or a single JSON array.
- Log
- Generates a record for every log line.
- Protobuf
- Generates a record for every protobuf message. By default, the origin assumes messages contain multiple protobuf messages.
- SDC Record
- Generates a record for every record. Use to process records generated by a Data Collector pipeline using the SDC Record data format.
- Text
- Generates a record for each line of text or for each section of text based on a custom delimiter.
- XML
- Generates records based on a user-defined delimiter element. Use an XML element directly under the root element or define a simplified XPath expression. If you do not define a delimiter element, the origin treats the XML file as a single record.
Configuring a WebSocket Server Origin
Configure a WebSocket Server origin to generate multiple threads for parallel processing of WebSocket client requests.