Microservice Pipelines

A microservice pipeline is a pipeline that creates a fine-grained service to perform a specific task.

For example, you might use a microservice pipeline to validate changes to user accounts. You can have the pipeline check for matching accounts to validate account creation requests, account updates, and account deletions. The pipeline can tag requests as actionable or not and send all responses back to the originating system for next steps.

At this time, you can create REST-based microservice pipelines that use standalone execution mode. Other microservice types and platforms are planned for future releases.

A REST-based microservice pipeline uses a REST Service origin to listen for JSON-formatted requests at an HTTP endpoint. In the pipeline, you can use any available processor to transform and route the records as needed. Then, use one or more Send Response to Origin destinations to send records with user-defined status codes to the REST Service origin. The origin then transmits JSON-formatted responses back to the originating REST API client.

For more information and a descriptive video, see our blog post on microservice pipelines. For a full description of a sample microservice pipeline and a tutorial on creating one, see our GitHub tutorial, "Creating a CRUD Microservice Pipeline".

Stages for Microservice Pipelines

When you configure a microservice pipeline, you can use the REST Service origin and any available Data Collector processor, destination, and executor. Below are the stages that are specifically enabled to perform tasks in REST-based microservice pipelines:

REST Service origin - Listens on an HTTP endpoint, parses the contents of all authorized requests, and sends responses back to the originating REST API. Creates multiple threads to enable parallel processing in a multithreaded pipeline.
Send Response to Origin destination - Sends records to the REST Service origin with the specified response.

Sample Pipeline

When you create a microservice pipeline, a sample microservice pipeline displays in the configuration canvas. You can edit the pipeline to suit your needs. Or, you can create a standalone pipeline and use the microservice stages in a clean canvas.

The sample pipeline shows how you might use the REST Service origin and several Send Response to Origin destinations in a microservice pipeline:

The sample pipeline uses a Stream Selector to route data to different streams based on the REST API request method, performs additional processing, and then passes the records to the Send Response to Origin destinations. The destinations pass the records back to the REST Service origin with the specified status codes, such as 200 for OK or 400 for bad requests. The pipeline error handling also passes records back to the origin, with the 500 status code representing an internal server error. The REST Service origin generates a response with the records it receives and sends it to the originating client.

The pipeline description contains instructions on interacting with it using the curl command line HTTP client. For a step-by-step explanation of the sample pipeline, see "Creating a Microservice Pipeline" in the Creating a CRUD Microservice Pipeline tutorial.

Creating a Microservice Pipeline

Create a microservice pipeline to create a fine-grained microservice.

From the Home page or Getting Started page, click Create New Pipeline.

Tip: To get to the Home page, click the Home icon.
In the New Pipeline window, enter a pipeline title and optional description, and select the type of pipeline to create:
- Data Collector Pipeline - Select to design a Microservice pipeline using an empty canvas.
- Data Collector Edge Pipeline - Do not use for Microservice pipelines.
- Microservice Pipeline - Select to design a microservice pipeline based on the sample pipeline or to review the sample pipeline.
When starting with a sample pipeline, explore the pipeline and make changes as needed.
When starting with a blank canvas, perform the following steps:
1. On the General tab of the pipeline properties, ensure that Execution Mode is set to Standalone.
2. On the Error Records tab, to pass pipeline error records to the microservice origin for inclusion in the response as error records, use the Send Response to Origin option.
3. Optionally, on the Error Records - Send Response to Origin tab, configure the status code to use with the error records. Default is 500, representing an internal server error.
4. Configure the pipeline using the REST Service origin, Send Response to Origin destinations, and other stages as needed.