Code Generation

streamsets.sdk.codegen.generator.PythonGenerator is an entry point that converts existing pipelines to Python SDK code. Supported pipeline definition sources are:

• streamsets.sdk.sch_models.Pipeline object

• .zip archive

• .json file

Prerequisites

To properly generate a script, a few preparatory steps are required:

1. Make sure your Control Hub Credential Id is exported as an environment variable e.g. SCH_CREDENTIAL_ID.

2. Make sure your Control Hub Token is exported as an environment variable e.g. SCH_TOKEN.

These environments variables will be used by the generated script during execution.

Example usage

The example below demonstrates how to regenerate Python SDK code from pipeline exported to pipeline.zip archive.

from streamsets.sdk.codegen import PythonGenerator


generator = PythonGenerator(
    source="/path/to/file/pipeline.zip",
    destination="/tmp/output.py"
)
generator.save()

Another example demonstrates how to regenerate Python SDK code from the streamsets.sdk.sch_models.Pipeline object. Additionally, configuration options were used to use different environment variable names that stores the Control Hub credentials.

from streamsets.sdk.codegen import PythonGenerator


generator = PythonGenerator(
    source="/path/to/file/pipeline.zip",
    destination="/tmp/output.py",
    sch_credential_id="SCH_PROD_CRED_ID",
    sch_token="SCH_PROD_TOKEN"
)
generator.save()