A Note on Engine Types

From version 7.0.1 onwards, we have centralized all supported engine types into a single enumeration class streamsets.sdk.utils.EngineType.

Previous to this version, depending on the object you would refer to a DataCollector engine as either COLLECTOR, data_collector or DC. For example,

engine = sch.engines.get(engine_type='COLLECTOR')
sch.get_pipeline_builder(engine_type='data_collector', engine_id=engine.id)
sch.get_deployment_builder.build(
    engine_type='DC',
    ...
)

These strings remain valid, although we now accept the types in streamsets.sdk.utils.EngineType on every context.

engine = sch.engines.get(engine_type='COLLECTOR')
sch.get_pipeline_builder(engine_type='COLLECTOR', engine_id=engine.id)
sch.get_deployment_builder.build(
    engine_type='COLLECTOR',
    ...
)

Additionally, the engine type attribute for an object (e.g. streamsets.sdk.sch_models.PipelineBuilder) will return the corresponding streamsets.sdk.utils.EngineType instance, no matter which string was used for the input.

engine = sch.engines.get(engine_type='COLLECTOR')
engine.engine_type # <EngineType.COLLECTOR: 'COLLECTOR'>

For convenience, we provide an instance of each engine type in streamsets.sdk.constants. For example,

from streamsets.sdk.constants import COLLECTOR

engine = sch.engines.get(engine_type=COLLECTOR)[0]
sch.get_pipeline_builder(engine_type=COLLECTOR, engine_id=engine.id)
sch.get_deployment_builder.build(
    engine_type=COLLECTOR,
    ...
)

Tip

The available engine types are COLLECTOR, TRANSFORMER and SNOWPARK.