Working with Data Governance Tools

1. Using the link provided in the email that you received from StreamSets, download the tarball containing the Apache Atlas stage library.
2. Extract the tarball.
   The tarball is extracted into the following directory:

   <stage library name>/lib/

   For example, the Apache Atlas version 0.8.1 stage library is extracted to:

   streamsets-datacollector-apache-atlas_0_8_1-lib/lib/

   The Apache Atlas version 1.1.0 stage library is extracted to:

   streamsets-datacollector-apache-atlas_1_1_0-lib/lib/

3. Create a local directory external to the Data Collector installation directory for the custom stage libraries. Use an external directory to enable use of the libraries after Data Collector upgrades.
   For example, if you installed Data Collector in the following directory:

   /opt/sdc/

   you might create the custom stage library directory at:

   /opt/sdc-user-libs

4. Copy the Apache Atlas stage library to this custom stage library directory, as follows:

   /opt/sdc-user-libs/<stage library name>/lib/

   For example, for Apache Atlas version 0.8.1:

   /opt/sdc-user-libs/streamsets-datacollector-apache-atlas_0_8_1-lib/lib/

   For Apache Atlas version 1.1.0:

   /opt/sdc-user-libs/streamsets-datacollector-apache-atlas_1_1_0-lib/lib/

5. Add the USER_LIBRARIES_DIR environment variable to the appropriate file and point it to the custom stage library directory.

   Modify environment variables using the method required by your installation type.

   Set the environment variable as follows:

   export USER_LIBRARIES_DIR="<custom stage library directory>"

   For example:

   export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"

6. When using the Java Security Manager, which is enabled by default, update the Data Collector security policy to include the custom stage library directory as follows:
   1. In the Data Collector configuration directory, open the security policy file:

      $SDC_CONF/sdc-security.policy

   2. Add the following lines to the file:

      // custom stage library directory
      grant codebase "file://<custom stage library directory>/-" {
         permission java.security.AllPermission;
      };

      For example:

      // custom stage library directory
      grant codebase "file:///opt/sdc-user-libs/-" {
         permission java.security.AllPermission;
      };

7. Restart Data Collector.

1. To enable metadata publishing to Apache Atlas, modify the Data Collector configuration file, $SDC_CONF/sdc.properties.
2. Add lineage publisher properties to the configuration file.

   For Apache Atlas 0.8.1, add the following lines to the file:

   # Lineage Publisher
   #
   lineage.publishers=atlas
   lineage.publisher.atlas.def=streamsets-datacollector-apache-atlas_0_8_1-lib::com_streamsets_pipeline_stage_plugin_atlas_AtlasLineagePublisher

   For Apache Atlas 1.1.0, add the following lines to the file:

   # Lineage Publisher
   #
   lineage.publishers=atlas
   lineage.publisher.atlas.def=streamsets-datacollector-apache-atlas_1_1_0-lib::com_streamsets_pipeline_stage_plugin_atlas_AtlasLineagePublisher1_1_0

3. If Data Collector is installed on a separate machine from the Apache Atlas server, copy the atlas-application.properties file to the Data Collector machine.

   StreamSets recommends storing the atlas-application.properties file in the $SDC_CONF directory. For example, for an RPM installation of Data Collector, store the file in the /etc/sdc/ directory.

4. Verify that the atlas-application.properties file is correctly configured.
   For example, if you copied the atlas-application.properties file to the Data Collector machine, you might need to modify the following properties which specify the Kafka installation on the Apache Atlas server:

   atlas.kafka.zookeeper.connect=localhost:9026
   atlas.kafka.bootstrap.servers=localhost:9027

   For more information on the properties in this file, see the Apache Atlas documentation.

5. On the Data Collector machine, define the -Datlas.conf system property in the SDC_JAVA_OPTS environment variable.

   Modify environment variables using the method required by your installation type.

   The -Datlas.conf system property defines the location of the local atlas-application.properties file. Define the Atlas system property as follows:

   export SDC_JAVA_OPTS="${SDC_JAVA_OPTS} -Datlas.conf=<configuration directory> -Xmx1024m  -Xms1024m -server"

   The defined configuration directory must end in a forward slash (/). For example, if you stored the file in the $SDC_CONF directory for an RPM installation of Data Collector, define the Atlas system property as follows:

   export SDC_JAVA_OPTS="${SDC_JAVA_OPTS} -Datlas.conf=/etc/sdc/ -Xmx1024m  -Xms1024m -server"

6. Restart Data Collector for the changed properties to take effect.

Download the StreamSets data model and deploy the data model to the Apache Atlas server.

1. Using the link provided in the email that you received from StreamSets, download the StreamSets data model for Apache Atlas:

   0090-streamsets_model.json

2. Deploy the data model to the machine where the Apache Atlas server is installed.
   On the Apache Atlas server, copy the StreamSets data model to the same directory that contains the Sqoop data model.
   For example, if using Hortonworks version 2.6.1 for testing, copy the StreamSets data model to the following directory:

   /usr/hdp/2.6.1.0-129/atlas/models/0090-streamsets_model.json

3. Restart the Apache Atlas server so that the server can recognize the newly deployed data model.

1. Using the link provided in the email that you received from StreamSets, download the tarball containing the Cloudera Navigator stage library for your version of Cloudera Manager.
   For example, if Data Collector has the Cloudera CDH stage library version 5.14 installed, download the streamsets-datacollector-cm_5_14-lib-1.6.0.tgz file.
2. Extract the tarball.
   The tarball is extracted into the following directory:

   <stage library name>/lib/

   For example:

   streamsets-datacollector-cm_5_14-lib/lib/

3. On every node that runs Data Collector, create a local directory for the custom stage libraries.
   For example, you might create the custom stage library directory at:

   /opt/sdc-user-libs

4. On every node that runs Data Collector, copy the extracted Cloudera Navigator stage library to the directory defined for the USER_LIBRARIES_DIR environment variable as follows:

   /opt/sdc-user-libs/<stage library name>/lib/

   For example:

   /opt/sdc-user-libs/streamsets-datacollector-cm_5_14-lib/lib/

5. In Cloudera Manager, select the StreamSets service and then click Configuration.
6. On the Configuration page, in the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-env.sh field, add the USER_LIBRARIES_DIR environment variable and point it to the custom stage library directory, as follows:

   export USER_LIBRARIES_DIR="<custom stage library directory>"

   For example:

   export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"

7. When using the Java Security Manager, which is enabled by default, update the Data Collector Advanced Configuration Snippet (Safety Valve) for sdc-security.policy property to include the custom stage library directory as follows:

   // custom stage library directory
   grant codebase "file://<custom stage library directory>/-" {
      permission java.security.AllPermission;
   };

   For example:

   // custom stage library directory
   grant codebase "file:///opt/sdc-user-libs/-" {
      permission java.security.AllPermission;
   };

8. Restart Data Collector.

Add lineage publisher properties to the Data Collector configuration file, $SDC_CONF/sdc.properties, and then configure the properties as needed. When administering Data Collector with Cloudera Manager, configure the Data Collector configuration properties through the StreamSets service in Cloudera Manager. Manual changes to the configuration file can be overwritten by Cloudera Manager.

1. To enable metadata publishing, modify the Data Collector configuration file, $SDC_CONF/sdc.properties.
2. Add the following lineage publisher properties to the configuration file:

   # Lineage Publisher
   #
   lineage.publishers=navigator
   lineage.publisher.navigator.def=streamsets-datacollector-cm_5_14-lib::com_streamsets_pipeline_stage_plugin_navigator_NavigatorLineagePublisher
   lineage.publisher.navigator.config.application_url=http://localhost:18630
   lineage.publisher.navigator.config.navigator_url=http://navigatorhost:7187
   lineage.publisher.navigator.config.namespace=sdc
   lineage.publisher.navigator.config.username=user
   lineage.publisher.navigator.config.password=passwd
   lineage.publisher.navigator.config.autocommit=false

3. To define the connection to Cloudera Navigator, modify the default values of the following properties:

   Metadata Publisher Property	Description
   lineage.publishers	Application to publish metadata to. Use the default value, navigator.
   lineage.publisher.navigator.def	Stage library name used to connect to Cloudera Navigator, entered in the following format: <stage library name>::com_streamsets_pipeline_ stage_plugin_navigator_NavigatorLineagePublisher Define the appropriate Cloudera CDH stage library name to enable metadata publishing. For example, if you downloaded version 6.1 of the stage library, define the stage library name as follows: streamsets-datacollector-cm_6_1-lib Do not modify the second half of the property value.
   lineage.publisher.navigator. config.application_url	URL to Data Collector. For example: http://<hostname>:18630/
   lineage.publisher.navigator. config.navigator_url	URL to the Cloudera Navigator UI. For example: http://<Navigator Metadata Server host>:<port number>/ <Navigator Metadata Server host> is the name of the host on which you are running the Navigator Metadata Server role. <port number> is the port configured for the role.
   lineage.publisher.navigator. config.namespace	Namespace that Cloudera Navigator uses to uniquely identify Data Collector properties.
   lineage.publisher.navigator. config.username	User name to connect to Cloudera Navigator.
   lineage.publisher.navigator. config.password	Password for the Cloudera Navigator account. To protect the password, store the password in an external location and then use a function to retrieve the password.
   lineage.publisher.navigator. config.autocommit	Enables Cloudera Navigator to immediately process the published metadata. In most cases, use the default value of false. Setting to true can use a large number of resources on the Cloudera Navigator machine.

4. If Cloudera Navigator is configured for TLS/SSL, add the following lines to the Lineage Publisher section of the Data Collector configuration file and then modify the default values of the properties:

   lineage.publisher.navigator.config.ssl_truststore_location=truststore.jks
   lineage.publisher.navigator.config.ssl_truststore_password=changeit

   Metadata Publisher Property for TLS/SSL	Description
   lineage.publisher.navigator. config.ssl_truststore_location	Path to the truststore file containing the Cloudera Navigator Metadata Server certificate. Enter an absolute path to the file or a path relative to the Data Collector resources directory: $SDC_RESOURCES.
   lineage.publisher.navigator. config.ssl_truststore_password	Password to the truststore file. To protect the password, store the password in an external location and then use a function to retrieve the password.

5. Restart Data Collector for the changed properties to take effect.

1. Using the link provided in the email that you received from StreamSets, download the tarball containing the Collibra stage library.
2. Extract the tarball.
   The tarball is extracted into the following directory:

   <stage library name>/lib/

   For example, the Collibra version 1.0.0 stage library is extracted to:

   streamsets-datacollector-collibra-lib/lib/

3. Create a local directory external to the Data Collector installation directory for the custom stage libraries. Use an external directory to enable use of the libraries after Data Collector upgrades.
   For example, if you installed Data Collector in the following directory:

   /opt/sdc/

   you might create the custom stage library directory at:

   /opt/sdc-user-libs

4. Copy the Collibra stage library to this custom stage library directory, as follows:

   /opt/sdc-user-libs/<stage library name>/lib/

   For example, for Collibra version 1.0.0:

   /opt/sdc-user-libs/streamsets-datacollector-collibra-lib/lib/

5. Add the USER_LIBRARIES_DIR environment variable to the appropriate file and point it to the custom stage library directory.

   Modify environment variables using the method required by your installation type.

   Set the environment variable as follows:

   export USER_LIBRARIES_DIR="<custom stage library directory>"

   For example:

   export USER_LIBRARIES_DIR="/opt/sdc-user-libs/"

6. When using the Java Security Manager, which is enabled by default, update the Data Collector security policy to include the custom stage library directory as follows:
   1. In the Data Collector configuration directory, open the security policy file:

      $SDC_CONF/sdc-security.policy

   2. Add the following lines to the file:

      // custom stage library directory
      grant codebase "file://<custom stage library directory>/-" {
         permission java.security.AllPermission;
      };

      For example:

      // custom stage library directory
      grant codebase "file:///opt/sdc-user-libs/-" {
         permission java.security.AllPermission;
      };

7. Restart Data Collector.

1. To enable metadata publishing to Collibra, modify the Data Collector configuration file, $SDC_CONF/sdc.properties.
2. Add the following lineage publisher properties to the configuration file:

   # Lineage Publisher
   lineage.publishers=collibra
   lineage.publisher.collibra.def=streamsets-datacollector-collibra-lib::com_streamsets_pipeline_stage_plugin_collibra_CollibraLineagePublisher
   lineage.publisher.collibra.config.collibra_url=<collibra url>
   lineage.publisher.collibra.config.community=<community name>
   lineage.publisher.collibra.config.username=<user name>
   lineage.publisher.collibra.config.password=<password>

3. To define the connection to Collibra, configure the properties as follows:

   Metadata Publisher Property	Description
   lineage.publishers	Application to publish metadata to. Use the default value: collibra.
   lineage.publisher.collibra.def	Stage library name used to connect to Collibra. Use the default value: streamsets-datacollector-collibra-lib::com_streamsets_pipeline_stage_plugin_collibra_CollibraLineagePublisher
   lineage.publisher.collibra.config.collibra_url	URL for your Collibra instance. Use the following format: https://<collibra host>:<collibra port>
   lineage.publisher.collibra.config.community	Community that Collibra uses to uniquely identify Data Collector properties. Specify the community name that you created in the prerequisite tasks.
   lineage.publisher.collibra.config.username	User name to connect to Collibra. Specify the user account that you assigned to the Community Manager role in the prerequisite tasks.
   lineage.publisher.collibra.config.password	Password for the Collibra account.

4. Restart Data Collector.