Upgrade an Installation from the Tarball

Stop all running pipelines and then shut down the previous version of Data Collector.

1. Use one of the following methods to stop all running pipelines:

   • If the Data Collector is not registered to work with StreamSets Control Hub, stop the pipelines using the Data Collector UI.

     From the Data Collector Home page, select all running pipelines in the list and then click the Stop icon.

   • If the Data Collector is registered to work with StreamSets Control Hub, stop all jobs running on the Data Collector using the Control Hub UI.

     From the Control Hub Jobs page, filter the jobs by engine and by engine label. Select all active jobs in the list and then click the Stop Jobs icon.

2. Use one of the following methods to shut down Data Collector:
   • To use the command line for shutdown when Data Collector is started as a service, use the required command for your operating system.

     For CentOS 6, Oracle Linux 6, Red Hat Enterprise Linux 6, or Ubuntu 14.04 LTS, use: service sdc stop

     For CentOS 7, Oracle Linux 7, Red Hat Enterprise Linux 7, or Ubuntu 16.04 LTS, use: systemctl stop sdc

   • To use the Data Collector UI, click Administration > Shut Down. When the confirmation dialog box appears, click Yes.

Update the Data Collector environment variables so that the new version of Data Collector uses a new configuration directory but the same working data, log, and resource directories as the previous version.

1. Update the directory environment variables to use the following values:

   Environment Variable	Value
   SDC_CONF	New location outside of the base Data Collector runtime directory and unique from the previous renamed directory. For example, if you renamed the previous configuration directory to /etc/sdc3000, use the value /etc/sdc.
   SDC_DATA	Same directory that the previous version used.
   SDC_LOG	Same directory that the previous version used.
   SDC_RESOURCES	Same directory that the previous version used.

2. If you use any of the following environment variables, add them to the required file, and set them to the same directory used in the previous version:
   • SDC_EXTERNAL_RESOURCES
   • STREAMSETS_LIBRARIES_EXTRA_DIR
   • USER_LIBRARIES_DIR
3. Manually update the required file with any other customized environment variable values that you defined in the previous version.
4. Use the following command to create the Data Collector configuration directory at /etc/sdc:

   mkdir /etc/sdc

5. Use the following command from the directory where you extracted the tarball to copy all files from etc into the Data Collector configuration directory that you just created:

   cp -R etc/* /etc/sdc

6. To run Data Collector as a service, change the owner of the /etc/sdc directory and all files in the directory to the system user and group that starts Data Collector.
   By default, Data Collector uses a system user and group named sdc.
7. Use the following command to set owner only permission on the form-realm.properties file in the /etc/sdc directory:

   chmod go-rwx /etc/sdc/form-realm.properties

1. Compare the previous and new versions of the sdc.properties file, and update the new file as needed with the same customized property values.
2. If you registered the previous Data Collector to work with StreamSets Control Hub, complete the following steps to update the configuration files used by Control Hub:
   1. Compare the previous and new version of the dpm.properties file, and update the new file as needed with the same customized property values.
   2. Replace the new version of the application-token.txt file with the previous version of the file.
      The previous version of the file includes the authentication token that this Data Collector instance requires to issue authenticated requests to Control Hub. As a result, you'll need Data Collector to use the previous version of the file.
3. Compare the previous and new versions of the remaining files, and update the new files as needed with the same customized property values:
   • The appropriate realm.properties file, based on the authentication type that you use.
   • credential stores properties file
   • email-password.txt
   • keystore files
   • LDAP files
   • log4j properties file
   • security policy file
   • Vault properties file

     As of version 2.7.0.0, most of the Vault configuration properties have been moved to the new credential stores properties file. The properties use the same name, with an added "credentialStore.vault.config" prefix. If you are upgrading from a version earlier than 2.7.0.0, copy any values that you customized in the previous Vault properties file into the same property names in the credential stores properties file.