Data Lake upgrade

When new versions or builds of Cloudera Runtime/Cloudera Manager are available for the Data Lake service, you can initiate a Data Lake upgrade. An OS upgrade may also be available. Use either the CDP CLI or the Management Console to initiate an upgrade.

There are several potential types of Data Lake upgrades:

  • Runtime and Cloudera Manager version upgrades, called "major/minor version" upgrades, which are available when a new Runtime and CM version is released.
  • Service pack upgrades, which do not change the major/minor Runtime or CM release, but upgrade the Data Lake to the latest CM and/or Runtime service packs available in the given version. These upgrades are made available as needed and can deliver targeted bug fixes for Cloudera Runtime, Cloudera Manager, or both.
  • OS upgrades, which do not change any CM or Runtime builds, but update the underlying cloud image. If an OS upgrade is available along with a major/minor version upgrade, the major/minor upgrade will incorporate the OS upgrade.

The type of upgrade that is available for a Data Lake is explicit on the Upgrade tab of the Data Lake details page. Any available upgrades are visible in the Target Cloudera Runtime Version drop-down menu. For example:

A major/minor version upgrade, where the major/minor versions of CM and Runtime are upgraded together:

A service pack upgrade, where no version upgrade is available, but a new Cloudera Runtime and/or Cloudera Manager service pack is available for upgrade:

An OS upgrade is designated by “(OS Upgrade Only)” when you select the drop-down menu:


The Data Lake upgrade process will:
  1. Check for a newer Cloudera Manager and Runtime version or build, and a new OS image.
  2. Automatically create a backup of the Data Lake (for major/minor and service pack upgrades).
  3. Execute the Data Lake upgrade.
  4. Verify the Data Lake state.

Major/minor version upgrades

Major/minor version upgrades are available as new versions of Cloudera Runtime and Cloudera Manager are released. Version upgrades combine a Runtime and CM upgrade into one operation. For example, this upgrade might involve upgrading from Runtime 7.2.8 to 7.2.9, and Cloudera Manager from 7.4.0 to 7.4.1.

Data Lake version upgrades require you to stop all of the Data Hubs in the environment before performing the Data Lake upgrade. The upgrade process stops all of the Data Lake services, downloads and distributes the new Cloudera Runtime, restarts the services, and deletes the old Cloudera Runtime. This process includes launching entirely new instance(s) from new cloud image(s). Once the Data Lake upgrade is complete, you can then either upgrade your Data Hub clusters to the same version as the Data Lake (if your Data Hub cluster type is supported for upgrade), or delete and recreate the clusters with the new Data Lake version. If your Data Hub is on Runtime version 7.2.16 or later, it is compatible with a Data Lake on a newer Runtime version (7.2.17+). You can independently upgrade your Data Hubs at a later time if you choose to, though it is not required

Service pack upgrades

The service pack upgrade process checks to see if a new Cloudera Manager or Cloudera Runtime (CDP) build is available, and then upgrades the Data Lake to the newest builds. Service pack upgrades do not upgrade to a new version of Cloudera Manager or Cloudera Runtime; they only upgrade to the latest service pack of a particular version. For example, a service pack upgrade cannot take the Data Lake from Runtime version 7.2.11 to version 7.2.12, but instead upgrades Runtime 7.2.11 to a newer 7.2.11 build. These upgrades are made available as needed and can deliver targeted bug fixes for Cloudera Runtime, Cloudera Manager, or both. If desired, specific older service packs can also be chosen using the CDP CLI. The service pack upgrade process follows the same steps as the version upgrade process.

OS upgrades

OS upgrades may be available sporadically as new images are created. OS upgrades are typically released to address security vulnerabilities or other issues on the host system. The OS upgrade process includes launching entirely new instances with the new OS image. An OS upgrade triggers the execution of any pre-service-deployment, post-cluster-manager-start, or post-service-deployment recipes.