Upgrading Data Hubs

You can upgrade a Data Hub cluster in one of two ways: Runtime and Cloudera Manager major/minor version upgrades and maintenance/“hotfix” upgrades.

Draft comment:
You can upgrade a Data Hub cluster in one of three ways: Runtime and Cloudera Manager major/minor version upgrades, maintenance/“hotfix” upgrades, and OS upgrades.

Cloudera Runtime and Cloudera Manager Major/Minor Version Upgrades

A CDP environment consists of both Data Lake and Data Hub clusters, and currently these clusters should run the same major/minor versions of Cloudera Runtime and Cloudera Manager. This major/minor version is the first three digits of the Platform Version displayed along with the Cloudera Manager information:

In the image above, the Runtime major/minor version is 7.2.7.

A Data Hub major/minor version upgrade initiates an upgrade of the major/minor Cloudera Manager and Runtime versions, as well as the required additional parcels (Spark3, Flink, Profiler, and Cloudera Flow Management). The root volumes, additional volumes, and the cluster state is retained. A major/minor version upgrade is an express upgrade where the services will not be available during the process.

A major/minor version upgrade is not an OS upgrade, as the OS/VM packages are not updated and VMs are not replaced. You can initiate an OS upgrade in a separate step. Major/minor upgrades do not update other CDP packages such as Salt.

A major/minor version upgrade does not redistribute the services according to the cluster template of the newer version. Nor does it add new services according to the new cluster template. Major/minor version upgrades do not add new service configurations in the new cluster template. These additional configurations have to be applied manually.

This is not a zero-downtime upgrade, and there will be service outages during the upgrade.

When a major/minor version upgrade is available, you will be able to select the target version for upgrade from the Upgrade tab at the bottom of the Data Hub details page:

For supported versions and templates, see Support matrix for major/minor upgrades.

Maintenance upgrades

The maintenance or “hotfix” upgrade process checks to see if a new Cloudera Runtime or Cloudera Manager hotfix is available, and then upgrades the Data Hub to the newest builds. Maintenance/hotfix upgrades do not upgrade to a new major/minor version of Runtime and CM; they only upgrade to the latest build of a hotfix version.

A maintenance upgrade can be conducted on a single Data Hub cluster in an environment, or multiple Data Hub clusters. You can perform a maintenance upgrade independent of a Data Lake upgrade.

When a maintenance upgrade is available, you will be able to select the target version (which is the same as the current version) for upgrade from the Upgrade tab at the bottom of the Data Hub details page. When you select the target version, note that the CM/CDP versions are the same, but the build numbers differ:

Maintenance upgrades are available from Runtime 7.2.7 onward for RAZ and non-RAZ Data Hubs.

Draft comment:
OS upgrade section commented out below.

OS Upgrades

An OS upgrade updates the OS and VM packages to those available in the latest pre-warmed image. This is done by replacing the VM, re-attaching the attached volumes, and restarting the services. In the process, the data on root volume (for example, parcels and service logs) is lost. On larger clusters, test the OS upgrade in a development environment as the upgrade may exceed some of the cloud resource limits.

An OS upgrade does not upgrade the platform version (CM, Runtime, and additional parcels). You can perform an OS upgrade independent of a Data Hub upgrade.

If an OS upgrade is available, it will appear in the Upgrade Data Hub menu with “(OS Upgrade Only)” when you select a Runtime version:

Limitations

  • This is not a zero-downtime upgrade and service outages will occur.
  • Ranger Authorization: Data Hub maintenance upgrades with RAZ are supported only for Runtime versions 7.2.7+. Major/minor version upgrades with RAZ are supported only for Runtime versions 7.2.10-7.2.12 to versions 7.2.14+.
  • During an OS upgrade, any data on the root volume (parcels, service logs, custom software) will be lost.
  • Cloudera Operational Database cannot be upgraded through the Data Hub user interface and must be upgraded through the CDP beta CLI. For more information see Upgrading Cloudera Operational Database.

Prerequisites

  • There is required downtime of the environment during upgrades, so plan the upgrade accordingly.
  • Verify using the Support matrix for Data Hub upgrades that all Data Hub clusters in the environment are a type and version supported for major/minor upgrades.
  • Test your applications against the new platform version in a separate environment before the Runtime/CM upgrade to ensure application compatibility with the new platform version.
  • The Data Lake and all the Data Hubs in an environment must be upgraded to the same major/minor version. If you plan to upgrade your Data Hub clusters to a later major/minor version, you must first backup and then upgrade the Data Lake to that version.
  • Verify that any Experiences you use are running the latest version available.