1. Preparing for Foreman upgrade

Review the following prerequisites and available upgrade paths before upgrading your current Foreman installation to Foreman nightly.

1.1. Upgrade path overview

You can upgrade from Foreman 3.15 to Foreman nightly. The upgrade process includes the following high-level steps:

  1. Ensuring that your Foreman servers and Smart Proxy servers are running on Foreman 3.15.

  2. Upgrading your Foreman server:

    1. Upgrading your Foreman server to nightly.

    2. Synchronizing the new nightly repositories.

  3. Upgrading your Smart Proxy servers to nightly.

You can upgrade your Smart Proxies separately over multiple maintenance windows because versions 3.15 and 3.14 remain compatible with your upgraded Foreman server nightly. Upgrading Smart Proxies separately can be useful in the following situations:

  • If you want to have several smaller outage windows instead of one larger window.

  • If Smart Proxies in your organization are managed by several teams and are located in different locations.

  • If you use a load-balanced configuration, you can upgrade one load-balanced Smart Proxy and keep other load-balanced Smart Proxies at one version lower. This allows you to upgrade all Smart Proxies one after another without any outage.

Smart Proxies at version 3.15 and 3.14 retain all of their previous functionality. New functionality added in the nightly release is available only after you upgrade your Smart Proxies to nightly.

1.2. Planning Foreman upgrade

Upgrading to Foreman nightly affects your entire Foreman infrastructure. Plan carefully before proceeding.

  • Read the Foreman nightly Release notes.

  • Consider whether any of your integrations need updating. Some Foreman API endpoints, Hammer CLI commands, and modules from the Foreman Ansible Collection can differ between versions of Foreman.

Foreman services are shut down during the upgrade. Ensure to plan for the required downtime. The upgrade process duration varies depending on your hardware configuration, network speed, and the amount of data that is stored on the server:

  • On average installations, upgrading Foreman server takes up to 30 minutes and upgrading a single Smart Proxy server takes up to 10 minutes.

  • On very large installations, upgrading Foreman server can take up to 1 – 2 hours and upgrading a single Smart Proxy server can take up to 15 – 30 minutes.

1.3. Following the progress of the upgrade

Because of the lengthy upgrade time, use a utility such as tmux to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously.

If you lose connection to the command shell where the upgrade command is running, you can see the logs in /var/log/foreman-installer/katello.log to check if the process completed successfully.

Additional resources

  • tmux(1) man page on your system

2. Upgrading Foreman

Upgrading Foreman includes upgrading your Foreman server, synchronizing the required repositories, and upgrading Smart Proxy servers.

2.1. Upgrading your Foreman server

Upgrading a Foreman server with access to the public internet includes backing up the server, ensuring that the system and Foreman configuration is up-to-date, and running the upgrade.

Warning
 If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating. You can use the --noop option with the foreman-installer to test for changes.
Prerequisites

  • If you have made manual edits to you DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, ensure DNS and DHCP configuration management is disabled:

    # foreman-installer \
--foreman-proxy-dhcp-managed false \
--foreman-proxy-dns-managed false

    This prevents the upgrade process from overwriting your DNS and DHCP configuration.

Procedure

  1. Create a backup of your Foreman server.

    • If your Foreman server runs on a virtual machine:

      1. Stop all Foreman services:

        # foreman-maintain service stop

      2. Take a snapshot.

      3. Start all Foreman services:

        # foreman-maintain service start

    • If your Foreman server runs on a physical machine, create a backup as described in Backing up Foreman server and Smart Proxy server in Administering Foreman.

  2. Update repositories:

    # dnf upgrade https://yum.theforeman.org/releases/nightly/el9/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/nightly/katello/el9/x86_64/katello-repos-latest.rpm

  3. Check for running tasks:

    # foreman-rake katello:upgrade_check

  4. Stop all Foreman services:

    # foreman-maintain service stop

  5. Update packages:

    # dnf upgrade

  6. Run the Foreman installer:

    # foreman-installer

  7. Determine if the system needs a reboot:

    # dnf needs-restarting --reboothint

  8. If the previous command told you to reboot, then reboot the system:

    # reboot

2.2. Performing post-upgrade tasks

  • If you cloned the default templates in Foreman to create custom templates, verify whether the default templates changed during the upgrade. If the default templates have changed, consider updating your custom templates to reflect these changes.

  • Optional: If the default provisioning templates have been changed during the upgrade, recreate any templates cloned from the default templates. If the custom code is executed before and/or after the provisioning process, use custom provisioning snippets to avoid recreating cloned templates. For more information about configuring custom provisioning snippets, see Creating Custom Provisioning Snippets in Provisioning hosts.

  • If your Foreman shows any discovered hosts, reboot them:

    1. In the Foreman web UI, navigate to Hosts > Discovered hosts.

    2. Select Any Organization to display all discovered hosts.

    3. Reboot all discovered hosts.

  • Pulp is introducing more data about container manifests to the API. This information allows Katello to display manifest labels, annotations, and information about the manifest type, such as if it is bootable or represents flatpak content. As a result, migrations must be performed to pull this content from manifests into the database.

This migration takes time, so if you depend on container content and need minimal upgrade downtime, use this procedure to migrate data.

Procedure

  1. Enter the following command in a tmux window on Foreman server for a pre-migration. This command migrates data while Foreman is running without any need for downtime and reduces future upgrade downtime:

    # foreman-maintain advanced procedure run pulpcore-container-handle-image-metadata

  2. If the manifest represents bootable or flatpak content, allow the container image API to display manifest labels, annotations by entering the following command:

    # foreman-rake katello:import_container_manifest_labels

2.3. Upgrading Smart Proxy servers

After you have upgraded your Foreman server and synchronized the required repositories, you can start upgrading your Smart Proxy servers.

Prerequisites

  • If you use content views to control updates to the Smart Proxy server base operating system or repository, update those content views with new repositories, then publish and promote their updated versions. For more information, see Managing content views in Managing content.

  • Ensure the base system of the Smart Proxy is registered to the newly upgraded Foreman server.

  • If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

    Warning

    Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
Upgrading Smart Proxy servers

  1. Create a backup.

  2. Update repositories:

    # dnf upgrade https://yum.theforeman.org/releases/nightly/el9/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/nightly/katello/el9/x86_64/katello-repos-latest.rpm

  3. Stop Smart Proxy services:

    # foreman-maintain service stop

  4. Update the required packages:

    # dnf upgrade

  5. Run the installer:

    # foreman-installer

  6. Determine if the system needs a reboot:

    # dnf needs-restarting --reboothint

  7. If the previous command told you to reboot, then reboot the system:

    # reboot

  8. Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.

Upgrading Smart Proxy servers using remote execution

  1. Create a backup or take a snapshot.

    For more information on backups, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.

  2. In the Foreman web UI, navigate to Monitor > Jobs.

  3. Click Run Job.

  4. From the Job category list, select Maintenance Operations.

  5. From the Job template list, select Smart Proxy Upgrade Playbook.

  6. In the Search Query field, enter the host name of the Smart Proxy.

  7. Ensure that Apply to 1 host is displayed in the Resolves to field.

  8. In the target_version field, enter the target version of the Smart Proxy.

  9. In the whitelist_options field, enter the options.

  10. Select the schedule for the job execution in Schedule.

  11. In the Type of query section, click Static Query.

2.4. Upgrading the external database operating system

If your Foreman uses an external database, you can upgrade the database from Enterprise Linux 8 to Enterprise Linux 9 while upgrading Foreman from 3.15 to nightly.

Prerequisites

  • Create a host running Enterprise Linux 9 for PostgreSQL server that follows the external database on Enterprise Linux 9 documentation. For more information, see Using external databases with Foreman.

Procedure

  1. Create a backup of your existing external database.

  2. Restore the backup on the new Enterprise Linux 9 server.

  3. Verify that Foreman can reach the new database:

    # PGPASSWORD='My_Foreman_Database_Password' psql -h postgres.example.com -p 5432 -U foreman -d foreman -c "SELECT 1 as ping"

  4. If your Foreman server can reach the new database server by using the old name, no further changes are required. Otherwise, reconfigure Foreman to use the new name:

    # foreman-installer \
--foreman-db-host newpostgres.example.com \
--katello-candlepin-db-host newpostgres.example.com \
--foreman-proxy-content-pulpcore-postgresql-host newpostgres.example.com

Appendix A: Troubleshooting permission issues

Foreman upgrades perform pre-upgrade checks. If the pre-upgrade check discovers permission issues, it fails with an error similar to the following one:

2024-01-29T20:50:09 [W|app|] Could not create role 'Ansible Roles Manager': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found:

If you see an error like this on your Foreman server, identify and remedy the permission issues.

Procedure

  1. On your Foreman server, identify permission issues:

    # foreman-maintain health check --label duplicate_permissions

  2. Fix permission issues:

    # foreman-rake db:seed
Verification

  • Rerun the check to ensure no permission issues remain:

    # foreman-maintain health check --label duplicate_permissions