In this guide, the terms upgrade, update, and migrate have the following meanings:

Upgrading

The process of advancing your Foreman server and Smart Proxy server installations from a y-stream release to the next, for example Foreman 3.3 to Foreman 3.4. For more information, see Upgrading Overview.

Updating

The process of advancing your Foreman server and Smart Proxy server installations from a z-stream release to the next, for example Foreman 3.4.0 to Foreman 3.4.1.

Migrating

The process of moving an existing Foreman installation to a new instance.

1. Upgrading Overview

Review prerequisites and available upgrade paths below before upgrading your current Foreman installation to Foreman 3.4.

Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

1.1. Prerequisites

Upgrading to Foreman 3.4 affects your entire Foreman infrastructure. Before proceeding, complete the following:

  • Read the Foreman 3.4 Release Notes.

  • Plan your upgrade path. For more information, see Upgrade Paths.

  • Plan for the required downtime. Foreman services are shut down during the upgrade. The upgrade process duration might vary depending on your hardware configuration, network speed, and the amount of data that is stored on the server.

    Upgrading Foreman takes approximately 1 – 2 hours.

    Upgrading Smart Proxy takes approximately 10 – 30 minutes.

  • Ensure that you have sufficient storage space on your server. For more information, see Preparing your Environment for Installation in Installing Foreman 3.4 Server with Katello 4.6 Plugin on RHEL/CentOS and Preparing your Environment for Installation in Installing an External Smart Proxy Server 3.4.

  • Back up your Foreman server and all Smart Proxy servers. For more information, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.

  • Plan for updating any scripts you use that contain Foreman API commands because some API commands differ between versions of Foreman.

Warning
 If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the installation script runs during upgrading or updating. You can use the --noop option with the foreman-installer script to test for changes.

1.2. Upgrade Paths

You can upgrade to Foreman 3.4 from Foreman 3.3.

Foreman servers and Smart Proxy servers on earlier versions must first be upgraded to Foreman 3.3. For more information, see the Foreman 3.3 Upgrade documentation.

High-Level Upgrade Steps

The high-level steps in upgrading Foreman to 3.4 are as follows:

  1. Upgrade Foreman server to 3.4. For more information, see Upgrading Foreman server.

  2. Upgrade all Smart Proxy servers to 3.4. For more information, see Upgrading Smart Proxy Servers.

  3. Upgrade to https://yum.theforeman.org/client/3.4/ on all content hosts. For more information, see Upgrading Content Hosts.

  4. Continue with Performing Post-Upgrade Tasks.

1.3. Following the Progress of the Upgrade

Because of the lengthy upgrade time, use a utility such as tmux to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information, see the tmux manual page.

If you lose connection to the command shell where the upgrade command is running you can see the logs in /var/log/foreman-installer/katello.log to check if the process completed successfully.

1.4. Upgrading Smart Proxies Separately from Foreman

You can upgrade Foreman to version 3.4 and keep Smart Proxies at version 3.3 until you have the capacity to upgrade them too.

All the functionality that worked previously works on 3.3 Smart Proxies. However, the functionality added in the 3.4 release will not work until you upgrade Smart Proxies to 3.4.

Upgrading Smart Proxies after upgrading Foreman can be useful in the following example scenarios:

  1. If you want to have several smaller outage windows instead of one larger window.

  2. If Smart Proxies in your organization are managed by several teams and are located in different locations.

  3. If you use a load-balanced configuration, you can upgrade one load-balanced Smart Proxy and keep other load-balanced Smart Proxies at one version lower. This allows you to upgrade all Smart Proxies one after another without any outage.

2. Upgrading Foreman

Use the following procedures to upgrade your existing Foreman to Foreman 3.4:

  1. Review Prerequisites.

  2. Upgrading Foreman server

  3. Upgrading Smart Proxy Servers

  4. Upgrading Content Hosts

  5. Performing Post-Upgrade Tasks

2.1. Upgrading Foreman server

This section describes how to upgrade Foreman server from 3.3 to 3.4. You can upgrade from any minor version of Foreman server 3.3.

Before You Begin

  • Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

  • Review and update your firewall configuration prior to upgrading your Foreman server. For more information, see Preparing your environment for installation in Installing Foreman 3.4 Server with Katello 4.6 Plugin on RHEL/CentOS.

  • Ensure that you do not delete the manifest from the Customer Portal or in the Foreman web UI because this removes all the entitlements of your content hosts.

  • If you have edited any of the default templates, back up the files either by cloning or exporting them. Cloning is the recommended method because that prevents them being overwritten in future updates or upgrades. To confirm if a template has been edited, you can view its History before you upgrade or view the changes in the audit log after an upgrade. In the Foreman web UI, navigate to Monitor > Audits and search for the template to see a record of changes made. If you use the export method, restore your changes by comparing the exported template and the default template, manually applying your changes.

Smart Proxy Considerations

  • If you use Content Views to control updates to a Smart Proxy server’s base operating system, or for Smart Proxy server repository, you must publish updated versions of those Content Views.

  • Note that Foreman server upgraded from 3.3 to 3.4 can use Smart Proxy servers still at 3.3.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
FIPS mode

You cannot upgrade Foreman server from a RHEL base system that is not operating in FIPS mode to a RHEL base system that is operating in FIPS mode.

To run Foreman server on a Red Hat Enterprise Linux base system operating in FIPS mode, you must install Foreman on a freshly provisioned RHEL base system operating in FIPS mode. For more information, see Preparing your environment for installation in Installing Foreman 3.4 Server with Katello 4.6 Plugin on RHEL/CentOS.

2.1.1. Upgrading a Connected Foreman server

Use this procedure for a Foreman server with access to the public internet

Warning
 If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the installation script runs during upgrading or updating. You can use the --noop option with the foreman-installer script to test for changes.
Upgrade Foreman server

  1. Stop all Foreman services:

    # foreman-maintain service stop

  2. Take a snapshot or create a backup:

    • On a virtual machine, take a snapshot.

    • On a physical machine, create a backup.

  3. Start all Foreman services:

    # foreman-maintain service start

  4. Optional: If you made manual edits to DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required.

  5. Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:

    # foreman-installer --foreman-proxy-dns-managed=false \
--foreman-proxy-dhcp-managed=false

  6. Optional: If you use PostgreSQL as an external database, install the postgresql-evr package on the PostgreSQL server:

    # dnf install postgresql-evr

  7. In the Foreman web UI, navigate to Hosts > Discovered hosts. On the Discovered Hosts page, power off and then delete the discovered hosts. From the Select an Organization menu, select each organization in turn and repeat the process to power off and delete the discovered hosts. Make a note to reboot these hosts when the upgrade is complete.

  8. Check for running tasks

    # foreman-rake katello:upgrade_check

  9. Update operating system packages:

    # dnf update

  10. Update repositories

    # dnf update https://yum.theforeman.org/releases/3.4/el8/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.6/katello/el8/x86_64/katello-repos-latest.rpm

  11. Ensure the module streams are enabled:

    # dnf module enable katello:el8 pulpcore:el8

  12. Stop all services:

    # foreman-maintain service stop

  13. Update the required packages:

    # dnf update

  14. Run the installer:

    # foreman-installer

  15. Determine if the system needs a reboot:

    # dnf needs-restarting --reboothint

  16. Optional: If the previous command told you to reboot, then reboot the system:

    # reboot

2.2. Upgrading Smart Proxy Servers

This section describes how to upgrade Smart Proxy servers from 3.3 to 3.4.

Before You Begin

  • You must upgrade Foreman server before you can upgrade any Smart Proxy servers. Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

  • If you use Content Views to control updates to the base operating system of Smart Proxy server, update those Content Views with new repositories, publish, and promote their updated versions. For more information, see Managing Content Views in Managing Content.

  • Ensure the Smart Proxy’s base system is registered to the newly upgraded Foreman server.

  • Ensure the Smart Proxy has the correct organization and location settings in the newly upgraded Foreman server.

  • Review and update your firewall configuration prior to upgrading your Smart Proxy server. For more information, see Preparing Your Environment for Smart Proxy Installation in Installing an External Smart Proxy Server 3.4.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
Upgrading Smart Proxy Servers

  1. Create a backup.

  2. Update repositories

    # dnf update https://yum.theforeman.org/releases/3.4/el8/x86_64/foreman-release.rpm \
https://yum.theforeman.org/katello/4.6/katello/el8/x86_64/katello-repos-latest.rpm

  3. Ensure the module streams are enabled:

    # dnf module enable katello:el8 pulpcore:el8

  4. Update the required packages:

    # dnf update

  5. Run the installer:

    # foreman-installer

  6. Determine if the system needs a reboot:

    # dnf needs-restarting --reboothint

  7. Optional: If the previous command told you to reboot, then reboot the system:

    # reboot

  8. Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.

2.3. Upgrading Content Hosts

The https://yum.theforeman.org/client/3.4/ repository provides katello-agent and katello-host-tools, which provide communication services for managing Errata.

Note
 The Katello agent is deprecated and will be removed in a future Foreman version. Migrate your workloads to use the remote execution feature to update clients remotely. For more information, see Migrating from Katello Agent to Remote Execution in Managing Hosts. Foreman now supports a pull-based provider as a replacement for the Katello agent.

For deployments using katello-agent and goferd, update all clients to the new version of katello-agent. For deployments not using katello-agent and goferd, update all clients to the new version of katello-host-tools. Complete this action as soon as possible so that your clients are fully compatible with Foreman server.

Prerequisites

  • You must have upgraded Foreman server.

  • You must have enabled the new https://yum.theforeman.org/client/3.4/ repositories on the Foreman.

  • You must have synchronized the new repositories in the Foreman.

  • If you have not previously installed katello-agent on your clients and you want to install it, use the manual method. For more information, see CLI Procedure.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.
CLI Procedure

  1. Log into the client system.

  2. Enable the https://yum.theforeman.org/client/3.4/ repository for this version of Foreman.

    # yum install https://yum.theforeman.org/client/3.4/el7/x86_64/foreman-client-release.rpm

  3. Depending on your configuration, complete one of the following steps:

    • If your deployment uses katello-agent and goferd, enter the following command to install or upgrade katello-agent:

      # yum install katello-agent

    • If your deployment does not use katello-agent and goferd, enter the following command to install or upgrade katello-host-tools:

      # yum install katello-host-tools

2.4. Upgrading the External Database

You can upgrade an external database from Enterprise Linux 7 to Enterprise Linux 8 while upgrading Foreman from 3.3 to 3.4.

Prerequisites

  • Create a new Enterprise Linux 8 based host for PostgreSQL server that follows the external database on Enterprise Linux 8 documentation. For more information, see Using External Databases with Foreman.

Procedure

  1. Create a backup.

  2. Restore the backup on the new server.

  3. If Foreman reaches the new database server via the old name, no further changes are required. Otherwise reconfigure Foreman to use the new name:

    # foreman-installer \
--foreman-db-host newpostgres.example.com \
--katello-candlepin-db-host newpostgres.example.com \
--foreman-proxy-content-pulpcore-postgresql-host newpostgres.example.com

2.5. Performing Post-Upgrade Tasks

Some of the procedures in this section are optional. You can choose to perform only those procedures that are relevant to your installation.

2.5.1. Upgrading Discovery

If you use the PXE-based discovery process, then you must complete the discovery upgrade procedure on Foreman and on any Smart Proxy server with hosts that you want to be listed in Foreman on the Hosts > Discovered hosts page.

This section describes updating the PXELinux template and the boot image passed to hosts that use PXE booting to register themselves with Foreman server.

From Foreman 3.4, provisioning templates now have a separate association with a subnet, and do not default to using the TFTP Smart Proxy for that subnet. If you create subnets after the upgrade, you must specifically enable the Foreman or a Smart Proxy to provide a proxy service for discovery templates and then configure all subnets with discovered hosts to use a specific template Smart Proxy.

During the upgrade, for every subnet with a TFTP proxy enabled, the template Smart Proxy is set to be the same as the TFTP Smart Proxy. After the upgrade, check all subnets to verify this was set correctly.

These procedures are not required if you do not use PXE booting of hosts to enable Foreman to discover new hosts.

Upgrading Discovery on Foreman server

  1. Update the Discovery template in the Foreman web UI:

    1. In the Foreman web UI, navigate to Hosts > Provisioning templates.

    2. On the PXELinux global default line, click Clone.

    3. Enter a new name for the template in the Name field, for example ACME PXE global default.

    4. In the template editor field, change the line ONTIMEOUT local to ONTIMEOUT discovery and click Submit.

    5. In the Foreman web UI, navigate to Administer > Settings.

    6. On the Provisioning tab, set Default PXE global template entry to a custom value for your environment.

    7. Locate Global default PXELinux template and click on its Value.

    8. Select the name of the newly created template from the menu and click Submit.

    9. In the Foreman web UI, navigate to Hosts > Provisioning templates.

    10. Click Build PXE Default, then click OK.

    Note

    If the template is modified, a Foreman upgrade overrides it to its default version. Once the PXE Default configuration is built, the template configured in the Settings is deployed to the TFTP. This can result in deploying the default template if the new template is correctly set in the Settings.

  2. In the Foreman web UI, go to Configure > Discovery Rules and associate selected organizations and locations with discovery rules.

Verifying Subnets have a Template Smart Proxy
Ensure all subnets with discovered hosts have a template Smart Proxy:

  1. In the Foreman web UI, navigate to Infrastructure > Subnets.

  2. Select the subnet you want to check.

  3. On the Smart Proxies tab, ensure a Template Smart Proxy has been set for this subnet.

For more information about configuring subnets with template Smart Proxies, see Configuring the Discovery Service in Provisioning Hosts.

2.5.2. Upgrading virt-who

If virt-who is installed on Foreman server or a Smart Proxy server, it will be upgraded when they are upgraded. No further action is required. If virt-who is installed elsewhere, it must be upgraded manually.

Before You Begin

If virt-who is installed on a host registered to Foreman server or a Smart Proxy server, first upgrade the host to the latest packages available in the https://yum.theforeman.org/client/3.4/ repository. For information about upgrading hosts, see Upgrading Content Hosts.

Upgrade virt-who Manually

  1. Upgrade virt-who.

    # yum upgrade virt-who

  2. Restart the virt-who service so the new version is activated.

    # systemctl restart virt-who

2.5.3. Migrating Ansible Content

The upgrade from Enterprise Linux 7 to Enterprise Linux 8 includes an upgrade from Ansible Engine 2.9 to Ansible Core 2.12.

If you have custom Ansible content such as playbooks, job templates inside REX, roles and collections on disk, and you rely on modules being delivered by the Ansible RPM on Foreman, you have to take additional steps to adapt your Ansible installation or migrate your Ansible content.

Ansible Core contains only essential modules. In terms of FQCN notation namespace.collection.module, you can continue to use ansible.builtin.*, but everything else is missing in Ansible Core. That means you will be no longer able to use non-builtin Ansible modules as you were used to and you have to get them from another source, eventually.

You have the following options to handle your Ansible content after the upgrade:

  • You can obtain additional community-maintained collections that provide the non-essential functionality from Ansible Galaxy. For more information, see Installing collections in the Galaxy User Guide.

  • You can rewrite your Ansible roles, templates and other affected content.

Note

If you want to download and install Ansible content on Smart Proxy that does not have a connection to an external Ansible Galaxy server, then you must pass the content through Foreman server instead of using the URL of the Ansible Galaxy server in the configuration on the Smart Proxy directly:

  1. Sync the content from a Ansible Galaxy server to a custom repository on your Foreman server.

  2. Configure Ansible on your Smart Proxy to download the content from Foreman server.

To find out what content is affected after the upgrade, you can run:

# ansible-lint --enable-list only-builtins -p path/to/your/playbook/play.yml

You need ansible-lint 6.1.0 or newer. You can install it from pypi.org.

2.5.4. Reclaiming PostgreSQL Space

The PostgreSQL database can use a large amount of disk space especially in heavily loaded deployments. Use this procedure to reclaim some of this disk space on Foreman.

Procedure

  1. Stop all services, except for the postgresql service:

    # foreman-maintain service stop --exclude postgresql

  2. Switch to the postgres user and reclaim space on the database:

    # su - postgres -c 'vacuumdb --full --all'

  3. Start the other services when the vacuum completes:

    # foreman-maintain service start

2.5.5. Tuning Foreman server with Predefined Profiles

If your Foreman deployment includes more than 5000 hosts, you can use predefined tuning profiles to improve performance of Foreman.

Note that you cannot use tuning profiles on Smart Proxies.

You can choose one of the profiles depending on the number of hosts your Foreman manages and available hardware resources.

The tuning profiles are available in the /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes directory.

When you run the foreman-installer command with the --tuning option, deployment configuration settings are applied to Foreman in the following order:

  1. The default tuning profile defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml file

  2. The tuning profile that you want to apply to your deployment and is defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/ directory

  3. Optional: If you have configured a /etc/foreman-installer/custom-hiera.yaml file, Foreman applies these configuration settings.

Note that the configuration settings that are defined in the /etc/foreman-installer/custom-hiera.yaml file override the configuration settings that are defined in the tuning profiles.

Therefore, before applying a tuning profile, you must compare the configuration settings that are defined in the default tuning profile in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml, the tuning profile that you want to apply and your /etc/foreman-installer/custom-hiera.yaml file, and remove any duplicated configuration from the /etc/foreman-installer/custom-hiera.yaml file.

default

Number of managed hosts: 0 – 5000

RAM: 20G

Number of CPU cores: 4

medium

Number of managed hosts: 5001 – 10000

RAM: 32G

Number of CPU cores: 8

large

Number of managed hosts: 10001 – 20000

RAM: 64G

Number of CPU cores: 16

extra-large

Number of managed hosts: 20001 – 60000

RAM: 128G

Number of CPU cores: 32

extra-extra-large

Number of managed hosts: 60000+

RAM: 256G

Number of CPU cores: 48+

Procedure

  1. Optional: If you have configured the custom-hiera.yaml file on Foreman server, back up the /etc/foreman-installer/custom-hiera.yaml file to custom-hiera.original. You can use the backup file to restore the /etc/foreman-installer/custom-hiera.yaml file to its original state if it becomes corrupted:

    # cp /etc/foreman-installer/custom-hiera.yaml \
/etc/foreman-installer/custom-hiera.original

  2. Optional: If you have configured the custom-hiera.yaml file on Foreman server, review the definitions of the default tuning profile in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml and the tuning profile that you want to apply in /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/. Compare the configuration entries against the entries in your /etc/foreman-installer/custom-hiera.yaml file and remove any duplicated configuration settings in your /etc/foreman-installer/custom-hiera.yaml file.

  3. Enter the foreman-installer command with the --tuning option for the profile that you want to apply. For example, to apply the medium tuning profile settings, enter the following command:

    # foreman-installer --tuning medium