1. Upgrading overview
Review prerequisites and available upgrade paths below before upgrading your current Foreman installation to Foreman nightly.
Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies separately from Foreman.
1.1. Prerequisites
Upgrading to Foreman nightly affects your entire Foreman infrastructure. Before proceeding, complete the following:
-
Read the Foreman nightly Release Notes.
-
Plan your upgrade path. For more information, see Upgrade paths.
-
Plan for the required downtime. Foreman services are shut down during the upgrade. The upgrade process duration might vary depending on your hardware configuration, network speed, and the amount of data that is stored on the server.
Upgrading Foreman takes approximately 1 – 2 hours.
Upgrading Smart Proxy takes approximately 10 – 30 minutes.
-
Ensure that you have sufficient storage space on your server. For more information, see Preparing your Environment for Installation in Installing Foreman Server with Katello nightly plugin on Enterprise Linux and Preparing your Environment for Installation in Installing Smart Proxy server.
-
Ensure that you have at least the same amount of free space on
/var/lib/pgsql
as that consumed by/var/lib/pgsql/data
. Upgrading to Foreman nightly involves a PostgreSQL 12 to PostgreSQL 13 upgrade. The contents of/var/lib/pgsql/data
are backed up during the PostgreSQL upgrade. -
Back up your Foreman server and all Smart Proxy servers. For more information, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.
-
Plan for updating any scripts you use that contain Foreman API commands because some API commands differ between versions of Foreman.
Warning
|
If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating.
You can use the --noop option with the foreman-installer to test for changes.
|
1.2. Upgrade paths
You can upgrade to Foreman nightly from Foreman 3.12.
The high-level steps in upgrading Foreman to nightly are as follows:
-
Upgrade Foreman server to nightly. For more information, see Foreman server upgrade considerations.
-
Upgrade all Smart Proxy servers to nightly. For more information, see Upgrading Smart Proxy servers.
1.3. Upgrading Smart Proxies separately from Foreman
You can upgrade Foreman to version nightly and keep Smart Proxies at version 3.12 until you have the capacity to upgrade them too.
All the functionality that worked previously works on 3.12 Smart Proxies. However, the functionality added in the nightly release will not work until you upgrade Smart Proxies to nightly.
Upgrading Smart Proxies after upgrading Foreman can be useful in the following example scenarios:
-
If you want to have several smaller outage windows instead of one larger window.
-
If Smart Proxies in your organization are managed by several teams and are located in different locations.
-
If you use a load-balanced configuration, you can upgrade one load-balanced Smart Proxy and keep other load-balanced Smart Proxies at one version lower. This allows you to upgrade all Smart Proxies one after another without any outage.
1.4. Following the progress of the upgrade
Because of the lengthy upgrade time, use a utility such as tmux
to suspend and reattach a communication session.
You can then check the upgrade progress without staying connected to the command shell continuously.
For more information, see the tmux
manual page.
If you lose connection to the command shell where the upgrade command is running you can see the logs in /var/log/foreman-installer/katello.log
to check if the process completed successfully.
2. Upgrading Foreman
Use the following procedures to upgrade your existing Foreman to Foreman nightly:
2.1. Foreman server upgrade considerations
This section describes how to upgrade Foreman server from 3.12 to nightly. You can upgrade from any minor version of Foreman server 3.12.
-
Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies separately from Foreman.
-
Review and update your firewall configuration prior to upgrading your Foreman server. For more information, see Preparing your environment for installation in Installing Foreman Server with Katello nightly plugin on Enterprise Linux.
-
Ensure that you do not delete the manifest from the Customer Portal or in the Foreman web UI because this removes all the entitlements of your content hosts.
-
If you have edited any of the default templates, back up the files either by cloning or exporting them. Cloning is the recommended method because that prevents them being overwritten in future updates or upgrades. To confirm if a template has been edited, you can view its History before you upgrade or view the changes in the audit log after an upgrade. In the Foreman web UI, navigate to Monitor > Audits and search for the template to see a record of changes made. If you use the export method, restore your changes by comparing the exported template and the default template, manually applying your changes.
-
If you use content views to control updates to a Smart Proxy server’s base operating system, or for Smart Proxy server repository, you must publish updated versions of those content views.
-
Note that Foreman server upgraded from 3.12 to nightly can use Smart Proxy servers still at 3.12.
Warning
|
If you implemented custom certificates, you must retain the content of both the Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed. |
You cannot upgrade Foreman server from a RHEL base system that is not operating in FIPS mode to a RHEL base system that is operating in FIPS mode.
To run Foreman server on a Red Hat Enterprise Linux base system operating in FIPS mode, you must install Foreman on a freshly provisioned RHEL base system operating in FIPS mode. For more information, see Preparing your environment for installation in Installing Foreman Server with Katello nightly plugin on Enterprise Linux.
2.2. Upgrading a connected Foreman server
Use this procedure for a Foreman server with access to the public internet
Warning
|
If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating.
You can use the --noop option with the foreman-installer to test for changes.
|
-
Stop all Foreman services:
# foreman-maintain service stop
-
Take a snapshot or create a backup:
-
On a virtual machine, take a snapshot.
-
On a physical machine, create a backup.
-
-
Start all Foreman services:
# foreman-maintain service start
-
Optional: If you made manual edits to DNS or DHCP configuration in the
/etc/zones.conf
or/etc/dhcp/dhcpd.conf
files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required. -
Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:
# foreman-installer \ --foreman-proxy-dhcp-managed=false \ --foreman-proxy-dns-managed=false
-
In the Foreman web UI, navigate to Hosts > Discovered hosts. On the Discovered Hosts page, power off and then delete the discovered hosts. From the Select an Organization menu, select each organization in turn and repeat the process to power off and delete the discovered hosts. Make a note to reboot these hosts when the upgrade is complete.
-
Check for running tasks
# foreman-rake katello:upgrade_check
-
Update operating system packages:
# dnf upgrade
-
Update repositories:
-
On Enterprise Linux 9:
# dnf upgrade https://yum.theforeman.org/releases/nightly/el9/x86_64/foreman-release.rpm \ https://yum.theforeman.org/katello/nightly/katello/el9/x86_64/katello-repos-latest.rpm
-
On Enterprise Linux 8:
# dnf upgrade https://yum.theforeman.org/releases/nightly/el8/x86_64/foreman-release.rpm \ https://yum.theforeman.org/katello/nightly/katello/el8/x86_64/katello-repos-latest.rpm
-
-
Disable the pulpcore module if it is enabled:
# dnf module disable pulpcore
-
On Enterprise Linux 8: Switch to the PostgreSQL 13 module:
# dnf -y module switch-to postgresql:13
-
If you are using an external database, upgrade your database to PostgreSQL 13.
-
On Enterprise Linux 8: Ensure the module streams are enabled:
# dnf module enable katello:el8
-
Stop all services:
# foreman-maintain service stop
-
Update the required packages:
# dnf upgrade
-
Run the installer:
# foreman-installer
-
Determine if the system needs a reboot:
# dnf needs-restarting --reboothint
-
If the previous command told you to reboot, then reboot the system:
# reboot
2.3. Performing post-upgrade tasks
-
Optional: If the default provisioning templates have been changed during the upgrade, recreate any templates cloned from the default templates. If the custom code is executed before and/or after the provisioning process, use custom provisioning snippets to avoid recreating cloned templates. For more information about configuring custom provisioning snippets, see Creating Custom Provisioning Snippets in Provisioning hosts.
-
Pulp is introducing more data about container manifests to the API. This information allows Katello to display manifest labels, annotations, and information about the manifest type, such as if it is bootable or represents flatpak content. As a result, migrations must be performed to pull this content from manifests into the database.
This migration takes time, so if you depend on container content and need minimal upgrade downtime, use this procedure to migrate data.
-
Enter the following command in a
tmux
window on Foreman server for a pre-migration. This command migrates data while Foreman is running without any need for downtime and reduces future upgrade downtime:# foreman-maintain advanced procedure run pulpcore-container-handle-image-metadata
-
If the manifest represents bootable or flatpak content, allow the container image API to display manifest labels, annotations by entering the following command:
# foreman-rake katello:import_container_manifest_labels
2.4. Upgrading Smart Proxy servers
This section describes how to upgrade Smart Proxy servers from 3.12 to nightly.
-
You must upgrade Foreman server before you can upgrade any Smart Proxy servers. Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies separately from Foreman.
-
If you use content views to control updates to the base operating system of Smart Proxy server, update those content views with new repositories, publish, and promote their updated versions. For more information, see Managing content views in Managing content.
-
Ensure the Smart Proxy’s base system is registered to the newly upgraded Foreman server.
-
Ensure the Smart Proxy has the correct organization and location settings in the newly upgraded Foreman server.
-
Review and update your firewall configuration prior to upgrading your Smart Proxy server. For more information, see Preparing Your Environment for Smart Proxy Installation in Installing a Smart Proxy Server nightly on Enterprise Linux.
Warning
|
If you implemented custom certificates, you must retain the content of both the Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed. |
-
Create a backup.
-
On a virtual machine, take a snapshot.
-
On a physical machine, create a backup.
For information on backups, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.
-
-
Update repositories:
-
On Enterprise Linux 9:
# dnf upgrade https://yum.theforeman.org/releases/nightly/el9/x86_64/foreman-release.rpm \ https://yum.theforeman.org/katello/nightly/katello/el9/x86_64/katello-repos-latest.rpm
-
On Enterprise Linux 8:
# dnf upgrade https://yum.theforeman.org/releases/nightly/el8/x86_64/foreman-release.rpm \ https://yum.theforeman.org/katello/nightly/katello/el8/x86_64/katello-repos-latest.rpm
-
-
Disable the pulpcore module if it is enabled:
# dnf module disable pulpcore
-
On Enterprise Linux 8: Switch to the PostgreSQL 13 module:
# dnf -y module switch-to postgresql:13
-
On Enterprise Linux 8: Ensure the module streams are enabled:
# dnf module enable katello:el8
-
Update the required packages:
# dnf upgrade
-
Run the installer:
# foreman-installer
-
Determine if the system needs a reboot:
# dnf needs-restarting --reboothint
-
If the previous command told you to reboot, then reboot the system:
# reboot
-
Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.
-
Create a backup or take a snapshot.
For more information on backups, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.
-
In the Foreman web UI, navigate to Monitor > Jobs.
-
Click Run Job.
-
From the Job category list, select Maintenance Operations.
-
From the Job template list, select Smart Proxy Upgrade Playbook.
-
In the Search Query field, enter the host name of the Smart Proxy.
-
Ensure that Apply to 1 host is displayed in the Resolves to field.
-
In the target_version field, enter the target version of the Smart Proxy.
-
In the whitelist_options field, enter the options.
-
Select the schedule for the job execution in Schedule.
-
In the Type of query section, click Static Query.
2.5. Upgrading the external database
You can upgrade an external database from Enterprise Linux 8 to Enterprise Linux 9 while upgrading Foreman from 3.12 to nightly.
-
Create a new Enterprise Linux 9 based host for PostgreSQL server that follows the external database on Enterprise Linux 9 documentation. For more information, see Using External Databases with Foreman.
-
Install PostgreSQL version 13 on the new Enterprise Linux host.
-
Create a backup.
-
Restore the backup on the new server.
-
If Foreman reaches the new database server via the old name, no further changes are required. Otherwise reconfigure Foreman to use the new name:
# foreman-installer \ --foreman-db-host newpostgres.example.com \ --katello-candlepin-db-host newpostgres.example.com \ --foreman-proxy-content-pulpcore-postgresql-host newpostgres.example.com
3. Upgrading Enterprise Linux on Foreman or Smart Proxy
Foreman and Smart Proxy are supported on both Enterprise Linux 8 and Enterprise Linux 9. You can upgrade your Foreman and Smart Proxy operating system from Enterprise Linux 8 to Enterprise Linux 9.
There are two methods to upgrade your Foreman or Smart Proxy from Enterprise Linux 8 to Enterprise Linux 9:
-
Leapp in-place upgrade
-
Migration by using backup and restore
-
Migration by using cloning
With Leapp, you can upgrade your Foreman or Smart Proxy in place therefore it is faster but imposes a downtime on the services. With migration, you can move your Foreman or Smart Proxy to a fresh Enterprise Linux 9 system. The Enterprise Linux 8 system remains operational during the migration, which reduces the downtime.
3.1. Upgrading Foreman or Smart Proxy to EL 9 in-place by using Leapp
You can use the Leapp tool to upgrade as well as to help detect and resolve issues that could prevent you from upgrading successfully.
-
Review upgrade warnings before you begin an upgrade. For more information, see Release notes.
-
Foreman nightly or Smart Proxy nightly running on Enterprise Linux 8.
-
Access to available repositories or a local mirror of repositories.
-
Enable the @theforeman/leapp COPR Repository, which contains Leapp packages with patches to support Foreman or Smart Proxy upgrades:
# dnf copr enable @theforeman/leapp
-
Install required packages:
# dnf install leapp leapp-upgrade-el8toel9
-
Install additional OS specific packages (
leapp-data-almalinux
for AlmaLinux,leapp-data-centos
for CentOS Stream, orleapp-data-rocky
for Rocky Linux). SeeID
in/etc/os-release
for your operating system code.# dnf install leapp-data-$ID
Note that this is not required for Red Hat Enterprise Linux installations.
-
Add Foreman specific repositories to
/etc/leapp/files/leapp_upgrade_repositories.repo
:[leapp-foreman] name=Foreman nightly baseurl=https://yum.theforeman.org/releases/nightly/el9/$basearch gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=1 gpgcheck=1 [leapp-foreman-plugins] name=Foreman plugins nightly baseurl=https://yum.theforeman.org/plugins/nightly/el9/$basearch enabled=1 gpgcheck=0 [leapp-katello] name=Katello nightly baseurl=https://yum.theforeman.org/katello/nightly/katello/el9/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman enabled=1 gpgcheck=1 [leapp-candlepin] name=Candlepin: an open source entitlement management system. baseurl=https://yum.theforeman.org/candlepin/4.4/el9/$basearch/ gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-candlepin enabled=1 gpgcheck=1 [leapp-pulpcore] name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages. baseurl=https://yum.theforeman.org/pulpcore/3.49/el9/$basearch/ gpgkey=https://yum.theforeman.org/pulpcore/3.49/GPG-RPM-KEY-pulpcore enabled=1 gpgcheck=1
-
Append Yum repositories for Puppet to
/etc/leapp/files/leapp_upgrade_repositories.repo
.-
For Puppet 8:
[leapp-puppet8] name=Puppet 8 Repository el 9 - $basearch baseurl=http://yum.puppetlabs.com/puppet8/el/9/$basearch gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-2025-04-06-puppet8-release enabled=1 gpgcheck=1
-
For Puppet 7:
[leapp-puppet7] name=Puppet 7 Repository el 9 - $basearch baseurl=http://yum.puppetlabs.com/puppet7/el/9/$basearch gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-2025-04-06-puppet7-release enabled=1 gpgcheck=1
-
-
Let Leapp analyze your system:
# leapp preupgrade
The first run will most likely report issues and inhibit the upgrade. Examine the report in the
/var/log/leapp/leapp-report.txt
file, answer all questions by usingleapp answer
, and manually resolve other reported problems. -
Run
leapp preupgrade
again and make sure that it does not report any more issues. -
Let Leapp create the upgrade environment:
# leapp upgrade
-
Reboot the system to start the upgrade.
After the system reboots, a live system conducts the upgrade, reboots to fix SELinux labels and then reboots into the final Enterprise Linux 9 system.
-
Wait for Leapp to finish the upgrade. You can monitor the process with
journalctl
:# journalctl -u leapp_resume -f
-
Verify the post-upgrade state. For more information, see Verifying the post-upgrade state in Upgrading from RHEL 8 to RHEL 9.
-
Perform post-upgrade tasks on the RHEL 9 system. For more information, see Performing post-upgrade tasks on the RHEL 9 system in Upgrading from RHEL 8 to RHEL 9.
-
Change SELinux to enforcing mode. For more information, see Changing SELinux mode to enforcing in Upgrading from RHEL 8 to RHEL 9.
-
For Enterprise Linux installations, unset the
subscription-manager
release:# subscription-manager release --unset
-
For more information on customizing the Leapp upgrade for your environment, see Customizing your Red Hat Enterprise Linux in-place upgrade.
3.2. Migrating Foreman or Smart Proxy to EL 9
You can migrate your existing Foreman server and Smart Proxy server from Enterprise Linux 8 to a freshly installed Enterprise Linux 9 system. The migration involves creating a backup of the existing Foreman server and Smart Proxy server, which you then restore on the new Enterprise Linux 9 system.
-
Perform a full backup of your Foreman server or Smart Proxy. This is the source Enterprise Linux 8 server that you are migrating. For more information, see Performing a full backup of Foreman server or Smart Proxy server in Administering Foreman.
-
Deploy a system with Enterprise Linux 9 and the same hostname and configuration as the source server. This is the target server.
-
Restore the Foreman server or Smart Proxy server backups. For more information, see Restoring Foreman server or Smart Proxy server from a backup in Administering Foreman.
Appendix A: Troubleshooting permission issues
Foreman upgrades perform pre-upgrade checks. If the pre-upgrade check discovers permission issues, it fails with an error similar to the following one:
2024-01-29T20:50:09 [W|app|] Could not create role 'Ansible Roles Manager': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found:
If you see an error like this on your Foreman server, identify and remedy the permission issues.
-
On your Foreman server, identify permission issues:
# foreman-maintain health check --label duplicate_permissions
-
Fix permission issues:
# foreman-rake db:seed
-
Rerun the check to ensure no permission issues remain:
# foreman-maintain health check --label duplicate_permissions