1. Upgrade Overview

This chapter details the prerequisites and available upgrade paths to Foreman 6.9-beta. Review this information before upgrading your current Foreman installation.

In this guide, the terms update, upgrade, and migrate have the following meanings:

Upgrading

The process of advancing your Foreman server and Smart Proxy server installations from a y-stream release to the next, for example Foreman 6.8 to Foreman 6.9-beta.

Updating

The process of advancing your Foreman server and Smart Proxy server installations from a z-stream release to the next, for example Foreman 6.9-beta.0 to Foreman 6.9-beta.1.

Migrating

The process of moving an existing Foreman installation to another Red Hat Enterprise Linux server.

For interactive upgrade instructions, you can also use the Foreman Upgrade Helper on the Red Hat Customer Portal. This application provides you with an exact guide to match your current version number. You can find instructions that are specific to your upgrade path, as well as steps to prevent known issues. For more information, see Foreman Upgrade Helper on the customer portal.

Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

1.1. Prerequisites

Upgrading to Foreman 6.9-beta affects your entire Foreman infrastructure. Before proceeding, complete the following:

  • Review this guide so that you are aware of the upgrade process and its impact.

  • Plan your upgrade path. For more information, see Upgrade Paths.

  • Until BZ#1665893 is resolved, read the Knowledgebase solution Candlepin gets stuck during startup forever, logging huge thread dump to error.log, and perform the resolution steps before beginning the upgrade.

  • Plan for the required downtime. Foreman services are shut down during the upgrade. The upgrade process duration might vary depending on your hardware configuration, network speed, and the amount of data that is stored on the server.

    Upgrading Foreman takes approximately 1 - 2 hours.

    Upgrading Smart Proxy takes approximately 10 - 30 minutes.

  • Ensure that you have sufficient storage space on your server. For more information, see Preparing your Environment for Installation in Installing Foreman server from a Connected Network and Preparing your Environment for Installation in Installing Smart Proxy server.

  • Back up your Foreman server and all Smart Proxy servers. For more information, see Backing Up Foreman server and Smart Proxy server in the Administering Foreman 6.8 guide.

  • Plan for updating any scripts you use that contain Foreman API commands because some API commands differ between versions of Foreman. For more information about changes in the API, see the Knowledgebase article API Changes Between Foreman Versions on the Red Hat Customer Portal.

Warning
If you customize configuration files, manually or use a tool such as Hiera, these customizations are overwritten when the installation script runs during upgrading or updating. You can use the --noop option with the foreman-installer script to test for changes. For more information, see the Red Hat Knowledgebase solution How to use the noop option to check for changes in Foreman config files during an upgrade.

1.2. Upgrade Paths

You can upgrade to Foreman from the previous version. Foreman servers and Smart Proxy servers on earlier versions must first be upgraded to the Foreman version previous to this. For more information, see the Foreman 6.8 Upgrading and Updating Foreman guide.

Overview of Foreman 6.9-beta Upgrade Paths
Figure 1. Overview of Foreman 6.9-beta Upgrade Paths
Warning
Upgrading from the Beta to GA version is not supported.

The high level steps in upgrading to Foreman 6.9-beta are as follows.

  1. Clone your existing Foreman servers. For more information, see Cloning Foreman server.

  2. Upgrade Foreman server and all Smart Proxy servers to Foreman 6.9-beta. For more information, see Upgrading Foreman server.

  3. Upgrade to https://yum.theforeman.org/client/2.4/ on all Foreman clients. For more information, see Upgrading Foreman Clients.

1.3. Following the Progress of the Upgrade

Because of the lengthy upgrade time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base. You can also see the screen manual page for more information.

If you lose connection to the command shell where the upgrade command is running you can see the logs in /var/log/foreman-installer/satellite.log to check if the process completed successfully.

1.4. Upgrading Smart Proxies Separately from Foreman

You can upgrade Foreman to the version 6.9-beta and keep Smart Proxies at the version 6.8 until you have bandwidth to upgrade them too.

All the functionality that worked previously works on 6.8 Smart Proxies. However, the functionality added in the 6.9-beta release will not work until you upgrade Smart Proxies to 6.9-beta.

Upgrading Smart Proxies after upgrading Foreman can be useful in the following example scenarios:

  1. If you want to have several smaller outage windows instead of one larger window.

  2. If Smart Proxies in your organization are managed by several teams and are located in different locations.

  3. If you use a load-balanced configuration, you can upgrade one load-balanced Smart Proxy and keep other load-balanced Smart Proxies at 1 version lower. This allows you to upgrade all Smart Proxies one after another without any outage.

2. Cloning Foreman server

When you upgrade Foreman server, you can optionally create a clone of your Foreman to ensure that you do not lose any data while you upgrade. After your upgrade is complete, you can then decommission the earlier version of Foreman server.

Use the following procedures to clone your Foreman instances to preserve your environments in preparation for upgrade.

The Foreman clone tool does not support migrating a Smart Proxy server to Red Hat Enterprise Linux 7. Instead you must backup the existing Smart Proxy server, restore it on Red Hat Enterprise Linux 7, then reconfigure Smart Proxy server.

Terminology

Ensure that you understand the following terms:

Source server: the server that you clone

Target server: the new server that you copy files to and clone the source server to.

2.1. Cloning Process Overview

  1. Back up the source server.

  2. Clone the source server to the target server.

  3. Power off the source server.

  4. Update the network configuration on the target server to match the target server’s IP address with its new host name.

  5. Restart goferd in Content hosts and Smart Proxies to refresh the connection.

  6. Test the new target server.

2.2. Prerequisites

To clone Foreman server, ensure that you have the following resources available:

  • A minimal install of Red Hat Enterprise Linux 7 server to become the target server. Do not install Red Hat Enterprise Linux 7 software groups, or third-party applications. Ensure that your server complies with all the specifications of Preparing your Environment for Installation in Installing Foreman server.

  • A backup from Foreman 6.8 that you make using the foreman-maintain backup script. You can use a backup with or without Pulp data.

  • A Foreman subscription for the target server.

Before you begin cloning, ensure the following conditions exist:

  • The target server is on an isolated network. This avoids unwanted communication with Smart Proxy servers and hosts

  • The target server has the capacity to store all your backup files from the source server.

Customized configuration files

If you have any customized configurations on your source server that are not managed by the foreman-installer tool or Foreman backup process, you must manually back up these files.

2.3. Pulp Data Considerations

You can clone Foreman server without including Pulp data. However, for your cloned environment to work, you do require Pulp data. If the target server does not have Pulp data. it is not a fully working Foreman.

To transfer Pulp data to a target server, you have two options:

  • Clone using backup with Pulp data

  • Clone using backup without Pulp data and copy /var/lib/pulp manually from source server.

If your pulp_data.tar file is greater than 500 GB, or if you use a slow storage system, such as NFS, and your pulp_data.tar file is greater than 100 GB, do not include pulp_data.tar in the backup because this can cause memory errors during extraction. Copy the pulp_data.tar file from the source server to the target server.

To back up without Pulp data

Follow the steps in the procedure in Cloning Foreman server and replace the steps that involve cloning with Pulp data with the following steps:

  1. Perform a backup with MongoDB and PostgreSQL databases active excluding the Pulp data:

    # {foreman-maintain} backup offline --skip-pulp-content \
    --assumeyes /var/backup
  2. Stop and disable the foreman-maintain services

    # {foreman-maintain} service stop
    # {foreman-maintain} service disable
  3. Copy the Pulp data to the target server:

    # rsync --archive --partial --progress --compress \
    /var/lib/pulp target_server.example.com:/var/lib/pulp

2.4. Cloning Foreman server

Use the following procedures to clone Foreman server. Note that because of the high volume of data that you must copy and transfer as part of these procedures, it can take a significant amount of time to complete.

2.4.1. Preparing the source server for cloning

On the source server, complete the following steps:

  1. Verify the Pool ID of your Foreman subscription:

    # subscription-manager list --consumed \
    --matches 'Foreman'|grep "Pool ID:"|awk '{print $3}'

    Note the Pool ID for later use.

  2. Remove the Foreman subscription.

    # subscription-manager remove --serial=$(subscription-manager list \
    --consumed \
    --matches '{ProjectName}'|grep "Serial:"|awk '{print $2}')
  3. Determine the size of the Pulp data:

    # du -sh /var/lib/pulp/
  4. If you have less than 500 GB of Pulp data, perform a backup with MongoDB and PostgreSQL databases active including the Pulp data. If you have more than 500 GB of Pulp data, skip the following steps and complete the steps in Pulp Data Considerations before you continue.

    # {foreman-maintain} backup offline --assumeyes /var/backup
  5. Stop and disable the foreman-maintain services:

    # {foreman-maintain} service stop
    # {foreman-maintain} service disable

2.4.2. Cloning to the Target Server

To clone your server, complete the following steps on your target server:

  1. The satellite-clone tool defaults to using /backup/ as the backup folder. If you copy to a different folder, update the backup_dir variable in the /etc/satellite-clone/satellite-clone-vars.yml file.

  2. Place the backup files from the source Foreman in the /backup/ folder on the target server. You can either mount the shared storage or copy the backup files to the /backup/ folder on the target server.

  3. Power off the source server.

  4. Enter the following commands to register to the Customer Portal, attach subscriptions, and enable only the required subscriptions:

    # subscription-manager register your_customer_portal_credentials
    # subscription-manager attach --pool=pool_ID
    # subscription-manager repos --disable=*
    # subscription-manager repos \
    --enable=rhel-7-server-rpms \
    --enable=rhel-server-rhscl-7-rpms \
    --enable=rhel-7-server-satellite-maintenance-6-rpms \
    --enable=rhel-7-server-satellite-6.8-rpms
  5. Install the satellite-clone package

    # {package-install-project} satellite-clone

    After you install the satellite-clone tool, you can adjust any configuration to suit your own deployment in the /etc/satellite-clone/satellite-clone-vars.yml file.

  6. Run the satellite-clone tool.

    # satellite-clone
  7. Reconfigure DHCP, DNS, TFTP and remote execution services. The cloning process disables these services on the target Foreman server to avoid conflict with the source Foreman server.

  8. Reconfigure and enable DHCP, DNS, TFTP in the Foreman web UI. For more information, see Configuring External Services on Foreman server in Installing Foreman server.

  9. Enable remote execution:

    # {installer-scenario} \
    --enable-foreman-plugin-remote-execution \
    --enable-foreman-proxy-plugin-remote-execution-ssh
  10. Log on to the Foreman web UI, with the username admin and the password changeme. Immediately update the admin password to secure credentials.

  11. Ensure that the correct organization is selected.

  12. Navigate to Content > Subscriptions, then click Manage Manifest.

  13. Click the Refresh button, then click Close to return to the list of subscriptions.

  14. Verify that the available subscriptions are correct.

  15. Follow the instructions in the /usr/share/satellite-clone/logs/reassociate_capsules.txt file to restore the associations between Smart Proxies and their lifecycle environments.

  16. Update your network configuration, for example, DNS, to match the target server’s IP address with its new host name. The satellite-clone tool changes the hostname to the source server’s hostname. If you want to change the hostname to something different, you can use the satellite-change-hostname tool. For more information, see Renaming a Foreman or Smart Proxy server in Administrating Foreman.

  17. If the source server uses the virt-who daemon, install and configure it on the target server. Copy all the virt-who configuration files in the /etc/virt-who.d/ directory from the source server to the same directory on the target server. For more information, see Configuring Virtual Machine Subscriptions.

After you perform an upgrade using the following chapters, you can safely decommission the source server.

3. Upgrading Red Hat Satellite

Warning
If you have Foreman installed in a high availability configuration, contact Red Hat Support before upgrading to Foreman 6.9-beta.

Use the following procedures to upgrade your existing Foreman to Foreman 6.9-beta:

Before upgrading, see Prerequisites.

3.1. Upgrading Foreman server

This section describes how to upgrade Foreman server from 6.8 to 6.9-beta. You can upgrade from any minor version of Foreman Server 6.8.

Before You Begin
  • Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

  • Review and update your firewall configuration prior to upgrading your Foreman server. For more information, see Preparing your environment for installation in Installing Foreman server.

  • Ensure that you do not delete the manifest from the Customer Portal or in the Foreman Web UI because this removes all the entitlements of your content hosts.

  • Back up and remove all Foreman hooks before upgrading. Restore any hooks only after Foreman is known to be working after the upgrade is complete.

  • If you have edited any of the default templates, back up the files either by cloning or exporting them. Cloning is the recommended method because that prevents them being overwritten in future updates or upgrades. To confirm if a template has been edited, you can view its History before you upgrade or view the changes in the audit log after an upgrade. In the web UI, Navigate to Monitor > Audits and search for the template to see a record of changes made. If you use the export method, restore your changes by comparing the exported template and the default template, manually applying your changes.

Smart Proxy Considerations
  • If you use Content Views to control updates to a Smart Proxy server’s base operating system, or for Smart Proxy server repository, you must publish updated versions of those Content Views.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.

Configuring the BASH shell

The BASH shell stores the location of a binary in a hash table. During the upgrade, the location of the foreman-maintain script is changed, but BASH does not register this change, and foreman-maintain fails if it calls the script after the change.

  • Optional: Before the upgrade, users of the BASH shell can set the checkhash option temporarily to ensure foreman-maintain works after the installer completes. Enter a command as follows in your BASH shell:

    # shopt -s checkhash
  • After a successful or failed upgrade, in all currently running BASH shells, enter the following command:

    # hash -d {foreman-maintain} 2> /dev/null
Upgrade Scenarios
FIPS mode

You cannot upgrade Foreman server from a RHEL base system that is not operating in FIPS mode to a RHEL base system that is operating in FIPS mode.

To run Foreman server on a Red Hat Enterprise Linux base system operating in FIPS mode, you must install Foreman on a freshly provisioned RHEL base system operating in FIPS mode. For more information, see Preparing your environment for installation in Installing Foreman server.

3.1.1. Upgrading a Connected Foreman server

Use this procedure for a Foreman server connected to the Red Hat Content Delivery Network.

Warning
If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the installation script runs during upgrading or updating. You can use the --noop option with the foreman-installer script to test for changes. For more information, see the Red Hat Knowledgebase solution How to use the noop option to check for changes in Foreman config files during an upgrade.
Upgrade Foreman server
  1. Create a backup.

  2. Optional: If you made manual edits to DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required.

  3. Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:

    # {foreman-installer} --foreman-proxy-dns-managed=false \
    --foreman-proxy-dhcp-managed=false
  4. Optional: If you use PostgreSQL as an external database, on the PostgreSQL server, install the rh-postgresql12-postgresql-evr package, which is available from the {RepoRHEL7ServerForemanServerProductVersion} repository:

    # yum install rh-postgresql12-postgresql-evr
  5. In the Foreman web UI, navigate to Hosts > Discovered hosts. On the Discovered Hosts page, power off and then delete the discovered hosts. From the Select an Organization menu, select each organization in turn and repeat the process to power off and delete the discovered hosts. Make a note to reboot these hosts when the upgrade is complete.

  6. Ensure that the Foreman Maintenance repository is enabled:

    # subscription-manager repos --enable \
    {RepoRHEL7ServerForemanMaintenanceProductVersion}
  7. Check the available versions to confirm the version you want is listed:

    # {foreman-maintain} upgrade list-versions
  8. Use the health check option to determine if the system is ready for upgrade. When prompted, enter the hammer admin user credentials to configure foreman-maintain with hammer credentials. These changes are applied to the /etc/foreman-maintain/foreman-maintain-hammer.yml file.

    # foreman-maintain upgrade check --target-version 6.8

    Review the results and address any highlighted error conditions before performing the upgrade.

  9. Because of the lengthy upgrade time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

    If you lose connection to the command shell where the upgrade command is running you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

  10. Perform the upgrade:

    # foreman-maintain upgrade run --target-version 6.8
  11. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel
  12. Optional: If a kernel update occurred since the last reboot, stop the foreman-maintain services and reboot the system:

    # {foreman-maintain} service stop
    # reboot
  13. If using a BASH shell, after a successful or failed upgrade, enter:

    # hash -d {foreman-maintain} service 2> /dev/null
  14. Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups that you make.

  15. If you make changes in the previous step, restart the foreman-maintain services.

    # {foreman-maintain} service restart
  16. If you have the OpenSCAP plug-in installed, but do not have the default OpenSCAP content available, enter the following command:

    # foreman-rake foreman_openscap:bulk_upload:default

3.2. Synchronizing the New Repositories

You must enable and synchronize the new 6.9-beta repositories before you can upgrade Smart Proxy servers and Foreman clients.

Procedure
  1. In the Foreman web UI, navigate to Content > Red Hat Repositories.

  2. Toggle the Recommended Repositories switch to the On position.

  3. From the list of results, expand the following repositories and click the Enable icon to enable the repositories:

    • To upgrade Foreman clients, enable the Red Hat https://yum.theforeman.org/client/2.4/ repositories for all Red Hat Enterprise Linux versions that clients use.

    • If you have Smart Proxy servers, to upgrade them, enable the following repositories too:

      Foreman Smart Proxy 6.9-beta (for RHEL 7 Server) (RPMs)

      Foreman Maintenance 6 (for RHEL 7 Server) (RPMs)

      Red Hat Ansible Engine {ForemanAnsibleVersion} RPMs for Red Hat Enterprise Linux 7 Server

      Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server

    Note

    If the 6.9-beta repositories are not available, refresh the Subscription Manifest. Navigate to Content > Subscriptions, click Manage Manifest, then click Refresh.

  4. Navigate to Content > Sync Status.

  5. Click the arrow next to the product to view the available repositories.

  6. Select the repositories for 6.9-beta.

  7. Click Synchronize Now.

    Important

    If an error occurs when you try to synchronize a repository, refresh the manifest. If the problem persists, raise a support request. Do not delete the manifest from the Customer Portal or in the Foreman web UI; this removes all the entitlements of your content hosts.

  8. If you use Content Views to control updates to the base operating system of Smart Proxy server, update those Content Views with new repositories, publish, and promote their updated versions. For more information, see Managing Content Views in the Content Management Guide.

3.3. Upgrading Smart Proxy servers

This section describes how to upgrade Smart Proxy servers from 6.8 to 6.9-beta.

Before You Begin
  • You must upgrade Foreman server before you can upgrade any Smart Proxy servers. Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies Separately from Foreman.

  • Ensure the Foreman Smart Proxy 6.9-beta repository is enabled in Foreman server and synchronized.

  • Ensure that you synchronize the required repositories on Foreman server. For more information, see Synchronizing the New Repositories.

  • If you use Content Views to control updates to the base operating system of Smart Proxy server, update those Content Views with new repositories and publish their updated versions. For more information, see Managing Content Views in the Content Management Guide.

  • Ensure the Smart Proxy’s base system is registered to the newly upgraded Foreman server.

  • Ensure the Smart Proxy has the correct organization and location settings in the newly upgraded Foreman server.

  • Review and update your firewall configuration prior to upgrading your Smart Proxy server. For more information, see Preparing Your Environment for Smart Proxy Installation in Installing Smart Proxy server.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.

Upgrading Smart Proxy servers
  1. Create a backup.

  2. Optional: If you made manual edits to DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required.

  3. Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:

    # {foreman-installer} --foreman-proxy-dns-managed=false \
    --foreman-proxy-dhcp-managed=false
  4. In the Foreman web UI, go to Hosts > Discovered hosts. If there are discovered hosts available, power off the hosts and then delete all entries under the Discovered hosts page. Select all other organizations in turn using the organization setting menu and repeat this action as required. Reboot these hosts after the upgrade has completed.

  5. Clean yum cache:

    # yum clean all
  6. Ensure that the rubygem-foreman_maintain package that provides foreman-maintain is installed and up to date:

    # yum install rubygem-foreman_maintain
  7. Update the gofer package:

    # yum update gofer
  8. Restart the goferd service:

    # systemctl restart goferd
  9. On Smart Proxy server, verify that the foreman_url setting points to the Foreman FQDN:

    # grep foreman_url /etc/foreman-proxy/settings.yml
  10. Check the available versions to confirm the version you want is listed:

    # {foreman-maintain} upgrade list-versions
  11. Because of the lengthy upgrade time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

    If you lose connection to the command shell where the upgrade command is running you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

  12. Use the health check option to determine if the system is ready for upgrade:

    # foreman-maintain upgrade check --target-version 6.8

    Review the results and address any highlighted error conditions before performing the upgrade.

  13. Perform the upgrade:

    # foreman-maintain upgrade run --target-version 6.8
    Warning

    If you run the command from a directory containing a config subdirectory, you will encounter the following error:

    ERROR: Scenario (config/capsule.yaml) was not found, can not continue.

    In such a case, change directory, for example to the root user’s home directory, and run the command again.

  14. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel
  15. Optional: If a kernel update occurred since the last reboot, stop the foreman-maintain services and reboot the system:

    # {foreman-maintain} service stop
    # reboot
  16. Optional: If you made manual edits to DNS or DHCP configuration files, check and restore any changes required to the DNS and DHCP configuration files using the backups made earlier.

  17. Optional: If you use custom repositories, ensure that you enable these custom repositories after the upgrade completes.

  18. Optional: If you plan to use Smart Proxy server as a proxy for discovered hosts, install the Discovery plug-in and turn on the hosts that were shut down prior to the upgrade:

    # yum install tfm-rubygem-smart_proxy_discovery.noarch

3.4. Upgrading Foreman Clients

The https://yum.theforeman.org/client/2.4/ repository provides katello-agent and katello-host-tools, which provide communication services for managing Errata.

Note that the Katello agent is deprecated and will be removed in a future Foreman version. Migrate your workloads to use the remote execution feature to update clients remotely. For more information, see Host Management Without Goferd and Katello Agent in the Managing Hosts Guide.

Currently, the Foreman 6.8 version of katello-agent and other client libraries in the https://yum.theforeman.org/client/2.4/ repository are not formally tested or supported against Foreman 6.9-beta.

For deployments using katello-agent and goferd, update all clients to the new version of katello-agent. For deployments not using katello-agent and goferd, update all clients to the new version of katello-host-tools. Complete this action as soon as possible so that your clients are fully compatible with Foreman server.

Prerequisites
  • You must have upgraded Foreman server.

  • You must have enabled the new https://yum.theforeman.org/client/2.4/ repositories on the Foreman.

  • You must have synchronized the new repositories in the Foreman.

  • If you have not previously installed katello-agent on your clients and you want to install, use the manual method. For more information, see Upgrade Foreman Clients Manually.

Warning

If you implemented custom certificates, you must retain the content of both the /root/ssl-build directory and the directory in which you created any source files associated with your custom certificates.

Failure to retain these files during an upgrade causes the upgrade to fail. If these files have been deleted, they must be restored from a backup in order for the upgrade to proceed.

Upgrade Foreman Clients Using the Bulk Repository Set UI:
  1. In the Foreman web UI, navigate to Hosts > Content Hosts and select the Content Hosts that you want to upgrade.

  2. From the Select Action list, select Manage Repository Sets.

  3. From the Repository Sets Management list, select the Foreman Tools 6.8 check box.

  4. From the Select Action list, select Override to Disabled, and click Done.

  5. When the process completes, on the same set of hosts from the previous steps, from the Select Action list Manage Repository Sets.

  6. From the Repository Sets Management list, select the Red Hat https://yum.theforeman.org/client/2.4/ check box.

  7. From the Select Action list, select Override to Enabled, and click Done.

  8. When the process completes, on the same set of hosts from the previous steps, from the Select Action list, select Manage Packages.

  9. In the Package search field, enter one of the following options depending on your configuration:

    • If your deployment uses katello-agent and goferd, enter katello-agent.

    • If your deployment does not use katello-agent and goferd, enter katello-host-tools.

  10. Until BZ#1649764 is resolved, from the Update list, you must select via remote execution. This is required because if you update the package using the Katello agent, the package update disrupts the communication between the client and Foreman or Smart Proxy server, which causes the update to fail. For more information, see Configuring and Setting Up Remote Jobs in the Managing Hosts guide.

Upgrade Foreman Clients Manually
  1. Log into the client system.

  2. Disable the repositories for the previous version of Foreman.

    # subscription-manager repos \
    --disable rhel-7-server-satellite-tools-6.8-rpms
  3. Enable the https://yum.theforeman.org/client/2.4/ repository for this version of Foreman.

    # subscription-manager repos \
    --enable=https://yum.theforeman.org/client/2.4/el7/x86_64/foreman-client-release.rpm
  4. Depending on your configuration, complete one of the following steps:

    • If your deployment uses katello-agent and goferd, enter the following command to install or upgrade katello-agent:

      # yum install katello-agent
    • If your deployment does not use katello-agent and goferd, enter the following command to install or upgrade katello-host-tools:

      # yum install katello-host-tools

4. Post-Upgrade Tasks

Some of the procedures in this section are optional. You can choose to perform only those procedures that are relevant to your installation.

If you use the PXE-based discovery process, then you must complete the discovery upgrade procedure on Foreman and on any Smart Proxy server with hosts that you want to be listed in Foreman on the Hosts > Discovered hosts page.

4.1. Upgrading Discovery

This section describes updating the PXELinux template and the boot image passed to hosts that use PXE booting to register themselves with Foreman server.

From Foreman 1.22, provisioning templates now have a separate association with a subnet, and do not default to using the TFTP Smart Proxy for that subnet. If you create subnets after the upgrade, you must specifically enable the Foreman or a Smart Proxy to provide a proxy service for discovery templates and then configure all subnets with discovered hosts to use a specific template Smart Proxy.

During the upgrade, for every subnet with a TFTP proxy enabled, the template Smart Proxy is set to be the same as the TFTP Smart Proxy. After the upgrade, check all subnets to verify this was set correctly.

These procedures are not required if you do not use PXE booting of hosts to enable Foreman to discover new hosts.

4.1.1. Upgrading Discovery on Foreman server

  1. Update the Discovery template in the Foreman web UI:

    1. Navigate to Hosts > Provisioning templates.

    2. On the PXELinux global default line, click Clone.

    3. Enter a new name for the template in the Name field, for example ACME PXE global default.

    4. In the template editor field, change the line ONTIMEOUT local to ONTIMEOUT discovery and click Submit.

    5. Navigate to Administer > Settings.

    6. Locate Global default PXELinux template and click on its Value.

    7. Select the name of the newly created template from the menu and click the tick button.

    8. Navigate to Hosts > Provisioning templates.

    9. Click Build PXE Default, then click OK.

  2. In the Foreman web UI, go to Configure > Discovery Rules and associate selected organizations and locations with discovery rules.

4.1.2. Upgrading Discovery on Smart Proxy servers

  1. Verify that the Foreman Discovery package is current on Foreman server.

    # yum install tfm-rubygem-foreman_discovery
  2. If an update occurred in the previous step, restart the foreman-maintain services.

    # {foreman-maintain} service restart
  3. Upgrade the Discovery image on the Foreman Smart Proxy that is either connected to the provisioning network with discovered hosts or provides TFTP services for discovered hosts.

    # yum install foreman-discovery-image
  4. On the same instance, install the package which provides the Proxy service, and then restart foreman-proxy service.

    # yum install tfm-rubygem-smart_proxy_discovery
    # service foreman-proxy restart
  5. In the Foreman web UI, go to Infrastructure > Smart Proxies and verify that the relevant Smart Proxy lists Discovery in the features column. Select Refresh from the Actions drop-down menu if necessary.

  6. Go to Infrastructure > Subnets and for each subnet on which you want to use discovery:

    1. Click the subnet name.

    2. On the Smart Proxies tab, ensure the Discovery Smart Proxy is set to a Smart Proxy you configured above.

4.1.3. Verifying Subnets have a Template Smart Proxy

Ensure all subnets with discovered hosts have a template Smart Proxy:
  1. In the Foreman web UI, navigate to Infrastructure > Subnets.

  2. Select the subnet you want to check.

  3. On the Smart Proxies tab, ensure a Template Smart Proxy has been set for this subnet.

For more information about configuring subnets with template Smart Proxies, see Configuring the Discovery Service in the Provisioning Guide

4.2. Upgrading virt-who

If virt-who is installed on Foreman server or a Smart Proxy server, it will be upgraded when they are upgraded. No further action is required. If virt-who is installed elsewhere, it must be upgraded manually.

Before You Begin

If virt-who is installed on a host registered to Foreman server or a Smart Proxy server, first upgrade the host to the latest packages available in the https://yum.theforeman.org/client/2.4/ repository. For information about upgrading hosts, see Upgrading Foreman Clients.

Upgrade virt-who Manually
  1. Upgrade virt-who.

    # yum upgrade virt-who
  2. Restart the virt-who service so the new version is activated.

    # systemctl restart virt-who.service

4.3. Removing the Previous Version of the Foreman Tools Repository

After completing the upgrade to Foreman 6.9-beta, the Foreman Tools 6.8 repository can be removed from Content Views and then disabled.

Disable Version 6.8 of the Foreman Tools Repository:

  1. In the Foreman web UI, navigate to Content > Red Hat Repositories.

  2. In the Enabled Repositories area, locate Foreman Tools 6.8 for RHEL 7 Server RPMs x86_64.

  3. Click the Disable icon to the right.

If the repository is still contained in a Content View then you cannot disable it. Packages from a disabled repository are removed automatically by a scheduled task.

4.4. Reclaiming MongoDB Space

The MongoDB database can use a large amount of disk space especially in heavily loaded deployments. Use this procedure to reclaim some of this disk space on Foreman.

Prerequisites
Procedure
  1. Stop Pulp services:

    # foreman-maintain service stop --only \
    pulp_celerybeat.service,pulp_resource_manager.service,pulp_streamer.service,pulp_workers.service,httpd
  2. Access the MongoDB shell:

    # mongo pulp_database
  3. Check the amount of disk space used by MongoDB before a repair:

    > db.stats()
  4. Ensure that you have free disk space equal to the size of your current MongoDB database plus 2 GB. If the volume containing the MongoDB database lacks sufficient space, you can mount a separate volume and use that for the repair.

  5. Enter the repair command. Note that the repair command blocks all other operations and can take a long time to complete, depending on the size of the database.

    > db.repairDatabase()
  6. Check the amount of disk space used by MongoDB after a repair:

    > db.stats()
  7. Exit the MongoDB shell:

    > exit
  8. Start Pulp services:

    # foreman-maintain service start

4.5. Reclaiming PostgreSQL Space

The PostgreSQL database can use a large amount of disk space especially in heavily loaded deployments. Use this procedure to reclaim some of this disk space on Foreman.

Procedure
  1. Stop all services, except for the postgresql service:

    # foreman-maintain service stop --exclude postgresql
  2. Switch to the postgres user and reclaim space on the database:

    # su - postgres -c 'vacuumdb --full --dbname=foreman'
  3. Start the other services when the vacuum completes:

    # foreman-maintain service start

4.6. Upgrading the MongoDB Storage Engine

When you complete the upgrade, you can optionally upgrade the MongoDB storage engine to WiredTiger. Note that if you already use WiredTiger, you do not have to perform this procedure after you upgrade. If you want to use WiredTiger, you must repeat the following procedure on Foreman server and all Smart Proxy servers. For more information about the WiredTiger storage engine, see WiredTiger Storage Engine in the MongoDB Manual.

Prerequisites

Before upgrading the storage engine, ensure that the following conditions exist:

  • Create a backup of the MongoDB storage.

  • Ensure that the /var/tmp directory has storage space that is at least twice the size of the /var/lib/mongodb directory.

  • Optional: On high traffic Foreman environments, use MongoDB repair to reclaim disk space. For more information, see the KCS article How to compact MongoDB files and/or reclaim disk space in "/var/lib/mongodb" in Foreman?.

  • Optional: On high traffic Foreman environments, use MongoDB compact to reclaim disk space. For more information, see compact in MongoDB Manual.

  • Optional: If you want to verify what version of MongoDB you currently use, enter the following command:

    # mongo pulp_database --eval "db.serverStatus().storageEngine"
Procedure

To upgrade the MongoDB storage engine, enter the following command on Foreman server and all Smart Proxy servers:

# {foreman-installer} --upgrade-mongo-storage-engine

4.7. Updating Templates, Parameters, Lookup Keys and Values

During the upgrade process, Foreman attempts to locate macros that are deprecated for Foreman 6.9-beta and converts old syntax to new syntax for the default {Product} templates, parameters, and lookup keys and values. However, {Product} does not convert old syntax in the custom templates that you have created and in the cloned templates.

The process uses simple text replacement, for example:

@host.params['parameter1'] -> host_param('parameter1')
@host.param_true?('parameter1') -> host_param_true?('parameter1')
@host.param_false?('parameter1') -> host_param_false?('parameter1')
@host.info['parameters'] -> host_enc['parameters']
Warning
If you use cloned templates in Foreman, verify whether the cloned templates have diverged from the latest version of the original templates in Foreman. The syntax for the same template can differ between versions of Foreman. If your cloned templates contain outdated syntax, update the syntax to match the latest version of the template.

To ensure that this text replacement does not break or omit any variables in your files during the upgrade, check all templates, parameters, and lookup keys and values for the old syntax and replace manually.

The following error occurs because of old syntax remaining in files after the upgrade:

 undefined method '#params' for Host::Managed::Jail
Fixing the outdated subscription_manager_registration snippet

Foreman 6.4 onwards uses the redhat_register snippet instead of the subscription_manager_registration snippet.

If you upgrade from Foreman 6.3 and earlier, ensure to replace the subscription_manager_registration snippet in your custom templates as follows:

<%= snippet "subscription_manager_registration" %>
               ↓
<%= snippet 'redhat_register' %>

4.8. Tuning Foreman server with Predefined Profiles

If your Foreman deployment includes more than 5000 hosts, you can use predefined tuning profiles to improve performance of Foreman.

Note that you cannot use tuning profiles on Smart Proxies.

You can choose one of the profiles depending on the number of hosts your Foreman manages and available hardware resources.

The tuning profiles are available in the /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes directory.

When you run the foreman-installer command with the --tuning option, deployment configuration settings are applied to Foreman in the following order:

  1. The default tuning profile defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml file

  2. The tuning profile that you want to apply to your deployment and is defined in the /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/ directory

  3. Optional: If you have configured a /etc/foreman-installer/custom-hiera.yaml file, Foreman applies these configuration settings.

Note that the configuration settings that are defined in the /etc/foreman-installer/custom-hiera.yaml file override the configuration settings that are defined in the tuning profiles.

Therefore, before applying a tuning profile, you must compare the configuration settings that are defined in the default tuning profile in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml, the tuning profile that you want to apply and your /etc/foreman-installer/custom-hiera.yaml file, and remove any duplicated configuration from the /etc/foreman-installer/custom-hiera.yaml file.

default

Number of managed hosts: 0-5000

RAM: 20G

Number of CPU cores: 4

medium

Number of managed hosts: 5001-10000

RAM: 32G

Number of CPU cores: 8

large

Number of managed hosts: 10001-20000

RAM: 64G

Number of CPU cores: 16

extra-large

Number of managed hosts: 20001-60000

RAM: 128G

Number of CPU cores: 32

extra-extra-large

Number of managed hosts: 60000+

RAM: 256G

Number of CPU cores: 48+

Procedure
  1. Optional: If you have configured the custom-hiera.yaml file on Foreman server, back up the /etc/foreman-installer/custom-hiera.yaml file to custom-hiera.original. You can use the backup file to restore the /etc/foreman-installer/custom-hiera.yaml file to its original state if it becomes corrupted:

    # cp /etc/foreman-installer/custom-hiera.yaml \
    /etc/foreman-installer/custom-hiera.original
  2. Optional: If you have configured the custom-hiera.yaml file on Foreman server, review the definitions of the default tuning profile in /usr/share/foreman-installer/config/foreman.hiera/tuning/common.yaml and the tuning profile that you want to apply in /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes/. Compare the configuration entries against the entries in your /etc/foreman-installer/custom-hiera.yaml file and remove any duplicated configuration settings in your /etc/foreman-installer/custom-hiera.yaml file.

  3. Enter the foreman-installer command with the --tuning option for the profile that you want to apply. For example, to apply the medium tuning profile settings, enter the following command:

    # foreman-installer --tuning medium

5. Updating Satellite Server, Capsule Server, and Content Hosts

Use this chapter to update your existing Foreman server, Smart Proxy server, and Content Hosts to a new minor version, for example, from 6.9-beta.0 to 6.9-beta.1.

Updates patch security vulnerabilities and minor issues discovered after code is released, and are often fast and non-disruptive to your operating environment.

Before updating, back up your Foreman server and all Smart Proxy servers. For more information, see Backing Up Foreman server and Smart Proxy server in the Administering Foreman guide.

5.1. Updating Foreman server

Prerequisites
  • Ensure that you have synchronized Foreman server repositories for Foreman, Smart Proxy, and https://yum.theforeman.org/client/2.4/.

  • Ensure each external Smart Proxy and Content Host can be updated by promoting the updated repositories to all relevant Content Views.

Warning
If you customize configuration files, manually or use a tool such as Hiera, these customizations are overwritten when the installation script runs during upgrading or updating. You can use the --noop option with the foreman-installer script to test for changes. For more information, see the Red Hat Knowledgebase solution How to use the noop option to check for changes in Foreman config files during an upgrade.

Updating Foreman server to the Next Minor Version

To Update Foreman server:
  1. Ensure the Foreman Maintenance repository is enabled:

    # subscription-manager repos --enable \
    {RepoRHEL7ServerForemanMaintenanceProductVersion}
  2. Check the available versions to confirm the next minor version is listed:

    # {foreman-maintain} upgrade list-versions
  3. Use the health check option to determine if the system is ready for upgrade. On first use of this command, foreman-maintain prompts you to enter the hammer admin user credentials and saves them in the /etc/foreman-maintain/foreman-maintain-hammer.yml file.

    # foreman-maintain upgrade check --target-version 6.8.z

    Review the results and address any highlighted error conditions before performing the upgrade.

  4. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

    If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

  5. Perform the upgrade:

    # foreman-maintain upgrade run --target-version 6.8.z
  6. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel
  7. Optional: If a kernel update occurred since the last reboot, stop the foreman-maintain services and reboot the system:

    # {foreman-maintain} service stop
    # reboot

5.2. Updating Disconnected Foreman server

Prerequisites
  • Before syncing the following repositories, set the download policy to Immediate. This is required because Foreman downloads all packages only during synchronization of repositories with the immediate download policy.

  • Ensure that you have synchronized the following Foreman server repositories for Foreman, Smart Proxy, and https://yum.theforeman.org/client/2.4/:

    • rhel-7-server-rpms

    • rhel-7-server-satellite-6.8-rpms

    • rhel-7-server-satellite-maintenance-6-rpms

    • rhel-server-rhscl-7-rpms

    • rhel-7-server-ansible-2.9-rpms

      For more information about configuring download policies, see Changing a download policy for a repository in the Content Management guide.

  • Ensure no Red Hat repositories are enabled by entering the command:

    # yum repolist
Updating Disconnected Foreman server to the Next Minor Version
  1. Create a new configuration file as follows:

    # vi /etc/yum.repos.d/redhat-local.repo
    
    [rhel-7-server-ansible-2.9-rpms]
    name=Ansible {ForemanAnsibleVersion} RPMs for Red Hat Enterprise Linux 7 Server x86_64
    baseurl=file:///var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/ansible/2.9/os/
    enabled=1
    
    [rhel-7-server-rpms]
    name=Red Hat Enterprise Linux 7 Server RPMs x86_64
    baseurl=file:///var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/os/
    enabled=1
    
    [{RepoRHEL7ServerForemanServerProductVersion}]
    name=Foreman for RHEL 7 Server RPMs x86_64
    baseurl=file:///var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/satellite/6.9-beta/os/
    enabled=1
    
    [{RepoRHEL7ServerForemanMaintenanceProductVersion}]
    name=Foreman Maintenance 6 for RHEL 7 Server RPMs x86_64
    baseurl=file:///var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/sat-maintenance/6/os/
    enabled=1
    
    [rhel-server-rhscl-7-rpms]
    name=Red Hat Software Collections RPMs for Red Hat Enterprise Linux 7 Server x86_64
    baseurl=file:///var/lib/pulp/published/yum/https/repos/Default_Organization/Library/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os/
    enabled=1
  2. In the configuration file, replace Default_Organization in the baseurl with the correct organization label. To obtain the organization label, enter the command:

    # ls /var/lib/pulp/published/yum/https/repos/
  3. Ensure that the rubygem-foreman_maintain package that provides foreman-maintain is installed and up to date:

    # yum install rubygem-foreman_maintain
  4. Check the available versions to confirm the next minor version is listed:

    # {foreman-maintain} upgrade list-versions
  5. Use the health check option to determine if the system is ready for the upgrade. On the first use of this command, foreman-maintain prompts you to enter the hammer admin user credentials and saves them in the /etc/foreman-maintain/foreman-maintain-hammer.yml file.

    # foreman-maintain upgrade check --target-version 6.9-beta.z

    Review the results and address any highlighted error conditions before performing the upgrade.

  6. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

    If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

  7. Perform the upgrade:

    # foreman-maintain upgrade run --target-version 6.9-beta.z
  8. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel
  9. Optional: If a kernel update occurred since the last reboot, stop the foreman-maintain services and reboot the system:

    # {foreman-maintain} service stop
    # reboot

5.3. Updating Smart Proxy server

Use this procedure to update Smart Proxy servers to the next minor version.

Procedure
  1. Update the gofer package:

    # yum install gofer
  2. Restart the goferd service:

    # systemctl restart goferd
  3. Ensure that the Foreman Maintenance repository is enabled:

    # subscription-manager repos --enable \
    {RepoRHEL7ServerForemanMaintenanceProductVersion}
  4. Check the available versions to confirm the next minor version is listed:

    # {foreman-maintain} upgrade list-versions
  5. Use the health check option to determine if the system is ready for upgrade:

    # foreman-maintain upgrade check --target-version 6.8.z

    Review the results and address any highlighted error conditions before performing the upgrade.

  6. Because of the lengthy update time, use a utility such as screen to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information about using the screen command, see How do I use the screen command? article in the Red Hat Knowledge Base.

    If you lose connection to the command shell where the upgrade command is running, you can see the logged messages in the /var/log/foreman-installer/satellite.log file to check if the process completed successfully.

  7. Perform the upgrade:

    # foreman-maintain upgrade run --target-version 6.8.z
  8. Check when the kernel packages were last updated:

    # rpm -qa --last | grep kernel
  9. Optional: If a kernel update occurred since the last reboot, stop the foreman-maintain services and reboot the system:

    # {foreman-maintain} service stop
    # reboot

5.4. Updating Content Hosts

Updating Content Hosts to the Next Minor Version

To Update a Content Host, enter the following commands:
  1. Until BZ#1649764 is resolved, update the gofer package:

    yum update gofer
  2. Restart goferd:

    # systemctl restart goferd
  3. Update all packages:

    # yum update
  4. Optional: If a kernel update occurred since the last reboot, reboot the system:

    # reboot