Upgrading Foreman to nightly

1. Upgrading overview

Review prerequisites and available upgrade paths below before upgrading your current Foreman installation to Foreman nightly.

1.1. Upgrade paths

You can upgrade to Foreman nightly from Foreman 3.13. For complete instructions on how to upgrade, see Upgrading Foreman.

The high-level steps in upgrading Foreman to nightly are as follows:

Upgrade your Foreman server to nightly.
Upgrade all Smart Proxy servers to nightly.

Smart Proxies at version 3.13 will keep working with your upgraded Foreman server nightly. After you upgrade Foreman server to nightly, you can upgrade your Smart Proxies separately over multiple maintenance windows.

Foreman services are shut down during the upgrade. Ensure to plan for the required downtime. The upgrade process duration might vary depending on your hardware configuration, network speed, and the amount of data that is stored on the server.

Upgrading Foreman server takes approximately 1 – 2 hours.
Upgrading Smart Proxy server takes approximately 10 – 30 minutes.

1.2. Prerequisites

Upgrading to Foreman nightly affects your entire Foreman infrastructure. Before proceeding, complete the following:

Read the Foreman nightly Release Notes.
Ensure that you have sufficient storage space on your server. For more information, see Preparing your Environment for Installation in Installing Foreman Server nightly on Debian/Ubuntu and Preparing your Environment for Installation in Installing Smart Proxy server.
Back up your Foreman server and all Smart Proxy servers. For more information, see Backing Up Foreman server and Smart Proxy server in Administering Foreman.
Plan for updating any scripts you use that contain Foreman API commands because some API commands differ between versions of Foreman.

Warning

If you customize configuration files, manually or using a tool such as Hiera, these changes are overwritten when the maintenance script runs during upgrading or updating. You can use the --noop option with the foreman-installer to test for changes.

1.3. Upgrading Smart Proxies separately from Foreman

You can upgrade Foreman to version nightly and keep Smart Proxies at version 3.13 until you have the capacity to upgrade them too.

All the functionality that worked previously works on 3.13 Smart Proxies. However, the functionality added in the nightly release will not work until you upgrade Smart Proxies to nightly.

Upgrading Smart Proxies after upgrading Foreman can be useful in the following example scenarios:

If you want to have several smaller outage windows instead of one larger window.
If Smart Proxies in your organization are managed by several teams and are located in different locations.

1.4. Following the progress of the upgrade

Because of the lengthy upgrade time, use a utility such as tmux to suspend and reattach a communication session. You can then check the upgrade progress without staying connected to the command shell continuously. For more information, see the tmux manual page.

If you lose connection to the command shell where the upgrade command is running you can see the logs in /var/log/foreman-installer/foreman.log to check if the process completed successfully.

2. Upgrading Foreman

Use the following procedures to upgrade your existing Foreman to Foreman nightly.

2.1. Foreman server upgrade considerations

This section describes how to upgrade Foreman server from 3.13 to nightly. You can upgrade from any minor version of Foreman server 3.13.

Before you begin

Review Prerequisites.
Note that you can upgrade Smart Proxies separately from Foreman. For more information, see Upgrading Smart Proxies separately from Foreman.
Review and update your firewall configuration. For more information, see Preparing your environment for installation in Installing Foreman Server nightly on Debian/Ubuntu.
If you have edited any of the default templates, back up the files either by cloning or exporting them. Cloning is the recommended method because that prevents them being overwritten in future updates or upgrades. To confirm if a template has been edited, you can view its History before you upgrade or view the changes in the audit log after an upgrade. In the Foreman web UI, navigate to Monitor > Audits and search for the template to see a record of changes made. If you use the export method, restore your changes by comparing the exported template and the default template, manually applying your changes.

Smart Proxy considerations

Note that Foreman server upgraded from 3.13 to nightly can use Smart Proxy servers still at 3.13.

2.2. Upgrading a connected Foreman server

Use this procedure for a Foreman server with access to the public internet

Warning

Upgrade Foreman server

Stop all Foreman services:
```
# foreman-maintain service stop
```
Take a snapshot or create a backup:
- On a virtual machine, take a snapshot.
- On a physical machine, create a backup.
Start all Foreman services:
```
# foreman-maintain service start
```
Optional: If you made manual edits to DNS or DHCP configuration in the /etc/zones.conf or /etc/dhcp/dhcpd.conf files, back up the configuration files because the installer only supports one domain or subnet, and therefore restoring changes from these backups might be required.
Optional: If you made manual edits to DNS or DHCP configuration files and do not want to overwrite the changes, enter the following command:
```
# foreman-installer \
--foreman-proxy-dhcp-managed=false \
--foreman-proxy-dns-managed=false
```
In the Foreman web UI, navigate to Hosts > Discovered hosts. On the Discovered Hosts page, power off and then delete the discovered hosts. From the Select an Organization menu, select each organization in turn and repeat the process to power off and delete the discovered hosts. Make a note to reboot these hosts when the upgrade is complete.
Stop all Foreman services:
```
# foreman-maintain service stop
```
Update repositories in /etc/apt/sources.list.d/foreman.list to consume content for Foreman nightly:
```
deb http://deb.theforeman.org/ My_Distribution_Code_Name nightly
deb http://deb.theforeman.org/ plugins nightly
```
Replace My_Distribution_Code_Name with the proper distribution code name based on the operating system of your Foreman server:
- bullseye for Debian 11
- bookworm for Debian 12
- jammy for Ubuntu 22.04
Update to Foreman nightly:
```
# apt-get update
# apt-get upgrade
```
Run foreman-installer:
```
# foreman-installer
```
Determine if the system needs a reboot:
```
$ ls /run/reboot-required
```
If the /run/reboot-required file exists, reboot the system:
```
# reboot
```

2.3. Performing post-upgrade tasks

Optional: If the default provisioning templates have been changed during the upgrade, recreate any templates cloned from the default templates. If the custom code is executed before and/or after the provisioning process, use custom provisioning snippets to avoid recreating cloned templates. For more information about configuring custom provisioning snippets, see Creating Custom Provisioning Snippets in Provisioning hosts.

Appendix A: Troubleshooting permission issues

Foreman upgrades perform pre-upgrade checks. If the pre-upgrade check discovers permission issues, it fails with an error similar to the following one:

2024-01-29T20:50:09 [W|app|] Could not create role 'Ansible Roles Manager': ERF73-0602 [Foreman::PermissionMissingException]: some permissions were not found:

If you see an error like this on your Foreman server, identify and remedy the permission issues.

Procedure

On your Foreman server, identify permission issues:

# foreman-maintain health check --label duplicate_permissions

Fix permission issues:
```
# foreman-rake db:seed
```

Verification

Rerun the check to ensure no permission issues remain:

# foreman-maintain health check --label duplicate_permissions