Administering Foreman

1. Tuning performance with predefined profiles

If your Foreman deployment includes a large number of hosts, you can use predefined tuning profiles to configure your Foreman server to use the optimal amount of CPU cores and memory. This can help improve Foreman performance.

Procedure

Select the profile that best matches your use case:

Table 1. Predefined tuning profiles
Tuning profile	Number of hosts	RAM	Number of CPU cores
default	0 – 5000	20G	4
medium	5001 – 10000	32G	8
large	10001 – 20000	64G	16
extra-large	20001 – 60000	128G	32
extra-extra-large	60000+	256G	48+

Apply a predefined tuning profile to your Foreman server:
```
# foremanctl deploy --tuning My_profile
```
Replace My_profile with the name of the profile you want to apply.

2. Managing users and roles

Foreman users represent individuals who access your Foreman and are scoped to organizations and locations. You assign roles to users to control which actions they can perform in your environment.

2.1. Managing Foreman users

As an administrator, you can create, modify and remove Foreman users. You can also configure access permissions for a user or a group of users by assigning them different roles.

2.1.1. Creating a user by using Foreman web UI

Create user accounts from the Foreman web UI to grant individuals access to your Foreman environment.

Users are strictly confined to their assigned organizations and locations. Users can only access and assign resources within the organizations and locations they belong to.

Procedure

In the Foreman web UI, navigate to Administer > Users.
Click Create User.
Enter the account details for the new user.
Click Submit to create the user.

The user account details that you can specify include the following:
- On the User tab, select an authentication source from the Authorized by list:
  - INTERNAL: to manage the user inside Foreman server.
  - EXTERNAL: to manage the user with external authentication. For more information, see Configuring authentication for Foreman users.
- On the Organizations tab, select organizations for the user. Select the default organization from the Default on login list. Foreman uses this organization for the user after login.
- On the Locations tab, select locations for the user. Select the default location from the Default on login list. Foreman uses this location for the user after login.

2.1.2. Creating a user by using Hammer CLI

Create user accounts by using Hammer CLI to grant individuals access to your Foreman environment.

Users are strictly confined to their assigned organizations and locations. Users can only access and assign resources within the organizations and locations they belong to.

Procedure

Create a user:
```
$ hammer user create \
--auth-source-id My_Authentication_Source \
--login My_User_Name \
--mail My_User_Mail \
--organization-ids My_Organization_ID_1,My_Organization_ID_2 \
--location-ids My_Location_ID_1,My_Location_ID_2 \
--password My_User_Password
```
Append --auth-source-id 1 so that the user is authenticated internally. Run hammer auth-source list to get a list of authentication sources. Add the --admin option to grant administrator privileges to the user.

You can modify the user details later by using the hammer user update command.

2.1.3. Assigning roles to a user by using Foreman web UI

Assign roles to a user in the Foreman web UI to control which permissions and resources they can access in your environment.

Procedure

In the Foreman web UI, navigate to Administer > Users.
Click the username of the user to be assigned one or more roles.

Note

If a user account is not listed, check that you are currently viewing the correct organization. To list all the users in Foreman, click Default Organization and then Any Organization.
Click the Locations tab, and select a location if none is assigned.
Click the Organizations tab, and check that an organization is assigned.
Click the Roles tab to display the list of available roles.
Select the roles to assign from the Roles list.

To grant all the available permissions, select the Administrator checkbox.
Click Submit.

To view the roles assigned to a user, click the Roles tab; the assigned roles are listed under Selected items. To remove an assigned role, click the role name in Selected items.

2.1.4. Assigning roles to a user by using Hammer CLI

Assign roles to a user with Hammer CLI to grant the permissions they need to perform tasks in Foreman.

Procedure

Assign roles to a user:

$ hammer user add-role \
--id My_User_ID \
--role My_Role_Name

2.1.5. Impersonating a different user account

As an administrator, you can impersonate another user by logging on to the Foreman web UI as that user. Impersonating another user is useful for testing and troubleshooting purposes because you can access exactly what the impersonated user can access in the system, including the same menus.

Audits are created to record the actions that the administrator performs while impersonating another user. However, all actions that an administrator performs while impersonating another user are recorded as having been performed by the impersonated user.

Prerequisites

You are logged on to the Foreman web UI as a user with administrator privileges for Foreman.

Procedure

In the Foreman web UI, navigate to Administer > Users.
To the right of the user that you want to impersonate, from the list in the Actions column, select Impersonate.

When you want to stop the impersonation session, in the upper right of the main menu, click the impersonation icon.

2.1.6. Creating an API-only user

You can create a user account that can interact only with the Foreman API. This is useful for service accounts or integrations that must not use the Foreman web UI.

Prerequisites

You have created a user and assigned roles to them. Note that this user must be authorized internally. For more information, see Managing users and roles.

Procedure

Log in to your Foreman as admin.
Navigate to Administer > Users and select a user.
On the User tab, set a password. Do not save or communicate this password with others. You can create pseudo-random strings on your console:
```
# openssl rand -hex 32
```
Create a Personal Access Token for the user. For more information, see Creating a Personal Access Token.

2.2. Managing SSH keys

Add SSH keys to user accounts so Foreman can deploy those keys during host provisioning.

Additional resources

2.2.1. Managing SSH keys for a user by using Foreman web UI

Add or remove SSH keys for a user in the Foreman web UI to control which keys Foreman deploys during provisioning.

Prerequisites

You are logged in to the Foreman web UI as an Admin user of Foreman or a user with the create_ssh_key permission enabled for adding SSH key and destroy_ssh_key permission for removing a key.

Procedure

In the Foreman web UI, navigate to Administer > Users.
From the Username column, click on the username of the required user.
Click on the SSH Keys tab.
- To Add SSH key
  1. Prepare the content of the public SSH key in a clipboard.
  2. Click Add SSH Key.
  3. In the Key field, paste the public SSH key content from the clipboard.
  4. In the Name field, enter a name for the SSH key.
  5. Click Submit.
- To Remove SSH key
  1. Click Delete on the row of the SSH key to be deleted.
  2. Click OK in the confirmation prompt.

2.2.2. Managing SSH keys for a user by using Hammer CLI

Add or remove SSH keys for a user with Hammer CLI to manage which keys Foreman deploys during provisioning.

Prerequisites

You are logged in to the Foreman web UI as an Admin user of Foreman or a user with the create_ssh_key permission enabled for adding SSH key and destroy_ssh_key permission for removing a key.

Procedure

To add an SSH key to a user, you must specify either the path to the public SSH key file, or the content of the public SSH key copied to the clipboard:

If you have the public SSH key file, enter the following command:

$ hammer user ssh-keys add \
--user-id user_id \
--name key_name \
--key-file ~/.ssh/id_rsa.pub

If you have the content of the public SSH key, enter the following command:

$ hammer user ssh-keys add \
--user-id user_id \
--name key_name \
--key ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNtYAAABBBHHS2KmNyIYa27Qaa7EHp+2l99ucGStx4P77e03ZvE3yVRJEFikpoP3MJtYYfIe8k 1/46MTIZo9CPTX4CYUHeN8= host@user

To delete an SSH key from a user, enter the following command:

$ hammer user ssh-keys delete --id key_id --user-id user_id

To view an SSH key attached to a user, enter the following command:
```
$ hammer user ssh-keys info --id key_id --user-id user_id
```
To list SSH keys attached to a user, enter the following command:
```
$ hammer user ssh-keys list --user-id user_id
```

2.3. Managing Personal Access Tokens

Personal Access Tokens allow you to authenticate API requests without using your password. You can set an expiration date for your Personal Access Token and you can revoke it if you decide it should expire before the expiration date.

2.3.1. Creating a Personal Access Token

Create a Personal Access Token to authenticate API requests without sharing your password.

Prerequisites

Your user account has a role that grants the create_personal_access_tokens permission.

Procedure

In the Foreman web UI, navigate to Administer > Users.
Select a user for which you want to create a Personal Access Token.
On the Personal Access Tokens tab, click Add Personal Access Token.
Enter a Name for you Personal Access Token.
Optional: Select the Expires date to set an expiration date. If you do not set an expiration date, your Personal Access Token will never expire unless revoked.

Click Submit. You now have the Personal Access Token available to you on the Personal Access Tokens tab.

Important

Ensure to store your Personal Access Token as you will not be able to access it again after you leave the page or create a new Personal Access Token. You can click Copy to clipboard to copy your Personal Access Token.

Verification

Make an API request to your Foreman server and authenticate with your Personal Access Token:

$ curl \
--user My_Username:My_Personal_Access_Token \
https://foreman.example.com/api/status

You should receive a response with status 200, for example:
```
{"foreman_version":"nightly.0","result":"ok","status":200,"version":"3.5.1.10","api_version":2}
```
If you go back to Personal Access Tokens tab, you can see the updated Last Used time next to your Personal Access Token.

2.3.2. Revoking a Personal Access Token

Revoke a Personal Access Token before its expiration date when a token is compromised or no longer needed for API access.

Procedure

In the Foreman web UI, navigate to Administer > Users.
Select a user for which you want to revoke the Personal Access Token.
On the Personal Access Tokens tab, locate the Personal Access Token you want to revoke.
Click Revoke in the Actions column next to the Personal Access Token you want to revoke.

Verification

Make an API request to your Foreman server and try to authenticate with the revoked Personal Access Token:
```
$ curl \
--user My_Username:My_Personal_Access_Token \
https://foreman.example.com/api/status
```

You receive the following error message:

{
  "error": {"message":"Unable to authenticate user My_Username"}
}

2.4. Creating and managing user groups

You can manage permissions of several users at once by organizing them into user groups. User groups themselves can be further grouped to create a hierarchy of permissions.

With Foreman, you can assign permissions to groups of users. You can also create user groups as collections of other user groups. If you use an external authentication source, you can map Foreman user groups to external user groups.

User groups are defined in an organizational context, meaning that you must select an organization before you can access user groups.

Additional resources

Managing external user groups in Configuring authentication for Foreman users

2.4.1. Creating a user group by using Foreman web UI

Create a user group in the Foreman web UI and define which roles the group will have.

Procedure

In the Foreman web UI, navigate to Administer > User Groups.
Click Create User group.
On the User Group tab, specify the name of the new user group and select group members:
- Select the previously created user groups from the User Groups list.
- Select users from the Users list.
On the Roles tab, select the roles you want to assign to the user group. Alternatively, select the Admin checkbox to assign all available permissions.
Click Submit.

2.4.2. Creating a user group by using Hammer CLI

Create a user group by using Hammer CLI and define which roles the group will have.

Procedure

Create a user group:

$ hammer user-group create \
--name My_User_Group_Name \
--role-ids My_Role_ID_1,My_Role_ID_2 \
--user-ids My_User_ID_1,My_User_ID_2

2.4.3. Removing a user group

Remove a user group from Foreman when you no longer need it or when you want to reorganize how permissions are assigned.

Procedure

In the Foreman web UI, navigate to Administer > User Groups.
Click Delete to the right of the user group you want to delete.
Click Confirm to delete the user group.

3. Backing up Foreman

You can back up your Foreman deployment to ensure the continuity of your Foreman deployment and associated data in case a disaster occurs. If your deployment uses custom configurations, you must consider how to handle these custom configurations when you plan your backup and disaster recovery policy.

3.1. Planning Foreman backup

Backing up your Foreman server requires you to plan scheduling, storage, and security so you avoid disruption and protect sensitive data.

3.1.1. Available backup methods

You can create an offline backup of your Foreman server.

Offline backup

All Foreman services are stopped during an offline backup to ensure data consistency. The backup process includes the following actions:

Preflight checks to verify that no tasks are running and that the database is consistent.
Service shutdown to stop all Foreman services cleanly.
PostgreSQL database dumps to backup content and configuration.
Configuration backup to archive the foremanctl state and configuration.
Content backup to back up Pulp content directory. This is an optional action. You can skip this action by using the --skip-pulp-content option with the foremanctl backup command.
Service restart to restore all Foreman services.

3.1.2. Best practices for backing up Foreman

Apply these recommendations when scheduling and storing backups so you protect sensitive data and avoid conflicts with other administrators.

Foreman community recommends backing up Foreman to a separate storage device on a separate system. The foremanctl backup command creates a backup of your Foreman server and all associated data.

Foreman services are unavailable during the backup. Coordinate with other administrators to ensure no conflicting tasks run during the backup window.

Warning

Request other users of Foreman server to save any changes and warn them that Foreman services are unavailable for the duration of the backup. Ensure no other tasks are scheduled for the same time as the backup.

You can schedule a backup by using cron.

Note	During offline backups, the services are inactive and Foreman is in a maintenance mode. A firewall rejects traffic from outside on port 443 to ensure there are no modifications triggered.

Encrypt or move the backup to a secure location to minimize the risk of damage or unauthorized access to the hosts. A backup has sensitive information from the /root/ssl-build directory. For example, it can have hostnames, SSH keys, request files and SSL certificates.
Because the foremanctl backup command does not automatically delete old backups, consider implementing backup retention policies, including rotating old backups and monitoring backup storage usage.

3.1.3. Directories created during backups

Foreman organizes backups in time-stamped subdirectories, which helps you identify and select the correct backup when restoring your system.

The foremanctl backup command creates a time-stamped subdirectory in the backup directory that you specify. The foremanctl backup command does not overwrite backups, therefore you must select the correct directory or subdirectory when restoring from a backup or an incremental backup.

Foreman creates the following default backup directory:

foreman-backup on Foreman server

If you want to set a custom directory name, add the --preserve-directory option and add a directory name. The backup is then stored in the directory you provide in the command line. If you use the --preserve-directory option, no data is removed if the backup fails.

3.1.4. Estimating the size of a backup

Estimate how much disk space a Foreman backup requires so you can ensure enough storage is available and avoid backup failures.

Procedure

Calculate the size of the uncompressed backup data:
1. Determine the size of PostgreSQL database data:
  # du -sh /var/lib/pgsql/data 100G /var/lib/pgsql/data
2. Determine the size of Pulp data:
  # du -sh /var/lib/pulp 100G /var/lib/pulp
3. Determine the size of the foremanctl configuration files:
  # du -sh /var/lib/foremanctl 10MB /var/lib/pulp

Calculate the expected size of the compressed backup data:

The compression ratio of the backup data is as follows:

Table 2. Backup data compression ratio
Data type	Directory	Compression ratio
PostgreSQL database files	`/var/lib/pgsql/data`	80 – 85%
Pulp data	`/var/lib/pulp`	Not compressed
foremanctl configuration	`/var/lib/foremanctl`	85%

In this example, the backup data is compressed as follows:

100 GB of PostgreSQL database data is compressed to 15-20 GB
100 GB of Pulp data is not compressed and occupies 100 GB of space
10 MB of foremanctl configuration is compressed to 1.5 MB

Add up the size of the uncompressed and compressed data, and add a 20% safety margin. For example:
- 201 GB of uncompressed data + 120 GB of compressed data = 321 GB
- 20% safety margin = 64 GB
- Total space required: 321 GB + 64 GB = 385 GB
To ensure a successful backup or restore, 385 GB of space must be allocated for the backup location.

3.2. Performing a full backup

Run a full offline backup on Foreman server to capture all Foreman content and configuration. A full backup is useful when you want to prepare for a future restore from scratch.

Prerequisites

Your backup location must have sufficient available disk space to store the backup. For more information, see Estimating the size of a backup.
To enable Foreman to save the backup to an NFS share, the root user of your Foreman server must be able to write to the NFS share. NFS export options such as root_squash and all_squash are known to prevent this. For more information, see Red Hat Enterprise Linux Configuring and using network files services and Red Hat Enterprise Linux Securing network services.

Warning

Procedure

Back up your Foreman server:
```
# foremanctl backup /var/foreman-backup
```

3.3. Performing a backup without Pulp content

Run an offline backup that excludes the contents of the Pulp directory. A backup without Pulp content is useful for debugging purposes and is only intended to provide access to configuration files without backing up the Pulp database.

Warning

Do not use a backup without Pulp content to restore your Foreman server for production use cases.

Prerequisites

Your backup location must have sufficient available disk space to store the backup. For more information, see Estimating the size of a backup.

Procedure

Back up your Foreman server without Pulp content:

# foremanctl backup --skip-pulp-content /var/backup_directory

4. Restoring Foreman from backup

You can restore your Foreman server from a backup to recover after failure or data loss. This process outlines how to restore the backup on the same server that generated the backup, and all data covered by the backup is deleted on the target system. If the original system is unavailable, provision a system with the same configuration settings and host name.

4.1. Restoring from a full backup

Restore your Foreman server from a full backup to return the system to the state at the time of the backup. When the restore process completes, all processes are online, and all databases and system configuration revert to the state at the time of the backup.

The restore process can take a long time to complete. The foremanctl restore command performs the following actions:

Validation checks to verify the backup integrity, disk space, and system requirements
Restoring databases, Pulp content, encryption keys, OAuth secrets, and passwords
Configuring and starting of services
Confirming that services are running as expected

Prerequisites

Your Foreman server must have the same host name, configuration, and be the same minor version (X.Y) as the original system.
The target directory must exist and be writable. The target directory is read from the configuration files contained within the backup archive.
If the backed up system had FIPS enabled, the system on which you are restoring must also have FIPS enabled.
Your system must have enough space to extract the backup data. Your system must also have enough space left for the backup data after the restoration. For more information, see Estimating the size of a backup.

Procedure

Restore the correct SELinux contexts:
```
# restorecon -Rv /
```
Install Foreman server on the system where you want to restore the backup. For more information, see Installing Foreman Server with Katello nightly plugin (containerized) on Enterprise Linux.
Copy the backup data to the local file system on Foreman server. Use /var/ or /var/tmp/.
Optional: Validate the backup to ensure that all files required for the restore process exist and the backup metadata is valid:
```
# foremanctl restore /var/backup_directory --validate
```
Run the restoration script:
```
# foremanctl restore /var/backup_directory --force
```
The --force option is required to confirm that the data from the fresh Foreman deployment installed on the system will be permanently deleted and replaced by the data from the backup.

Next steps

If you create a new instance of your Foreman server, decommission the old instance after restoring the backup. Cloned instances are not supposed to run in parallel in a production environment.

Troubleshooting

Review the /var/log/foreman/production.log and /var/log/messages log files.

5. Renewing certificates

You can renew default self-signed server and client certificates as well as the default self-signed certificate authority on Foreman server to prevent them from expiring. You can also renew the custom SSL certificate as well as the custom certificate authority certificate on Foreman server and on Smart Proxy server.

5.1. Renewing self-signed certificates on Foreman server

You can renew expired self-signed server and client certificates on Foreman server to restore secure connections. You can also specify a custom validity period instead of the default 7300 days.

Procedure

Renew server and client certificates:
```
# foremanctl deploy --certificate-renew
```
To set a custom validity period, append --certificate-validity-days My_Days with the validity period in days for the new certificate.

Verification

In your browser, go to your Foreman server login page, for example, https://foreman.example.com, and inspect the certificate in the browser. This is typically displayed as a shield, padlock, or tune icon next to the address bar depending on your browser.
On the command line, verify the new certificate validity period:
```
# echo | openssl s_client -connect 127.0.0.1:443 2>/dev/null | openssl x509 -noout -dates
```
The output displays the notBefore and notAfter dates of the certificate, where notBefore is the date when the certificate becomes valid and notAfter is the date when it expires.

5.2. Renewing self-signed CA certificates

The self-signed certificate authority (CA) certificate on your Foreman server is valid for 20 years by default. If you need to renew it because the expiration date is close, follow the below procedure.

Procedure

Renew CA certificates:
```
# foremanctl deploy --certificate-ca-renew
```
To set a custom validity period, append --certificate-ca-validity-days My_Days with the validity period in days for the new certificate.

Verification

In your browser, go to your Foreman server login page, for example, https://foreman.example.com, and inspect the CA certificate in the browser. This is typically displayed as a shield, padlock, or tune icon next to the address bar depending on your browser.
On the command line, verify the new certificate validity period:
```
# echo | openssl s_client -connect 127.0.0.1:443 2>/dev/null
```
The output displays the whole SSL connection information. Check the NotBefore and NotAfter dates of CA entry in the "Certificate chain" section, where NotBefore is the date when the certificate becomes valid and NotAfter is the date when it expires.

Next steps

Refresh the CA certificate on your Smart Proxy servers. For more information, see Configuring Smart Proxy server with SSL certificates in Installing a Smart Proxy Server nightly on Enterprise Linux.
Refresh the CA certificate on your hosts. For more information, see Refreshing the self-signed CA certificate on hosts in Managing hosts.

5.3. Renewing a custom SSL certificate on Foreman server

You can deploy a renewed custom SSL certificate on Foreman server to replace an expiring certificate and maintain trusted access to the Foreman web UI and API.

Prerequisites

You have created a new Certificate Signing Request (CSR) and sent it to the Certificate Authority to sign the certificate. Refer to the Configuring Foreman server with a custom SSL certificate guide before creating a new CSR because the Server certificate must have X.509 v3 Key Usage and Extended Key Usage extensions with required values. In return, you will receive the Foreman server certificate and CA bundle.

Procedure

Deploy the renewed CA certificates to Foreman server:

# foremanctl deploy \
--certificate-source custom_server \
--certificate-server-certificate "/root/foreman_cert/foreman_cert.pem" \
--certificate-server-key "/root/foreman_cert/foreman_cert_key.pem" \
--certificate-server-ca-certificate "/root/foreman_cert/ca_cert_bundle.pem"

Verification

Access the Foreman web UI from your local machine. For example, https://foreman.example.com.
In your browser, view the certificate details to verify the deployed certificate.

5.4. Renewing a custom server CA certificate

If you need to update the certification authority (CA) certificate that signed your Foreman server and Smart Proxy servers certificates, add the new CA certificate and use a temporary dual CA certificate file to retain the HTTPS connections to your Foreman server during the renewal.

Procedure

Check if the existing server certificate can be validated with the new CA certificate.

# openssl verify -CAfile /root/foreman_cert/old_ca_cert_bundle.pem /root/foreman_cert/foreman_cert.pem

If the check did not succeed, use both the old and the new CA certificate while updating consumers.

Add the new SSL certificate to the CA certificate file on Foreman server and keep the old SSL certificate.

# cat /root/foreman_cert/old_ca_cert_bundle.pem /root/foreman_cert/new_ca_cert_bundle.pem > /root/foreman_cert/ca_cert_bundle.pem

Renew the certificates on Foreman server.

# foremanctl deploy \
--certificate-source custom_server \
--certificate-server-certificate "/root/foreman_cert/foreman_cert.pem" \
--certificate-server-key "/root/foreman_cert/foreman_cert_key.pem" \
--certificate-server-ca-certificate "/root/foreman_cert/ca_cert_bundle.pem"

Refresh the certificates on any Smart Proxy servers.
Deploy the dual CA certificate on hosts.
Remove the old certificate from the CA certificates file on Foreman server, so the CA certificate file contains only the new SSL certificate.
```
# cat /root/foreman_cert/new_ca_cert_bundle.pem > /root/foreman_cert/ca_cert_bundle.pem
```

Renew the certificates on Foreman server.

# foremanctl deploy \
--certificate-source custom_server \
--certificate-server-certificate "/root/foreman_cert/foreman_cert.pem" \
--certificate-server-key "/root/foreman_cert/foreman_cert_key.pem" \
--certificate-server-ca-certificate "/root/foreman_cert/ca_cert_bundle.pem"

Refresh the certificates on any Smart Proxy servers.
Deploy the new CA certificate on hosts.

Additional resources

6. Managing logs for Foreman

With centralized logging, you can monitor Foreman service health, diagnose errors, and control log retention and disk usage in a single interface. You can also collect diagnostic data and attach it to support cases.

6.1. Logging architecture overview

Foreman server manages all service logs by using a centralized system built on systemd and journald. Instead of searching through separate log files in different directories, you can manage all service activity from a single pipeline.

All Foreman services send their log output to stdout by default. The container runtime automatically captures this data stream and passes it to the logging daemon.

You interact with the logging architecture through the following key components:

With the journald service, you collect and store all service logs in one place. By default, you lose log history when the system reboots as journald stores log data in volatile memory. You can configure journald to store logs in persistent storage so that you can retain log history across system restarts.

For details on configuring the journald service, see the official Red Hat Enterprise Linux documentation.
With journalctl, you can view, search, and filter logs for any service from a single interface without opening individual log files.
Use the foremanctl deploy command to change log levels or adjust logging verbosity.

Additional resources

6.2. Viewing service logs

You can view and filter logs for any Foreman service by using journalctl with the appropriate unit filter. Use correlation IDs to trace a single request across multiple services such as Foreman and Pulp.

Prerequisites

Foreman server is installed and running.
You have root access to the Foreman server host.

Procedure

To view logs for a specific service:
```
# journalctl --unit My_Service
```

To follow log output in real time, add the --follow flag:

# journalctl --follow --unit My_Service

For example:

# journalctl --follow --unit foreman

Example output:

Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Started GET "/api/v2/ping" for 192.168.124.1
Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Processing by Api::V2::PingController#ping as JSON
Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Completed 200 OK in 186ms (Views: 12.8ms | ActiveRecord: 8.1ms | Allocations: 42460)

In this output, eb515c9c is the correlation ID.

To trace a request across services, locate the correlation ID in the log output and filter by it:

# journalctl --grep My_Correlation_ID

For example:

# journalctl --grep eb515c9c

Example output:

Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Started GET "/api/v2/ping" for 192.168.124.1
Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Processing by Api::V2::PingController#ping as JSON
Aug 10 07:19:29 foreman.example.com pulp-api[1193]: pulp [eb515c9c-cb85-44fa-bd6e-c49eaab71bc5]: "GET /pulp/api/v3/status/ HTTP/1.1" 200 6250
Aug 10 07:19:29 foreman.example.com foreman[1632]: 2026-08-10T07:19:29 [I|app|eb515c9c] Completed 200 OK in 186ms (Views: 12.8ms | ActiveRecord: 8.1ms | Allocations: 42460)

Verification

Verify that the command returns log entries for the expected service:
```
# journalctl --unit My_Service -n 5
```

6.3. Setting service log levels

You can set the log level for Foreman services by using foremanctl deploy parameters.

At the initial deployment, when the containers are first created, the log level is set to the default value of info. You can change the log level for all services or for individual services.

Procedure

To set the global log level for all supported services, run:
```
# foremanctl deploy --log-level Log_Level
```
For example, to set the global log level to debug, run:
```
# foremanctl deploy --log-level debug
```

To override the log level for a specific service, use a service-specific parameter, such as --foreman-log-level and --foreman-proxy-log-level.

For example, to set only Foreman to error while keeping other services at the global level, run:

# foremanctl deploy --foreman-log-level error

After you complete debugging, reset the log level to the default to avoid filling up host disk space:

# foremanctl deploy --reset-foreman-log-level

Note

Each service inherits the default value from the log-level parameter, unless you set the value for the service by using the My_Service-log-level parameter. Running foremanctl deploy --log-level New_Log_Level or foremanctl deploy --reset-log-level does not affect the --My_Service-log-level parameters.

Verification

Verify that a new log level took effect as actions occur:
```
# journalctl --unit foreman --follow
```

Additional resources

Log level parameters

6.3.1. Log level parameters

You can control the log detail level of containerized Foreman services with the foremanctl deploy log level parameters.

The --log-level parameter sets the global baseline for all services. The valid values are: debug, info, warn, error, fatal out of which info is the default.

Per-service parameters override this value and translate it to the native vocabulary of each service:

--foreman-log-level: Foreman service. Values: debug, info, warn, error, fatal.
--foreman-proxy-log-level: Foreman Proxy. Values: debug, info, warn, error, fatal.
--pulp-log-level: Pulp services (pulpcore-api, pulpcore-content, pulpcore-worker). Values: debug, info, warning, error, critical.
--valkey-log-level: Valkey. Independent of --log-level. Values: debug, verbose, notice, warning, nothing. Default: notice.

These services do not have dedicated foremanctl deploy log level parameters and are not affected by the global --log-level parameter:

Candlepin
PostgreSQL
httpd

7. Monitoring Foreman resources

You can use Foreman web UI to monitor your Foreman environment and hosts. This includes details of hosts, such as configuration or compliance, and information about content and Smart Proxies.

7.1. Using the Foreman content dashboard

The Foreman content dashboard contains various widgets. Use the dashboard to get a quick overview of the resources currently in use in Foreman.

In the Foreman web UI, navigate to Monitor > Dashboard to access the content dashboard. The dashboard can be rearranged by clicking on a widget and dragging it to a different position. The following widgets are available:

Host Configuration Status

An overview of the configuration states and the number of hosts associated with it during the last reporting interval. The following table shows the descriptions of the possible configuration states.

Table 3. Host configuration states
Icon	State	Description
	Hosts that had performed modifications without error	Host that successfully performed modifications during the last reporting interval.
	Hosts in error state	Hosts on which an error was detected during the last reporting interval.
	Good host reports in the last 35 minutes	Hosts without error that did not perform any modifications in the last 35 minutes.
	Hosts that had pending changes	Hosts on which some resources would be applied but Puppet was configured to run in the `noop` mode.
	Out of sync hosts	Hosts that were not synchronized and the report was not received during the last reporting interval.
	Hosts with no reports	Hosts for which no reports were collected during the last reporting interval.
	Hosts with alerts disabled	Hosts which are not being monitored.

Click the particular configuration status to view hosts associated with it.

Host Configuration Chart

A pie chart shows the proportion of the configuration status and the percentage of all hosts associated with it.

Latest Events

A list of messages produced by hosts including administration information, product changes, and any errors.

Monitor this section for global notifications sent to all users and to detect any unusual activity or errors.

Run Distribution (last 30 minutes)

A graph shows the distribution of the running OpenVox agents during the last puppet interval which is 30 minutes by default. In this case, each column represents several reports received from clients during 3 minutes.

New Hosts

A list of the recently created hosts. Click the host for more details.

Task Status

A summary of all current tasks, grouped by their state and result. Click the number to see the list of corresponding tasks.

Latest Warning/Error Tasks

A list of the latest tasks that have been stopped due to a warning or error. Click a task to see more details.

Discovered Hosts

A list of all bare-metal hosts detected on the provisioning network by the Discovery plugin.

Latest Errata

A list of all errata available for hosts registered to Foreman.

Content Views

A list of all content views in Foreman and their publish status.

Sync Overview

An overview of all products or repositories enabled in Foreman and their synchronization status. All products that are in the queue for synchronization, are unsynchronized or have been previously synchronized are listed in this section.

Host Collections

A list of all host collections in Foreman and their status, including the number of hosts in each host collection.

Virt-who Configuration Status

An overview of the status of reports received from the virt-who daemon running on hosts in the environment. The following table shows the possible states.

Table 4. virt-who configuration states
State	Description
No Reports	No report has been received because either an error occurred during the virt-who configuration deployment, or the configuration has not been deployed yet, or virt-who cannot connect to Foreman during the scheduled interval.
No Change	No report has been received because hypervisor did not detect any changes on the virtual machines, or virt-who failed to upload the reports during the scheduled interval. If you added a virtual machine but the configuration is in the No Change state, check that virt-who is running.
OK	The report has been received without any errors during the scheduled interval.
Total Configurations	A total number of virt-who configurations.

Click the configuration status to see all configurations in this state.

The widget also lists the three latest configurations in the No Change state under Latest Configurations Without Change.

Latest Compliance Reports

A list of the latest compliance reports. Each compliance report shows several rules passed (P), failed (F), or othered (O). Click the host for the detailed compliance report. Click the policy for more details on that policy.

Compliance Reports Breakdown

A pie chart shows the distribution of compliance reports according to their status.

Insights Actions

Insights is a tool embedded in Foreman that checks the environment and suggests actions you can take. The actions are divided into 4 categories: Availability, Stability, Performance, and Security.

Insights Risk Summary

A table shows the distribution of the actions according to the risk levels. Risk level represents how critical the action is and how likely it is to cause an actual issue. The possible risk levels are: Low, Medium, High, and Critical.

7.2. Foreman task management

You can monitor and manage all planned or performed tasks, such as repositories synchronized, errata applied, and content views published. This is useful to track progress, troubleshoot issues, or adjust timeout settings for low bandwidth or high latency environments.

Reviewing the log: To review the log, in the Foreman web UI, navigate to Monitor > Foreman Tasks > Tasks.

In the Task window, you can search for specific tasks, view their status, details, and elapsed time since they started. You can also cancel and resume one or more tasks.

The tasks are managed using the Dynflow engine. Remote tasks have a timeout which can be adjusted as needed.

Adjusting timeout settings

In the Foreman web UI, navigate to Administer > Settings.
Enter %_timeout in the search box and click Search. The search should return four settings, including a description.
In the Value column, click the icon next to a number to edit it.
Enter the desired value in seconds, and click Save.

Note

Adjusting the %_finish_timeout values might help in case of low bandwidth. Adjusting the %_accept_timeout values might help in case of high latency.

When a task is initialized, any back-end service that will be used in the task, such as Candlepin or Pulp, will be checked for correct functioning. If the check fails, you will receive an error similar to the following one:

There was an issue with the backend service candlepin: Connection refused – connect(2).

If the back-end service checking feature turns out to be causing any trouble, it can be disabled as follows.

Disabling checking for services

In the Foreman web UI, navigate to Administer > Settings.
Enter check_services_before_actions in the search box and click Search.
In the Value column, click the icon to edit the value.
From the drop-down menu, select false.
Click Save.

7.3. Configuring RSS notifications

You can configure custom RSS feed notifications to receive event alerts from your preferred sources, such as project blogs or internal news feeds, instead of the default notification sources.

To view Foreman event notification alerts, click the Notifications icon in the upper right of the screen.

By default, the Notifications area displays RSS feed events published in the Foreman Blog.

The feed is refreshed every 12 hours and the Notifications area is updated whenever new events become available.

You can configure the RSS feed notifications by changing the URL feed. The supported feed format is RSS 2.0 and Atom. For an example of the RSS 2.0 feed structure, see the Foreman Blog feed. For an example of the Atom feed structure, see the Foreman blog feed.

Procedure

In the Foreman web UI, navigate to Administer > Settings and select the Notifications tab.
In the RSS URL row, click the edit icon in the Value column and type the required URL.
In the RSS enable row, click the edit icon in the Value column to enable or disable this feature.

7.4. Monitoring Foreman server

Audit records list the changes made by all users on Foreman. You can use this information for maintenance and troubleshooting.

Procedure

In the Foreman web UI, navigate to Monitor > Audits to view the audit records.
Obtain a list of all audit attributes:
```
# foreman-rake audits:list_attributes
```

7.5. Monitoring Smart Proxy server

In the Foreman web UI, you can find various information about Smart Proxy server and its services. This information is useful to maintain Smart Proxy server and troubleshoot any issues.

7.5.1. Viewing general Smart Proxy information

You can view general information about your Smart Proxies in the Foreman web UI, such as their status, enabled features, and managed hosts, to verify configuration or troubleshoot connectivity issues.

In the Foreman web UI, navigate to Infrastructure > Smart Proxies to view a table of Smart Proxy servers registered to Foreman server. The information contained in the table answers the following questions:

Is Smart Proxy server running?: This is indicated by a green icon in the Status column. A red icon indicates an inactive Smart Proxy, use the service foreman-proxy restart command on Smart Proxy server to activate it.
What services are enabled on Smart Proxy server?: In the Features column, you can verify if, for example, your Smart Proxy provides a DHCP service or acts as a Pulp mirror. Smart Proxy features can be enabled during installation or configured in addition. For more information, see Installing Smart Proxy server.
What organizations and locations is Smart Proxy server assigned to?: A Smart Proxy server can be assigned to multiple organizations and locations, but only Smart Proxies belonging to the currently selected organization are displayed. To list all Smart Proxies, select Any Organization from the context menu in the top left corner.

After changing the Smart Proxy configuration, select Refresh from the drop-down menu in the Actions column to ensure the Smart Proxy table is up to date.

Click the Smart Proxy name to view further details. At the Overview tab, you can find the same information as in the Smart Proxy table. In addition, you can answer to the following questions:
Which hosts are managed by Smart Proxy server?: The number of associated hosts is displayed next to the Hosts managed label. Click the number to view the details of associated hosts.
How much storage space is available on Smart Proxy server?: The amount of storage space occupied by the Pulp content in /var/lib/pulp is displayed. Also the remaining storage space available on the Smart Proxy can be ascertained.

7.5.2. Monitoring Smart Proxy services

You can view the status and configuration details of Smart Proxy services in Foreman web UI. This helps you verify that your Smart Proxy services are running correctly and troubleshoot any issues.

Procedure

In the Foreman web UI, navigate to Infrastructure > Smart Proxies and click the name of the selected Smart Proxy.
- At the Services tab, you can find basic information on Smart Proxy services, such as the list of DNS domains, or the number of Pulp workers.
  
  The appearance of the page depends on what services are enabled on Smart Proxy server. Services providing more detailed status information can have dedicated tabs at the Smart Proxy page. For more information, see Monitoring Puppet on Smart Proxy.

7.5.3. Monitoring Puppet on Smart Proxy

You can monitor Puppet activity and certificate management for each Smart Proxy.

Note	The Puppet and Puppet CA tabs are available only if you have Puppet enabled in your Foreman.

Procedure

In the Foreman web UI, navigate to Infrastructure > Smart Proxies.
Click the name of your Smart Proxy.
- At the Puppet tab you can find the following:
  - A summary of Puppet events, an overview of latest Puppet runs, and the synchronization status of associated hosts at the General sub-tab.
  - A list of Puppet environments at the Environments sub-tab.
- At the Puppet CA tab you can find the following:
  - A certificate status overview and the number of autosign entries at the General sub-tab.
  - A table of CA certificates associated with the Smart Proxy at the Certificates sub-tab. Here you can inspect the certificate expiry data, or cancel the certificate by clicking Revoke.
  - A list of autosign entries at the Autosign entries sub-tab. Here you can create an entry by clicking New or delete one by clicking Delete.

Additional resources

Enabling Puppet integration with Foreman in Configuring hosts by using Puppet

8. Using webhooks in Foreman

A webhook is a way for a web page or web application to provide other applications with information in real time. You can use webhooks to define a call to an external API based on Foreman internal event by using a fire-and-forget message exchange pattern.

8.1. Webhook behavior and integration

You can use webhooks as a way to integrate Foreman with external systems. For example, webhooks can integrate with monitoring systems to send alerts when a host is created or deleted.

Webhooks are only triggered after an event occurs. The request usually contains details of the event. An event triggers callbacks, such as sending an e-mail confirming a host has been provisioned. The application sending the request does not wait for the response, or ignores it.

Because webhooks use HTTP, no new infrastructure needs be added to existing web services. Webhooks are useful where the action you want to perform in the external system can be achieved through its API.

Payload of a webhook is created from webhook templates. Webhook templates use the same ERB syntax as Provisioning templates. Available variables:

@event_name: Name of an event.
@webhook_id: Unique event ID.
@payload: Payload data, different for each event type. To access individual fields, use @payload[:key_name] Ruby hash syntax.
@payload[:object]: Database object for events triggered by database actions (create, update, delete). Not available for custom events.
@payload[:context]: Additional information as hash like request and session UUID, remote IP address, user, organization and location.

Additional resources

Available webhook events

8.2. Installing the webhooks plugin

Use the following procedure to install the webhooks plugin. Then, you can configure Foreman server to send webhook requests.

Procedure

On your Foreman server, install the webhooks plugin:
```
# foremanctl deploy --add-feature webhooks
```

8.3. Creating a webhook template

Webhook templates are used to generate the body of HTTP request to a configured target when a webhook is triggered. You can create a webhook template in the Foreman web UI.

Procedure

In the Foreman web UI, navigate to Administer > Webhook > Webhook Templates.
Click Clone an existing template or Create Template.
Enter a name for the template.
Use the editor to make changes to the template payload.

A webhook HTTP payload must be created using Foreman template syntax. The webhook template can use a special variable called @object that can represent the main object of the event. @object can be missing in case of certain events. You can determine what data are actually available with the @payload variable.

For more information, see Template Writing Reference in Managing hosts and for available template macros and methods, visit /templates_doc on Foreman server.
Optional: Enter the description and audit comment.
Assign organizations and locations.
Click Submit.

Example 1. Creating a webhook template

When creating a webhook template, you must follow the format of the target application for which the template is intended. For example, an application can expect a "text" field with the webhook message. Refer to the documentation of your target application to find more about how your webhook template format should look like.

Running remote execution jobs

This webhook template defines a message with the ID and result of a remote execution job. The webhook which uses this template can be subscribed to events such as Actions Remote Execution Run Host Job Succeeded or Actions Remote Execution Run Host Job Failed.

{
    "text": "job invocation <%= @object.job_invocation_id %> finished with result <%= @object.task.result %>"
}

Creating users

This webhook template defines a message with the login and email of a created user. The webhook which uses this template should be subscribed to the User Created event.

{
    "text": "user with login <%= @object.login %> and email <%= @object.mail %> created"
}

8.4. Creating a webhook

When creating a webhook in the Foreman web UI, you can customize events, payloads, HTTP authentication, content type, and headers.

Procedure

In the Foreman web UI, navigate to Administer > Webhook > Webhooks.
Click Create new.
From the Subscribe to list, select an event.
Enter a Name for your webhook.
Enter a Target URL. Webhooks make HTTP requests to pre-configured URLs. The target URL can be a dynamic URL.
Click Template to select a template. Webhook templates are used to generate the body of the HTTP request to Foreman server when a webhook is triggered.
Enter an HTTP method.
Optional: If you do not want activate the webhook when you create it, uncheck the Enabled flag.
Click the Credentials tab.
Optional: If HTTP authentication is required, enter User and Password.
Optional: Uncheck Verify SSL if you do not want to verify the server certificate against the system certificate store or Foreman CA.
On the Additional tab, enter the HTTP Content Type. For example, application/json, application/xml or text/plain on the payload you define. The application does not attempt to convert the content to match the specified content type.
Optional: Provide HTTP headers as JSON. ERB is also allowed.

8.5. Available webhook events

Review the webhook events that are available from the Foreman web UI.

Action events trigger webhooks only on success, so if an action fails, a webhook is not triggered.

For more information about payload, go to Administer > About > Support > Templates DSL. A list of available types is provided in the following table. Some events are marked as custom, in that case, the payload is an object object but a Ruby hash (key-value data structure) so syntax is different.

Event name Description Payload

Event name	Description	Payload
`Actions Katello Content View Promote Succeeded`	A content view was successfully promoted.	`Actions::Katello::ContentView::Promote`
`Actions Katello Content View Publish Succeeded`	A repository was successfully synchronized.	`Actions::Katello::ContentView::Publish`
`Actions Remote Execution Run Host Job Succeeded`	A generic remote execution job succeeded for a host. This event is emitted for all Remote Execution jobs, when complete.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Errata Install Succeeded`	Install errata using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Group Install Succeeded`	Install package group using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Package Install Succeeded`	Install package using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Group Remove`	Remove package group using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Package Remove Succeeded`	Remove package using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Service Restart Succeeded`	Restart Services using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Group Update Succeeded`	Update package group using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Package Update Succeeded`	Update package using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Puppet Run Host Succeeded`	Perform a single Puppet run.	`Actions::RemoteExecution::RunHostJob`
`Actions Remote Execution Run Host Job Katello Module Stream Action Succeeded`	Perform a module stream action using the Katello interface.	`Actions::RemoteExecution::RunHostJob`
`Build Entered`	A host entered the build mode.	Custom event: `@payload[:id]` (host id), `@payload[:hostname]` (host name).
`Build Exited`	A host build mode was canceled, either it was successfully provisioned or the user canceled the build manually.	Custom event: `@payload[:id]` (host id), `@payload[:hostname]` (host name).
`Config Report Created/Updated/Destroyed`	Common database operations on a configuration report.	`ConfigReport`
`Content View Created/Updated/Destroyed`	Common database operations on a content view.	`Katello::ContentView`
`Domain Created/Updated/Destroyed`	Common database operations on a domain.	`Domain`
`Host Created/Updated/Destroyed`	Common database operations on a host.	`Host`
`Hostgroup Created/Updated/Destroyed`	Common database operations on a hostgroup.	`Hostgroup`
`Model Created/Updated/Destroyed`	Common database operations on a model.	`Model`
`Status Changed`	Global host status of a host changed.	Custom event: `@payload[:id]` (host id), `@payload[:hostname]`, `@payload[:global_status]` (hash)
`Subnet Created/Updated/Destroyed`	Common database operations on a subnet.	`Subnet`
`Template Render Performed`	A report template was rendered.	`Template`
`User Created/Updated/Destroyed`	Common database operations on a user.	`User`

Actions Katello Content View Promote Succeeded

A content view was successfully promoted.

Actions::Katello::ContentView::Promote

Actions Katello Content View Publish Succeeded

A repository was successfully synchronized.

Actions::Katello::ContentView::Publish

Actions Remote Execution Run Host Job Succeeded

A generic remote execution job succeeded for a host. This event is emitted for all Remote Execution jobs, when complete.