1. Introduction to performance tuning

Tune Foreman to improve system response times and handle larger numbers of managed hosts. Performance tuning helps optimize resource usage and ensures smooth operations as your infrastructure scales.

You can tune Foreman in three primary scenarios:

  • Selecting the appropriate deployment size during initial installation

  • Optimizing an existing deployment to handle increased load

  • Troubleshooting performance degradation

Foreman includes several components that you can tune independently to achieve optimal performance for your environment. You can tune at the operating system level (tuned profiles, transparent hugepage settings), application level (Puma workers and threads, Apache httpd processes, Dynflow orchestrator, remote execution transport), database level (PostgreSQL configuration), and caching level (Redis configuration).

Before tuning your deployment, review the hardware and software requirements in Preparing your environment for Foreman server installation in Installing Foreman Server with Katello 5.0 plugin on Enterprise Linux. Most configuration changes require a Foreman restart, so test all tuning changes in a staging environment and set up monitoring to evaluate the effectiveness of changes in your specific environment.

Your environment may differ from the testing environment used to develop these recommendations, so results may vary. Monitor your deployment health after configuration changes. Foreman provides a health check command to verify system status.

2. Selecting an installation size

You can tune your Foreman server based on expected host counts and hardware allocation by using built-in tuning profiles included in Foreman that are available by using the installation routine’s tuning flag.

There are four sizes provided based on estimates of the number of hosts your Foreman manages. You can find the specific tuning settings for each profile in the configuration files contained in /usr/share/foreman-installer/config/foreman.hiera/tuning/sizes.

Name Number of hosts Recommend RAM Recommend Cores

default

0 – 5000

20 GiB

4

medium

5000 – 10000

32 GiB

8

large

10000 – 20000

64 GiB

16

extra-large

20000 – 60000

128 GiB

32

extra-extra-large

60000+

256 GiB+

48+
Procedure

  1. Select an installation size: default, medium, large, extra-large, or extra-extra-large. The default value is default.

  2. Run foreman-installer:

    # foreman-installer --tuning "My_Installation_Size"

  3. Optional: Run a health check. For more information, see Introduction to performance tuning.

  4. Optional: Refer to Puma application server tunings to tune the Ruby app server directly.

Additional resources

3. Tuning hardware and operating system

You can tune the hardware and operating system to optimize Foreman performance for your scenario.

CPU

The more physical cores that are available to Foreman, the higher throughput can be achieved for the tasks. Some of the Foreman components such as Puppet and PostgreSQL are CPU intensive applications and can really benefit from the higher number of available CPU cores.

Memory

The higher amount of memory available in the system running Foreman, the better will be the response times for the Foreman operations. Since Foreman uses PostgreSQL as the database solutions, any additional memory coupled with the tunings will provide a boost to the response times of the applications due to increased data retention in the memory.

Disk

Foreman should be deployed using SSDs to avoid performance bottlenecks. Components such as PostgreSQL benefit from using SSDs due to their lower latency compared to HDDs.

Network

The communication between the Foreman server and Smart Proxies is impacted by the network performance. A decent network with a minimum jitter and low latency is required to enable hassle free operations such as Foreman server and Smart Proxies synchronization. At the very least, ensure it is not causing connection resets and similar problems.

Server Power Management

Your server by default is likely to be configured to conserve power. While this is a good approach to keep the max power consumption in check, it also has a side effect of lowering the performance that Foreman can achieve. For a server running Foreman, it is recommended to set the BIOS to enable the system to be run in performance mode to boost the maximum performance levels that Foreman can achieve.

3.1. Enabling tuned profiles

Enable tuned profiles to automatically optimize Foreman performance settings. Tuned profiles apply optimizations tailored to your environment.

Procedure

  1. Install tuned on your Smart Proxy:

    # dnf install tuned

  2. Check if tuned is running:

    # systemctl status tuned

  3. If tuned is not running, enable it:

    # systemctl enable --now tuned

  4. Optional: View a list of available tuned profiles:

    # tuned-adm list

  5. Enable a tuned profile depending on your scenario:

    # tuned-adm profile "My_Tuned_Profile"

3.2. Transparent Hugepage and database performance

Transparent Hugepage can reduce database performance in Foreman. PostgreSQL and Redis workloads perform poorly when this feature is enabled.

Transparent Hugepage is a memory management technique used by the Linux kernel to reduce the overhead of using the Translation Lookaside Buffer (TLB) by using larger sized memory pages. Due to databases having Sparse Memory Access patterns instead of Contiguous Memory access patterns, database workloads often perform poorly when Transparent Hugepage is enabled.

In deployments with databases running on separate servers, the benefit of using Transparent Hugepage on the Foreman server might be limited.

Additional resources

4. Puma application server tunings

Puma handles Foreman web requests and API calls. Tuning Puma optimizes host operations and API handling for large deployments.

4.1. Puma workers and threads auto-tuning

By default, Puma automatically configures workers and threads based on available CPU and memory. Auto-tuning works well for most deployments, but specific workload patterns may require manual tuning to optimize performance or resource allocation.

If you do not specify Puma workers and thread values, the foreman-installer sets a balanced number of workers using this formula:

min(CPU_COUNT * 1.5, RAM_IN_GB - 1.5)

This should be fine for most cases, but with some usage patterns, tuning is needed to either limit the amount of resources dedicated to Puma so other Foreman components can use them. Each Puma worker consumes around 1 GiB of RAM.

View your current Foreman server settings
# cat /etc/systemd/system/foreman.service.d/installer.conf
View the currently active Puma workers
# systemctl status foreman

4.2. Manually tuning Puma workers and thread count

Manual tuning allows you to override auto-tuning when your specific workload or hardware configuration requires different worker and thread counts than the defaults provide.

For example, the following configuration uses two workers and five threads:

# foreman-installer \
--foreman-foreman-service-puma-workers 2 \
--foreman-foreman-service-puma-threads-max 5

4.3. Puma workers and threads recommendations

These recommendations are based on performance testing with concurrent host registrations across different deployment sizes. Use these test results to make informed decisions about worker and thread counts for your specific deployment size and workload patterns.

The main test was concurrent registration with different numbers of workers and threads. These recommendations focus on concurrent registration performance and might not match your exact use case. For example, if your setup focuses on content with many publishes and promotes, limit Puma resources to favor Pulp and PostgreSQL.

Name Number of hosts RAM Cores Recommended Puma threads Recommended Puma workers

default

0 – 5000

20 GiB

4

16

4 – 6

medium

5000 – 10000

32 GiB

8

16

8 – 12

large

10000 – 20000

64 GiB

16

16

12 – 18

extra-large

20000 – 60000

128 GiB

32

16

16 – 24

extra-extra-large

60000+

256 GiB+

48+

16

20 – 26

Worker count is more important than thread count. We have seen up to 52% performance increase by adjusting workers. Although the installer uses 5 threads by default, we recommend 16 threads with all the tuning profiles in the table above. We have seen up to 23% performance increase with 16 threads (14% for 8 and 10% for 32) compared to 4 threads.

These recommendations use concurrent registrations as the test case. Your Foreman might have different workloads, such as more balanced operations beyond registrations. The default 5 threads is also a good choice.

These are some of our measurements that lead us to these recommendations:

4 workers, 4 threads 4 workers, 8 threads 4 workers, 16 threads 4 workers, 32 threads

Improvement

0%

14%

23%

10%

Use 4 – 6 workers on a default setup (4 CPUs). We have seen 25% better performance with 5 workers versus 2 workers. However, 8 workers performed 8% worse than 2 workers. See the table below:

2 workers, 16 threads 4 workers, 16 threads 6 workers, 16 threads 8 workers, 16 threads

Improvement

0%

26%

22%

-8%

Use 8 – 12 workers on a medium setup (8 CPUs) – see table below:

2 workers, 16 threads 4 workers, 16 threads 8 workers, 16 threads 12 workers, 16 threads 16 workers, 16 threads

Improvement

0%

51%

52%

52%

42%

Use 16 – 24 workers on a 32 CPU setup. This was tested on a Foreman deployment with 90 GiB of RAM. Memory became a factor as the system started swapping. An extra-large setup should have 128 GiB. More workers caused problems at higher registration concurrency levels, so we do not recommend it.

4 workers, 16 threads 8 workers, 16 threads 16 workers, 16 threads 24 workers, 16 threads 32 workers, 16 threads 48 workers, 16 threads

Improvement

0%

37%

44%

52%

too many failures

too many failures

4.4. Puma workers configuration

Worker count determines how many parallel processes handle incoming requests. The relationship between CPU resources and worker count directly impacts Foreman performance scaling, particularly for concurrent operations like host registrations.

If you have enough CPUs, adding more workers improves performance. For example, we have compared Foreman setups with 8 and 16 CPUs:

Table 1. foreman-installer options used to test effect of workers count
Foreman VM with 8 CPUs, 40 GiB RAM Foreman VM with 16 CPUs, 40 GiB RAM

--foreman-foreman-service-puma-threads-max 16

--foreman-foreman-service-puma-threads-max 16

--foreman-foreman-service-puma-workers {2|4|8|16}

--foreman-foreman-service-puma-workers {2|4|8|16}

In an 8 CPU setup, changing workers from 2 to 16 improved concurrent registration time by 36%. In a 16 CPU setup, the same change improved it by 55%.

More workers also increase total registration concurrency. In our tests, 2 workers handled up to 480 concurrent registrations. More workers improved this.

4.5. Puma threads configuration

Thread count determines how many parallel operations each worker process can handle. The relationship between threads and workers impacts resource utilization and parallel request handling without increasing memory consumption.

For example, we have compared these two setups:

Foreman VM with 8 CPUs, 40 GiB RAM Foreman VM with 8 CPUs, 40 GiB RAM

--foreman-foreman-service-puma-threads-max 16

--foreman-foreman-service-puma-threads-max 8

--foreman-foreman-service-puma-workers 2

--foreman-foreman-service-puma-workers 4

Using more workers with the same total threads gives about 11% speedup for highly concurrent registrations. More workers do not use more CPU or RAM but provide better performance.

5. Tuning Apache HTTPD performance

Tune Apache httpd to improve Foreman performance and increase concurrency for Foreman web UI and API requests.

Apache httpd is a core component of Foreman. It handles web requests from the Foreman web UI and APIs.

5.1. Tuning Apache httpd child processes

Tune Apache httpd child processes to handle more concurrent requests and prevent HTTP 503 errors during high load.

By default, httpd uses the event request handling mechanism. When requests exceed the maximum number of child processes, httpd returns an HTTP 503 Service Unavailable error. When httpd runs out of processes, some Foreman components may fail because they depend on httpd processes.

You can adjust httpd event settings to handle more concurrent requests based on your expected peak load.

Warning

Configuring these numbers in custom-hiera.yaml locks them. If you change these numbers using foreman-installer --tuning=My_Tuning_Option, your custom-hiera.yaml will overwrite this setting. Set your numbers only if you have specific need for it.
Procedure

  1. Modify the number of concurrent requests in /etc/foreman-installer/custom-hiera.yaml by changing or adding the following lines:

    apache::mod::event::serverlimit: 64
apache::mod::event::maxrequestworkers: 1024

  2. Reconfigure your Foreman:

    # foreman-installer

    The example is identical to running foreman-installer --tuning=medium or higher on Foreman server.

6. Tuning Dynflow services

When many hosts report to Foreman and run concurrent tasks, increase Dynflow workers to improve task execution performance.

Dynflow is the workflow management system and task orchestrator for Foreman. It executes tasks in an out-of-order manner. When many hosts report to Foreman and run tasks, tune Dynflow by setting how many executors it can launch.

For more information about the tunings involved related to Dynflow, see https://foreman.example.com/foreman_tasks/sidekiq.

Foreman contains a Dynflow service called dynflow-sidekiq that runs tasks scheduled by Dynflow. Sidekiq workers can be grouped into queues so many tasks of one type do not block tasks of another type.

Foreman community recommends increasing sidekiq workers to scale the Foreman tasking system for bulk concurrent tasks. Typical tasks are multiple content view publications and promotions, content synchronizations, and synchronizations to Smart Proxy servers. There are two options:

  • Increase the number of threads used by a worker (worker concurrency). This has limited impact for values larger than five due to how Ruby implements thread concurrency.

  • Increase the number of workers (recommended).

Procedure

  1. Increase the number of workers from one worker to three while remaining five threads/concurrency of each:

    # foreman-installer --foreman-dynflow-worker-instances 3    # optionally, add --foreman-dynflow-worker-concurrency 5

  2. Optional: Check if there are three worker services:

    # systemctl -a | grep dynflow-sidekiq@worker-[0-9]
dynflow-sidekiq@worker-1.service        loaded    active   running   Foreman jobs daemon - worker-1 on sidekiq
dynflow-sidekiq@worker-2.service        loaded    active   running   Foreman jobs daemon - worker-2 on sidekiq
dynflow-sidekiq@worker-3.service        loaded    active   running   Foreman jobs daemon - worker-3 on sidekiq

7. Tuning pull-based REX transport

You can adjust MQTT broker settings and client polling intervals to optimize pull-based remote execution performance for the scale and network conditions of your environment.

Additional resources

7.1. Increasing host limit for pull-based REX transport

Tune the mosquitto MQTT server on Smart Proxy to support more than 1024 connected hosts for larger deployments.

Prerequisites
Procedure

  1. On your Smart Proxy, set the upper limit of connected hosts for pull-based remote execution in /etc/foreman-installer/custom-hiera.yaml:

    systemd::dropin_files:
  limits.conf:
    unit: mosquitto.service
    content: "[Service]\nLimitNOFILE=5000\n"

    This example configures the mosquitto service on your Smart Proxy to handle up to 5000 hosts.

  2. Re-run the installer for the changes to take effect:

    # foreman-installer

7.2. Decreasing performance impact of the pull-based REX transport

Configure MQTT rate limiting and timing parameters to prevent connection exhaustion on Smart Proxy server when running high-concurrency remote execution jobs in pull-based mode.

When Foreman server is configured with the pull-based transport mode for remote execution jobs by using the Script provider, Smart Proxy server sends notifications about new jobs to hosts through MQTT. This notification does not include the actual workload that the host is supposed to execute. After a host receives a notification about a new remote execution job, it queries Smart Proxy server for its actual workload. During the job, the host periodically sends outputs of the job to Smart Proxy server, further increasing the number of requests to Smart Proxy server.

These requests to Smart Proxy server together with high concurrency allowed by the MQTT protocol can cause exhaustion of available connections on Smart Proxy server. Some requests might fail, making some child tasks of remote execution jobs unresponsive. This also depends on actual job workload, as some jobs are causing additional load to Foreman server, making it compete for resources if hosts are registered to Foreman server.

To avoid this, configure your Foreman server and Smart Proxy server with the following parameters:

  • MQTT Time To Live – Time interval in seconds given to the host to pick up the job before considering the job undelivered

  • MQTT Resend Interval – Time interval in seconds at which the notification should be re-sent to the host until the job is picked up or canceled

  • MQTT Rate Limit – Number of jobs that are allowed to run at the same time. You can limit the concurrency of remote execution by tuning the rate limit which means you are going to put more load on Foreman.

Smart Proxy server uses Webrick HTTP server (no httpd or Puma involved), so there is no simple way to increase its capacity.

Procedure

  • Tune the MQTT parameters on your Foreman server:

    # foreman-installer \
--foreman-proxy-plugin-remote-execution-script-mqtt-rate-limit My_MQTT_Rate_Limit \
--foreman-proxy-plugin-remote-execution-script-mqtt-resend-interval My_MQTT_Resend_Interval \
--foreman-proxy-plugin-remote-execution-script-mqtt-ttl My_MQTT_Time_To_Live

8. Tuning the PostgreSQL database

Configure PostgreSQL database settings to improve response times. Tuning PostgreSQL reduces query latency and improves system responsiveness, especially with high database activity.

The PostgreSQL authors recommend disabling Transparent Hugepage on servers running PostgreSQL.

Additional resources

8.1. Setting PostgreSQL tuning parameters

Tune PostgreSQL by modifying the postgresql.conf file. This works regardless of the tuning profile.

Procedure

  1. Append /etc/foreman-installer/custom-hiera.yaml to tune PostgreSQL:

    postgresql::server::config_entries:
  max_connections: 1000
  shared_buffers: 2GB
  work_mem: 8MB
  autovacuum_vacuum_cost_limit: 2000

    This tuning configuration alters the following keys:

    max_connections

    Maximum number of connections PostgreSQL can accept.

    shared_buffers

    Memory used by all active connections to store data for database operations. Set this between 2 GiB to 25% of total system memory depending on database operation frequency.

    work_mem

    Memory allocated per process to store temporary results of database operations. Set this to 8 MB for most heavy operations.

    autovacuum_vacuum_cost_limit

    Cost limit for the vacuuming operation that cleans up dead tuples in database relations. This defines how many tuples can be processed in a single run. Foreman community recommends 2000 for the medium, large, extra-large, and extra-extra-large profiles.

  2. Reconfigure your Foreman:

    # foreman-installer

8.2. Benchmarking raw database performance

Use pgbench to measure raw PostgreSQL performance and validate storage behavior on your Foreman environment. Benchmarking helps you establish baseline performance metrics and verify that tuning changes improve database responsiveness.

Prerequisites

  • A non-production system with a valid backup.

Warning

  • Never run benchmarks on a production system or without a valid backup.

  • Before you start testing, determine how large the database files are. Testing with a very small database does not produce meaningful results. For example, if the database is only 20 GiB and the buffer pool is 32 GiB, the benchmark does not show problems with a large number of connections because the data fits entirely in the buffer pool.
Procedure

  1. To identify the largest tables by disk space for Candlepin, Foreman, and Pulp, use the postgres-size-report script in the satellite-support Git repository. For more information, see github.com/RedHatSatellite/satellite-support.

  2. Ensure the PostgreSQL data directory /var/lib/pgsql has enough space for the benchmark workload. You might need to resize the volume to 100 GiB or more, depending on how long the benchmark runs and how much data it generates.

    Note

    Choice of filesystem for PostgreSQL data directory might matter as well.

  3. Install the PGBench utility:

    # dnf install postgresql-contrib

9. Tuning the Redis cache

Understanding Redis tuning helps you improve caching and task tracking performance. Proper Redis setup keeps memory use stable and improves Foreman responsiveness.

Redis is an in-memory data store. Foreman and Pulp use Redis for caching and Dynflow uses Redis to track its tasks. Given the way Foreman uses Redis, its memory consumption should be stable.

The Redis authors recommend disabling Transparent Hugepage on servers running Redis.

Additional resources

10. Smart Proxy configuration tuning

Smart Proxy tuning distributes load to deliver content across networks. Properly configured Smart Proxies reduce Foreman server load and improve remote execution performance.

Smart Proxies do not improve performance for operations that extensively use the Foreman API, such as host registration or package profile update.

10.1. Smart Proxy performance tests

Review performance test results for different Smart Proxy hardware configurations to determine the optimal sizing for your deployment. Choose the right resources based on your expected workload for content delivery, host registration, and remote execution tasks.

Foreman community tested the following configurations:

Smart Proxy HW configuration CPUs RAM

minimal

4

12 GiB

large

8

24 GiB

extra large

16

46 GiB
Content delivery use case

In a download test where we concurrently downloaded a 40MB repo of 2000 packages on 100, 200, .. 1000 hosts, we saw roughly 50% improvement in average download duration every time when we doubled Smart Proxy server resources. For more precise numbers, see the table below.

Concurrent downloading hosts Minimal (4 CPU and 12 GiB RAM) → Large (8 CPU and 24 GiB RAM) Large (8 CPU and 24 GiB RAM) → Extra Large (16 CPU and 46 GiB RAM) Minimal (4 CPU and 12 GiB RAM) → Extra Large (16 CPU and 46 GiB RAM)

Average Improvement

~ 50% (e.g. for 700 concurrent downloads in average 9 seconds vs. 4.4 seconds per package)

~ 40% (e.g. for 700 concurrent downloads in average 4.4 seconds vs. 2.5 seconds per package)

~ 70% (e.g. for 700 concurrent downloads in average 9 seconds vs. 2.5 seconds per package)

When we compared download performance from Foreman server vs. from Smart Proxy server, we have seen only about 5% speedup, but that is expected as Smart Proxy server’s main benefit is in getting content closer to geographically distributed clients (or clients in different networks) and in handling part of the load Foreman server would have to handle itself. In some smaller hardware configurations (8 CPUs and 24 GiB), Foreman server was not able to handle downloads from more than 500 concurrent clients, while a Smart Proxy server with the same hardware configuration was able to service more than 1000 and possibly even more.

Concurrent registrations use case

For concurrent registrations, a bottleneck is usually CPU speed, but all configs were able to handle even high concurrency without swapping. Hardware resources used for Smart Proxy have only minimal impact on registration performance. For example, Smart Proxy server with 16 CPUs and 46 GiB RAM have at most a 9% registration speed improvement when compared to a Smart Proxy server with 4 CPUs and 12 GiB RAM. During periods of very high concurrency, you might experience timeouts in the Smart Proxy server to Foreman server communication. You can alleviate this by increasing the default timeout by using the following tunable in /etc/foreman-installer/custom-hiera.yaml:

apache::mod::proxy::proxy_timeout: 600
Remote execution use case

We have tested executing remote execution jobs by using both SSH and Ansible backend on 500, 2000 and 4000 hosts. All configurations were able to handle all of the tests without errors, except for the smallest configuration (4 CPUs and 12 GiB memory) which failed to finish on all 4000 hosts.

Report issue