1. Introduction to performance tuning
Tune Foreman to improve system response times and handle larger numbers of managed hosts. Performance tuning helps optimize resource usage and ensures smooth operations as your infrastructure scales.
You can tune Foreman in three primary scenarios:
-
Selecting the appropriate deployment size during initial installation
-
Optimizing an existing deployment to handle increased load
-
Troubleshooting performance degradation
Foreman includes several components that you can tune independently to achieve optimal performance for your environment. You can tune at the operating system level (tuned profiles, transparent hugepage settings), application level (Puma workers and threads, Apache httpd processes, Dynflow orchestrator, remote execution transport), database level (PostgreSQL configuration), and caching level (Redis configuration).
Before tuning your deployment, review the hardware and software requirements in Preparing your environment for Foreman server installation in Installing Foreman Server 5.0 on Enterprise Linux. Most configuration changes require a Foreman restart, so test all tuning changes in a staging environment and set up monitoring to evaluate the effectiveness of changes in your specific environment.
Your environment may differ from the testing environment used to develop these recommendations, so results may vary. Monitor your deployment health after configuration changes. Foreman provides a health check command to verify system status.
2. Tuning hardware and operating system
You can tune the hardware and operating system to optimize Foreman performance for your scenario.
- CPU
-
The more physical cores that are available to Foreman, the higher throughput can be achieved for the tasks. Some of the Foreman components such as Puppet and PostgreSQL are CPU intensive applications and can really benefit from the higher number of available CPU cores.
- Memory
-
The higher amount of memory available in the system running Foreman, the better will be the response times for the Foreman operations. Since Foreman uses PostgreSQL as the database solutions, any additional memory coupled with the tunings will provide a boost to the response times of the applications due to increased data retention in the memory.
- Disk
-
Foreman should be deployed using SSDs to avoid performance bottlenecks. Components such as PostgreSQL benefit from using SSDs due to their lower latency compared to HDDs.
- Network
-
The communication between the Foreman server and Smart Proxies is impacted by the network performance. A decent network with a minimum jitter and low latency is required to enable hassle free operations such as Foreman server and Smart Proxies synchronization. At the very least, ensure it is not causing connection resets and similar problems.
- Server Power Management
-
Your server by default is likely to be configured to conserve power. While this is a good approach to keep the max power consumption in check, it also has a side effect of lowering the performance that Foreman can achieve. For a server running Foreman, it is recommended to set the BIOS to enable the system to be run in performance mode to boost the maximum performance levels that Foreman can achieve.
2.1. Enabling tuned profiles
Enable tuned profiles to automatically optimize Foreman performance settings. Tuned profiles apply optimizations tailored to your environment.
-
Install
tunedon your Smart Proxy:# dnf install tuned
-
Check if
tunedis running:# systemctl status tuned
-
If
tunedis not running, enable it:# systemctl enable --now tuned
-
Optional: View a list of available
tunedprofiles:# tuned-adm list
-
Enable a
tunedprofile depending on your scenario:# tuned-adm profile "My_Tuned_Profile"
2.2. Transparent Hugepage and database performance
Transparent Hugepage can reduce database performance in Foreman. PostgreSQL and Redis workloads perform poorly when this feature is enabled.
Transparent Hugepage is a memory management technique used by the Linux kernel to reduce the overhead of using the Translation Lookaside Buffer (TLB) by using larger sized memory pages. Due to databases having Sparse Memory Access patterns instead of Contiguous Memory access patterns, database workloads often perform poorly when Transparent Hugepage is enabled.
In deployments with databases running on separate servers, the benefit of using Transparent Hugepage on the Foreman server might be limited.
3. Puma application server tunings
Puma handles Foreman web requests and API calls. Tuning Puma optimizes host operations and API handling for large deployments.
3.1. Puma workers and threads auto-tuning
By default, Puma automatically configures workers and threads based on available CPU and memory. Auto-tuning works well for most deployments, but specific workload patterns may require manual tuning to optimize performance or resource allocation.
If you do not specify Puma workers and thread values, the foreman-installer sets a balanced number of workers using this formula:
min(CPU_COUNT * 1.5, RAM_IN_GB - 1.5)
This should be fine for most cases, but with some usage patterns, tuning is needed to either limit the amount of resources dedicated to Puma so other Foreman components can use them. Each Puma worker consumes around 1 GiB of RAM.
# cat /etc/systemd/system/foreman.service.d/installer.conf
# systemctl status foreman
3.2. Manually tuning Puma workers and thread count
Manual tuning allows you to override auto-tuning when your specific workload or hardware configuration requires different worker and thread counts than the defaults provide.
For example, the following configuration uses two workers and five threads:
# foreman-installer \ --foreman-foreman-service-puma-workers 2 \ --foreman-foreman-service-puma-threads-max 5
3.3. Puma workers and threads recommendations
These recommendations are based on performance testing with concurrent host registrations across different deployment sizes. Use these test results to make informed decisions about worker and thread counts for your specific deployment size and workload patterns.
The main test was concurrent registration with different numbers of workers and threads. These recommendations focus on concurrent registration performance and might not match your exact use case.
| Name | Number of hosts | RAM | Cores | Recommended Puma threads | Recommended Puma workers |
|---|---|---|---|---|---|
default |
0 – 5000 |
20 GiB |
4 |
16 |
4 – 6 |
medium |
5000 – 10000 |
32 GiB |
8 |
16 |
8 – 12 |
large |
10000 – 20000 |
64 GiB |
16 |
16 |
12 – 18 |
extra-large |
20000 – 60000 |
128 GiB |
32 |
16 |
16 – 24 |
extra-extra-large |
60000+ |
256 GiB+ |
48+ |
16 |
20 – 26 |
Worker count is more important than thread count. We have seen up to 52% performance increase by adjusting workers. Although the installer uses 5 threads by default, we recommend 16 threads with all the tuning profiles in the table above. We have seen up to 23% performance increase with 16 threads (14% for 8 and 10% for 32) compared to 4 threads.
These recommendations use concurrent registrations as the test case. Your Foreman might have different workloads, such as more balanced operations beyond registrations. The default 5 threads is also a good choice.
These are some of our measurements that lead us to these recommendations:
| 4 workers, 4 threads | 4 workers, 8 threads | 4 workers, 16 threads | 4 workers, 32 threads | |
|---|---|---|---|---|
Improvement |
0% |
14% |
23% |
10% |
Use 4 – 6 workers on a default setup (4 CPUs). We have seen 25% better performance with 5 workers versus 2 workers. However, 8 workers performed 8% worse than 2 workers. See the table below:
| 2 workers, 16 threads | 4 workers, 16 threads | 6 workers, 16 threads | 8 workers, 16 threads | |
|---|---|---|---|---|
Improvement |
0% |
26% |
22% |
-8% |
Use 8 – 12 workers on a medium setup (8 CPUs) – see table below:
| 2 workers, 16 threads | 4 workers, 16 threads | 8 workers, 16 threads | 12 workers, 16 threads | 16 workers, 16 threads | |
|---|---|---|---|---|---|
Improvement |
0% |
51% |
52% |
52% |
42% |
Use 16 – 24 workers on a 32 CPU setup. This was tested on a Foreman deployment with 90 GiB of RAM. Memory became a factor as the system started swapping. An extra-large setup should have 128 GiB. More workers caused problems at higher registration concurrency levels, so we do not recommend it.
| 4 workers, 16 threads | 8 workers, 16 threads | 16 workers, 16 threads | 24 workers, 16 threads | 32 workers, 16 threads | 48 workers, 16 threads | |
|---|---|---|---|---|---|---|
Improvement |
0% |
37% |
44% |
52% |
too many failures |
too many failures |
3.4. Puma workers configuration
Worker count determines how many parallel processes handle incoming requests. The relationship between CPU resources and worker count directly impacts Foreman performance scaling, particularly for concurrent operations like host registrations.
If you have enough CPUs, adding more workers improves performance. For example, we have compared Foreman setups with 8 and 16 CPUs:
| Foreman VM with 8 CPUs, 40 GiB RAM | Foreman VM with 16 CPUs, 40 GiB RAM |
|---|---|
|
|
|
|
In an 8 CPU setup, changing workers from 2 to 16 improved concurrent registration time by 36%. In a 16 CPU setup, the same change improved it by 55%.
More workers also increase total registration concurrency. In our tests, 2 workers handled up to 480 concurrent registrations. More workers improved this.
3.5. Puma threads configuration
Thread count determines how many parallel operations each worker process can handle. The relationship between threads and workers impacts resource utilization and parallel request handling without increasing memory consumption.
For example, we have compared these two setups:
| Foreman VM with 8 CPUs, 40 GiB RAM | Foreman VM with 8 CPUs, 40 GiB RAM |
|---|---|
|
|
|
|
Using more workers with the same total threads gives about 11% speedup for highly concurrent registrations. More workers do not use more CPU or RAM but provide better performance.
4. Tuning Apache HTTPD performance
Tune Apache httpd to improve Foreman performance and increase concurrency for Foreman web UI and API requests.
Apache httpd is a core component of Foreman. It handles web requests from the Foreman web UI and APIs.
5. Tuning Dynflow services
When many hosts report to Foreman and run concurrent tasks, increase Dynflow workers to improve task execution performance.
Dynflow is the workflow management system and task orchestrator for Foreman. It executes tasks in an out-of-order manner. When many hosts report to Foreman and run tasks, tune Dynflow by setting how many executors it can launch.
For more information about the tunings involved related to Dynflow, see https://foreman.example.com/foreman_tasks/sidekiq.
Foreman contains a Dynflow service called dynflow-sidekiq that runs tasks scheduled by Dynflow.
Sidekiq workers can be grouped into queues so many tasks of one type do not block tasks of another type.
Foreman community recommends increasing sidekiq workers to scale the Foreman tasking system for bulk concurrent tasks. There are two options:
-
Increase the number of threads used by a worker (worker concurrency). This has limited impact for values larger than five due to how Ruby implements thread concurrency.
-
Increase the number of workers (recommended).
-
Increase the number of workers from one worker to three while remaining five threads/concurrency of each:
# foreman-installer --foreman-dynflow-worker-instances 3 # optionally, add --foreman-dynflow-worker-concurrency 5
-
Optional: Check if there are three worker services:
# systemctl -a | grep dynflow-sidekiq@worker-[0-9] dynflow-sidekiq@worker-1.service loaded active running Foreman jobs daemon - worker-1 on sidekiq dynflow-sidekiq@worker-2.service loaded active running Foreman jobs daemon - worker-2 on sidekiq dynflow-sidekiq@worker-3.service loaded active running Foreman jobs daemon - worker-3 on sidekiq
6. Tuning pull-based REX transport
You can adjust MQTT broker settings and client polling intervals to optimize pull-based remote execution performance for the scale and network conditions of your environment.
6.1. Increasing host limit for pull-based REX transport
Tune the mosquitto MQTT server on Smart Proxy to support more than 1024 connected hosts for larger deployments.
-
You have enabled pull-based remote execution on your Smart Proxy. For more information, see Configuring pull-based transport for remote execution in Installing a Smart Proxy Server 5.0 on Enterprise Linux.
-
On your Smart Proxy, set the upper limit of connected hosts for pull-based remote execution in
/etc/foreman-installer/custom-hiera.yaml:systemd::dropin_files: limits.conf: unit: mosquitto.service content: "[Service]\nLimitNOFILE=5000\n"This example configures the
mosquittoservice on your Smart Proxy to handle up to 5000 hosts. -
Re-run the installer for the changes to take effect:
# foreman-installer
6.2. Decreasing performance impact of the pull-based REX transport
Configure MQTT rate limiting and timing parameters to prevent connection exhaustion on Smart Proxy server when running high-concurrency remote execution jobs in pull-based mode.
When Foreman server is configured with the pull-based transport mode for remote execution jobs by using the Script provider, Smart Proxy server sends notifications about new jobs to hosts through MQTT. This notification does not include the actual workload that the host is supposed to execute. After a host receives a notification about a new remote execution job, it queries Smart Proxy server for its actual workload. During the job, the host periodically sends outputs of the job to Smart Proxy server, further increasing the number of requests to Smart Proxy server.
These requests to Smart Proxy server together with high concurrency allowed by the MQTT protocol can cause exhaustion of available connections on Smart Proxy server. Some requests might fail, making some child tasks of remote execution jobs unresponsive. This also depends on actual job workload, as some jobs are causing additional load to Foreman server, making it compete for resources if hosts are registered to Foreman server.
To avoid this, configure your Foreman server and Smart Proxy server with the following parameters:
-
MQTT Time To Live – Time interval in seconds given to the host to pick up the job before considering the job undelivered
-
MQTT Resend Interval – Time interval in seconds at which the notification should be re-sent to the host until the job is picked up or canceled
-
MQTT Rate Limit – Number of jobs that are allowed to run at the same time. You can limit the concurrency of remote execution by tuning the rate limit which means you are going to put more load on Foreman.
Smart Proxy server uses Webrick HTTP server (no httpd or Puma involved), so there is no simple way to increase its capacity.
-
Tune the MQTT parameters on your Foreman server:
# foreman-installer \ --foreman-proxy-plugin-remote-execution-script-mqtt-rate-limit My_MQTT_Rate_Limit \ --foreman-proxy-plugin-remote-execution-script-mqtt-resend-interval My_MQTT_Resend_Interval \ --foreman-proxy-plugin-remote-execution-script-mqtt-ttl My_MQTT_Time_To_Live
7. Tuning the PostgreSQL database
Configure PostgreSQL database settings to improve response times. Tuning PostgreSQL reduces query latency and improves system responsiveness, especially with high database activity.
The PostgreSQL authors recommend disabling Transparent Hugepage on servers running PostgreSQL.
7.1. Setting PostgreSQL tuning parameters
Tune PostgreSQL by modifying the postgresql.conf file.
This works regardless of the tuning profile.
-
Append
/etc/foreman-installer/custom-hiera.yamlto tune PostgreSQL:postgresql::server::config_entries: max_connections: 1000 shared_buffers: 2GB work_mem: 8MB autovacuum_vacuum_cost_limit: 2000This tuning configuration alters the following keys:
max_connections-
Maximum number of connections PostgreSQL can accept.
shared_buffers-
Memory used by all active connections to store data for database operations. Set this between 2 GiB to 25% of total system memory depending on database operation frequency.
work_mem-
Memory allocated per process to store temporary results of database operations. Set this to 8 MB for most heavy operations.
autovacuum_vacuum_cost_limit-
Cost limit for the vacuuming operation that cleans up dead tuples in database relations. This defines how many tuples can be processed in a single run. Foreman community recommends
2000for the medium, large, extra-large, and extra-extra-large profiles.
-
Reconfigure your Foreman:
# foreman-installer
7.2. Benchmarking raw database performance
Use pgbench to measure raw PostgreSQL performance and validate storage behavior on your Foreman environment.
Benchmarking helps you establish baseline performance metrics and verify that tuning changes improve database responsiveness.
-
A non-production system with a valid backup.
|
Warning
|
|
-
To identify the largest tables by disk space for Foreman, use the postgres-size-report script in the satellite-support Git repository. For more information, see github.com/RedHatSatellite/satellite-support.
-
Ensure the PostgreSQL data directory
/var/lib/pgsqlhas enough space for the benchmark workload. You might need to resize the volume to 100 GiB or more, depending on how long the benchmark runs and how much data it generates.NoteChoice of filesystem for PostgreSQL data directory might matter as well.
-
Install the PGBench utility:
# dnf install postgresql-contrib
8. Tuning the Redis cache
Understanding Redis tuning helps you improve caching and task tracking performance. Proper Redis setup keeps memory use stable and improves Foreman responsiveness.
Redis is an in-memory data store. Foreman uses Redis for caching and Dynflow uses Redis to track its tasks. Given the way Foreman uses Redis, its memory consumption should be stable.
The Redis authors recommend disabling Transparent Hugepage on servers running Redis.