Configuring Linux for High Performance

Modified on Thu, 23 Mar 2023 at 12:53 PM

Introduction

To ensure optimal and expected performance from a Blackcore server there are a few essential configuration changes that must be made to the Linux environment. This guide will cover the steps needed to make the CPU run at the maximum frequency at all times, as well as reduce latency due to C-state transitions. This the minimum config required.

Linux Environment

For this guide we will be using RHEL 8.4 with kernel version 4.18

You must have the intel_pstate driver installed and enabled. This should already be enabled by default, however if you compile your own kernel you might need to add the driver manually.

The steps may differ in other Linux distributions and/or kernels.

Controlling C-States

Lastly we need to edit the grub configuration to boot with some specific kernel parameters. It is worth noting that Linux has complete control over the CPU and can override some BIOS settings. As such we need to set the following parameters for best performance.

Intel_idle.max_cstate=0

This tells the intel_pstate driver to stay in the lowest C-state possible, which for the intel_pstate driver is C1.

There is some latency that occurs when switching from C1 (idle) to C0 (load), as such we want to keep the CPU in C0 state at all times to remove this latency.

Idle=poll

This will ‘spin’ a thread on all cores waiting for an instruction, keeping the CPU in the C0 state, thus removing any latency changing from C1 -> C0 state.

Add the kernel parameters to GRUB "GRUB_CMDLINE_LINUX" option:

# nano /etc/default/grub

Regenerate the grub files for both BIOS and UEFI bootloaders

# grub2-mkconfig -o /boot/grub2/grub.cfg

# grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

Reboot the unit.

Verify the kernel was booted with the correct parameters:

# cat /proc/cmdline

Finally, we can use turbostat to check that all cores are running at max frequency, and are in the C0 state (“100% BUSY”)

# turbostat

‘cpupower’ utilities

On some distributions, it may be necesary to configure the CPU scaling governor. We will be using the cpupower set of utilities to manage CPU scaling and governors. It automatically selects the correct modules to load as well as providing a service which we will use to enable our settings at boot.

Install cpupower via:

# yum install cpupowerutils

This will install the Kernel-tools package if not already installed.

Check the currently loaded module and governor:

# cpupower frequency-info

analyzing CPU 0:

driver: intel_pstate

CPUs which run at the same hardware frequency: 0

CPUs which need to have their frequency coordinated by software: 0

maximum transition latency: 0.97 ms.

hardware limits: 1.20 GHz - 3.80 GHz

available cpufreq governors: performance, powersave

current policy: frequency should be within 1.20 GHz and 3.80 GHz.

The governor "powersave" may decide which speed to use

within this range.

current CPU frequency is 1.20 GHz (asserted by call to hardware).

boost state support:

The correct driver should be intel_pstate, highlighted in green above. The current governor (powersave) is highlighted in blue.

We need to change the CPU governor from powersave to performance:

# cpupower frequency-set -g performance

Setting cpu: 0

Setting cpu: 1

Setting cpu: 2

[...]

On some systems you may need to configure a service to enable the governor on startup.

Create the service:

# systemctl enable cpupower

Created symlink from /etc/systemd/system/multi-user.target.wants/cpupower.service to /usr/lib/systemd/system/cpupower.service.

Disable the tuned service:

# systemctl disable tuned

Save the changes and reboot the machine. Check the service has started correctly:

# systemctl status cpupower

cpupower.service - Configure CPU power related settings

Loaded: loaded (/usr/lib/systemd/system/cpupower.service; enabled; vendor preset: disabled)

Active: active (exited) since Wed 2016-09-14 16:12:18 BST; 4min 46s ago

Process: 3118 ExecStop=/usr/bin/cpupower $CPUPOWER_STOP_OPTS (code=exited, status=0/SUCCESS)

Process: 3130 ExecStart=/usr/bin/cpupower $CPUPOWER_START_OPTS (code=exited, status=0/SUCCESS)

Main PID: 3130 (code=exited, status=0/SUCCESS)

Sep 14 16:12:18 localhost.localdomain cpupower[3130]: Setting cpu: 15

Sep 14 16:12:18 localhost.localdomain cpupower[3130]: Setting cpu: 16

Sep 14 16:12:18 localhost.localdomain cpupower[3130]: Setting cpu: 17

Sep 14 16:12:18 localhost.localdomain cpupower[3130]: Setting cpu: 18

Sep 14 16:12:18 localhost.localdomain cpupower[3130]: Setting cpu: 19

Sep 14 16:12:18 localhost.localdomain systemd[1]: Started Configure CPU power related settings.

Relevant parts highlighted in green.

Finally, check the current governor is correctly set to performance:

# cpupower frequency-info | grep --color '”performance”'

The governor "performance" may decide which speed to use

This guide only serves as a starting point for optimising Linux for high performance. Advanced configuration such as nohz and isolcpus are beyond the scope of this document. Also note that configuration may need to be tailored to specific software and user requirements.