cgroups

Create a new control group

Flag	Usage
-g controller:/path	defines the control group(s) to be added : controller is a list of controllers /path is the relative path to control groups in the given controllers list

See complete example

Delete a control group

See complete example

Run a task in the given control group

See complete example

Print parameter(s) of given control group(s)

Set parameter(s) of given control group(s)

See complete example

List all control groups

Create some groups so that we have something to list :

cgcreate -g cpu:/cpuGroup_Parent/cpuGroup_childA
cgcreate -g cpu:/cpuGroup_Parent/cpuGroup_childB

List groups :

lscgroup

list everything (pretty verbose)

lscgroup | grep cpu

cpu,cpuacct:/
	cpu,cpuacct:/cpuGroup_Parent
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childB
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childA
	cpuset:/

Because of grep, this also lists groups using the cpuset controller.

lscgroup cpu:/

List only the control groups using the cpu controller :

cpu,cpuacct:/
	cpu,cpuacct:/cpuGroup_Parent
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childB
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childA

lscgroup cpu:groupName

lscgroup cpu:/cpuGroup_Parent

cpu,cpuacct:/cpuGroup_Parent/
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childB
	cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childA

lscgroup cpu:/cpuGroup_Parent/cpuGroup_childA

cpu,cpuacct:/cpuGroup_Parent/cpuGroup_childA/

The / leading the path looks optional

Delete groups made just for the example :

cgdelete cpu:/cpuGroup_Parent/cpuGroup_childA
cgdelete cpu:/cpuGroup_Parent/cpuGroup_childB
cgdelete cpu:/cpuGroup_Parent
lscgroup cpu:/

cpu,cpuacct:/			i.e. "empty resultset"

Description, definitions and the likes

cgroups (abbreviated from control groups) is a Linux kernel feature that limits, accounts for, and isolates the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes.
There are two versions of cgroups :
- v1 appeared in 2007.
- v2 is a redesign / rewrite, and appeared in Linux kernel 4.5 (2016).
Unlike v1, cgroups v2 has only a single process hierarchy and discriminates between processes, not threads. (source)
Despite cgroups v2 being "not-that-new", most articles / demos / ... (and examples below) are about cgroups v1. To know which version is currently implemented :
mount | grep 'type cgroup'
- v1 has cgroup type
- v2 has cgroup2 type
cgroups features
a control group (aka cgroup) is a collection of processes that are bound by the same criteria and associated with a set of parameters or limits. These groups can be hierarchical, meaning that each group inherits limits from its parent group. The kernel provides access to multiple controllers (aka subsystems) through the cgroup interface :
- the memory controller limits memory use
- cpuacct accounts CPU usage
- ...
- Cgroups version 1 controllers

/proc files :

/proc/cgroups: This file contains information about the controllers that are compiled into the kernel.
cat /proc/cgroups | tr '\t' '|' | column -s '|' -t
/proc/pid/cgroup: describes control groups to which the process with the corresponding PID belongs.

Setup :

apt install libcgroup1 cgroup-bin

This will also install cgroup-tools as a dependency.
... and that's it !

Usage

cgroups can be configured 3 main ways (source) :

Manually via the /sys/fs/cgroup pseudo filesystem :
- requires lots of mount, mkdir and echo something >> some/file
- complex and pretty error-prone
- not reboot-proof
- doesn't seem worth the headache considering other solutions below
Via commands of the libcg library (see examples) :
- easier than the manual approach
- allows to manage things on-the-fly with its wrapper utilities such as cgcreate, cgclassify, cgexec, ...
- not reboot-proof
Via systemd commands and config files (see examples)

Remember you don't generally mix systemd and libcg implementations.

With the libcg commands :

As a first basic test, we'll try to limit CPU usage. To do so, we need :

CPU-hungry commands. Those will do the job :
- yes > /dev/null
- cat /dev/urandom > /dev/null
They can use up to 100% of a CPU, but it's still safe to run them on your workstation as you can stop them with a simple Ctrl-c.
a CPU monitoring utility. top or htop can do the job.

There is a "detail" that is not (well|at all)? explained in articles introducing to cgroups when it comes to limit the CPU usage. Articles show examples, but nothing works as described.
Most of these examples exhibit the cpu.shares setting without referring to its definition, then forget a VERY important information : the "CPU share" amount given to any "controlled group" is relative to the total shares given to all groups. In other words : if you use cpu.shares once, it can't work. Let me explain :

Scenario	CPU share given to the cgroup of the process we're trying to limit	CPU share given to any other cgroup	Total shares defined	Relative weight of shares given to the cgroup limiting the CPU usage
1	x (x >0)	0 (i.e. cpu.shares used only once)	x	100%
2	x	x	2x	50%
3	x	9x	10x	10%

Once you get this, it is pretty straightforward .

With cpu.shares :

create 2 groups :
cgcreate -g cpu:/cpuLimited
cgcreate -g cpu:/cpuNotLimited
define limits :
cgset -r cpu.shares=10 cpuLimited
cgset -r cpu.shares=10 cpuNotLimited
execute commands within each group :
cgexec -g cpu:/cpuLimited yes >/dev/null &
cgexec -g cpu:/cpuNotLimited cat /dev/urandom > /dev/null &
observe the result with htop : you'll see both processes using ~50% of the CPU each (you can kill them now)
The only way the chosen numbers (here 10 / 10) matter is how they relate to each other. To check this, try with :
- 100 / 900
- 1000 / 9000
In both cases, you should see the yes process limited to 10% of the CPU. Setting lower values like 1 / 9 should lead to the same behavior. However, these low values interfere with existing values, which ends behaving slightly differently (but still limited anyway).

Most articles repeat things like By default, all groups inherit 1,024 shares or 100% of CPU time., then show examples where 1024 stands for 100% CPU, 512 for 50% of CPU, 338 for 33% and so on... Either I've not understood this correctly, or this explanation is pretty misleading as well as partially wrong.
clean before leaving :
cgdelete cpu:/cpuLimited
cgdelete cpu:/cpuNotLimited

With cpu.cfs_period_us and cpu.cfs_quota_us (see also) :

These directives can be used to set a hard limit on CPU usage, even when no other process is running.

us is for "µs".

cpu.cfs_period_us: this defines a period of time, in microseconds. Let's choose 1s = 1000000µs
cpu.cfs_quota_us: this defines the amount of time (still in microseconds) during each period (aka cpu.cfs_period_us), when the corresponding group can use the CPU. So if you target 10% of CPU usage, this value has to be 10% of cpu.cfs_period_us = 0.1s = 100000µs.

create a group :
cgcreate -g cpu:/myGroup
set values :
cgset -r cpu.cfs_period_us=1000000 myGroup
cgset -r cpu.cfs_quota_us=100000 myGroup
run a CPU-hungry command :
cgexec -g cpu:/myGroup cat /dev/urandom > /dev/null &
and see what happens with htop
clean before leaving :
cgdelete cpu:/myGroup

Do the same with already running processes :

So far, we defined rules and ran processes by explicitly launching them within a control group limiting their CPU usage. Now, we're going to do it the opposite way, i.e. we'll apply rules to already running processes.

For both tests, you should run htop in a dedicated terminal.

With cpu.shares :
1. simulate some already running processes :
  yes >/dev/null &
  
  cat /dev/urandom > /dev/null &
```
[1] 1030
[2] 1031
```
  You should already observe both processes eating 100% of the CPU (50% each)
2. create 2 groups :
  cgcreate -g cpu:/cpuLimited
  cgcreate -g cpu:/cpuNotLimited
3. attach processes to groups :
  cgclassify -g cpu:/cpuLimited $(pidof yes)
  cgclassify -g cpu:/cpuNotLimited $(pidof cat)
4. observe /sys/fs/cgroup files :
  cat /sys/fs/cgroup/cpu/cpu{,Not}Limited/tasks
```
1030
1031
```
  - Each control group has a tasks file containing the PIDs of the processes it controls. cgclassify saves us from manually having to : echo $(pidof command) >> /sys/fs/cgroup/cpu/groupName/tasks
  - Since /sys/fs/cgroup is a pseudo filesystem, the tasks file will always appear 0-sized despite having data.
5. Now processes are under control, let's set some limits :
  cgset -r cpu.shares=100 cpuLimited
  cgset -r cpu.shares=900 cpuNotLimited
  observe the result in htop. Try with different values :
  cgset -r cpu.shares=600 cpuLimited
  cgset -r cpu.shares=300 cpuNotLimited
6. cgdelete -g cpu:/cpuLimited
  cgdelete -g cpu:/cpuNotLimited
  When deleting the control groups, we "release" the limiting conditions, and both processes now each take again 50% of the CPU.
With cpu.cfs_period_us and cpu.cfs_quota_us :
1. Let's again simulate 1 already-running process (while still checking CPU usage in a dedicated terminal with htop) :
  cat /dev/urandom > /dev/null &
2. create a control group :
  cgcreate -g cpu:/myGroup
3. attach process to group :
  cgclassify -g cpu:/myGroup $(pidof cat)
4. set limits :
  cgset -r cpu.cfs_period_us=1000000 myGroup
  cgset -r cpu.cfs_quota_us=100000 myGroup
  
  The CPU usage should be down to 10%.
5. cgdelete -g cpu:/myGroup

Deleting a control group releases the constraints this group enforces.
Looks like it doesn't matter whether you cgclassify then cgset (i.e. "invite", then "restrict") or the other way around.

Persistent Groups :

So far, everything we did was configured on-the-fly and is not reboot-proof. We'll see now how to make persistent groups.

It starts by editing /etc/cgconfig.conf
Debian comes with no default /etc/cgconfig.conf, but a template is available as /usr/share/doc/cgroup-tools/examples/cgconfig.conf (source) :
cp /usr/share/doc/cgroup-tools/examples/cgconfig.conf /etc/cgconfig.conf

Add :

group cpuLimited {
	cpu {
		cpu.shares = 100;
		}
	}

group cpuNotLimited {
	cpu {
		cpu.shares = 900;
		}
	}

Enable + start the service :
systemctl enable cgconfig && systemctl start cgconfig
If this fails saying :
```
Failed to enable unit: File cgconfig.service: No such file or directory
```
Use this command instead to load the configuration (source) :
cgconfigparser -l /etc/cgconfig.conf
start htop in a dedicated terminal
then start processes :
cgexec -g cpu:cpuLimited yes >/dev/null &
cgexec -g cpu:cpuNotLimited cat /dev/urandom > /dev/null &
You should observe the yes command limited to 10% in CPU usage, and cat using 90%.

You can do the same with the cpu.cfs_period_us and cpu.cfs_quota_us directives :

group myGroup {
	cpu {
		cpu.cfs_period_us = 1000000;
		cpu.cfs_quota_us  =  100000;
		}
	}

load the configuration (this actually creates the /sys/fs/cgroup/cpu/myGroup directory and files)
cgexec -g cpu:/myGroup cat /dev/urandom > /dev/null &

Clean before leaving :

cgdelete cpu:/{cpuNotLimited,cpuLimited,myGroup}

This {,,} construct is a shell trick that has nothing to do with cgroups .



http://www.linuxembedded.fr/2011/03/bien-utiliser-les-cgroups/

https://www.cloudsigma.com/howto-cgroups/

https://www.linuxjournal.com/content/everything-you-need-know-about-linux-containers-part-i-linux-control-groups-and-process

https://oakbytes.wordpress.com/2012/09/02/cgroup-cpu-allocation-cpu-shares-examples/



cgclear cpu (looks like this unmounts A LOT (?))
systemctl restart cgroupfs-mount.service


grep cgroup /proc/filesystems
	==> lists whether cgroup is supported

cgroups - Limit, account for, and isolate resources usage

cgcreate

Usage

Flags

Example

cgdelete

Usage

Example

cgexec

Usage

Example

cgget

Usage

cgset

Usage

Example

lscgroup

Usage

Example

Create some groups so that we have something to list :

List groups :

Delete groups made just for the example :

Description, definitions and the likes

/proc files :

Setup :

Usage

Example

With the libcg commands :

With cpu.shares :

With cpu.cfs_period_us and cpu.cfs_quota_us (see also) :

Do the same with already running processes :

Persistent Groups :