The timing report explains where fapolicyd spent time during a bounded window. That is useful only if the workload is repeatable. This article covers the new stress harness and how to use it with status, metrics, and timing.
The stress helper is not installed by make install. It is a development, QE, sizing, and regression tool. It generates high-rate fanotify decision traffic against a running fapolicyd daemon by creating process trees and running specific workloads.
It can exercise process startup tracking, subject cache collisions, object cache churn, interpreter handling, no-shebang script handling, file opens, execs, and large file reads.
 |
| The stress harness creates controlled pressure so fapolicyd's behavior can be measured. |
Building the Harness
Build it explicitly:
./configure --enable-stressThe binary is:
make -j32
src/tests/stress/fapolicyd-stress--enable-stress defaults to off. A normal build does not enter this directory and does not build the helper.
If the daemon is enforcing policy, remember that this locally built helper is not automatically trusted. The interpreter workloads also use scripts from the source tree. Add the helper and scripts to the file trust database before running tests against an enforcing daemon:
stress_dir="$PWD/src/tests/stress"If you rebuild the helper, update the trust entry because size or hash may change.
sudo fapolicyd-cli --file add "$stress_dir/fapolicyd-stress"
sudo fapolicyd-cli --file add "$stress_dir/scripts"
sudo fapolicyd-cli --update
Workloads
The harness can run several workloads:
fork-exec tight fork/exec loopsfork-exec is the default and the best starting point for subject-cache and startup-state pressure. It repeatedly forks a child that execs a configured command. This creates a lot of short-lived process activity, which is exactly the kind of workload that can expose startup tracking and subject cache collision problems.
exec-open opens configured command paths and executes them
interpreter runs a script directly and through a shell
noshebang exercises programmatic content without a #! line
hash creates and reads a large file
churn creates many small files to churn the object cache
all runs every workload
exec-open adds file open traffic around the exec stream. This is useful when you want both execution and object-open activity.
interpreter and noshebang are for programmatic content paths. The first runs a script directly and through a shell. The second attempts a direct exec of a file without a `#!` line and then runs it through the selected shell.
hash creates a large generated file and reads it. This is useful when integrity mode or policy makes hashing visible in the timing report.
churn creates many distinct small files and opens them in rotation. This is for object cache churn.
all is broad coverage. It is not where I would start if I were trying to understand one bottleneck.
Process Shape
The basic process controls are roots, fanout, and depth. The estimated leaf process count is:
roots * fanout ^ depthMore leaves means more concurrent process pressure. Wide process trees are the main way to create subject-cache collisions. That is what you need if you are testing startup-state behavior and subject deferral.
A representative command is:
src/tests/stress/fapolicyd-stress --workload fork-exec --roots 32 \That creates 256 leaf processes. Each leaf runs the selected workload until the time limit expires. Note, the --timing parameter tells the stress harness to arm and stop the timer to automatically generate a timing report so that you do not need to mess with fapolicyd-cli.
--fanout 8 --depth 1 --iterations 0 --seconds 60 --timing
Smoke Test
A short local smoke test without daemon reports looks like this:
./fapolicyd-stress --no-status --workload fork-exec --roots 2 --iterations 10
fapolicyd stress harness
workload: fork-exec
roots: 2
fanout: 1
depth: 0
estimated leaf processes: 2
iterations per leaf: 10
seconds: 0
workdir: /tmp/fapolicyd-stress.ShyQtW
Workload summary:
wall_seconds: 0.040
operations: 20
errors: 0
throughput_ops_per_sec: 495.1
Timed Pressure Test
For a timed fork/exec pressure test against a configured daemon, run:
./fapolicyd-stress --workload fork-exec --roots 32 --seconds 30 --timing
fapolicyd stress harness
workload: fork-exec
roots: 32
fanout: 1
depth: 0
estimated leaf processes: 32
iterations per leaf: 100
seconds: 30
workdir: /tmp/fapolicyd-stress.uxEEQg
Workload summary:
wall_seconds: 1.801
operations: 3200
errors: 0
throughput_ops_per_sec: 1776.3
Daemon status deltas:
Inter-thread max queue depth: before=4 after=95
Subject deferred events: before=0 after=0
Subject defer max depth: before=0 after=0
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=0 after=0 delta=0
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=0 after=0 delta=0
Subject collisions: before=0 after=16 delta=16
Subject evictions: before=11 after=30 delta=19
Object collisions: before=70 after=74 delta=4
Object evictions: before=70 after=74 delta=4
Allowed accesses: before=3447 after=60541 delta=57094
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 56953
Max queue depth during timing: 95
Timed throughput: 31537.4 decisions/sec
Active decision rate: 32206.7 decisions/sec
Decision latency: avg=31.0 us max=3.31 ms p95_bucket=<=100us
With --timing, the harness verifies that the daemon has timing_collection=manual, starts timing, runs the workload, stops timing, and parses a short summary from the timing report. It also captures status and metrics before and after the workload when it can.
This gives three views:
harness output what workload was generatedThe harness output has two different throughput ideas. throughput_ops_per_sec is the harness's local operation rate. It is not the same as daemon decision throughput. One local operation can generate zero, one, or multiple fanotify permission events. The timing report's decision count and throughput are the daemon-side numbers.
metrics deltas what counters moved
timing report where decision time went
Subject Deferral Testing
For subject deferral testing, use the early eviction preset:
[run the following command: sudo src/tests/stress/fapolicyd-stress --preset early-evict --timing]
./fapolicyd-stress --preset early-evict --timing
fapolicyd stress harness
workload: fork-exec
roots: 32
fanout: 8
depth: 1
estimated leaf processes: 256
iterations per leaf: 0
seconds: 60
workdir: /tmp/fapolicyd-stress.jID2g5
Workload summary:
wall_seconds: 60.150
operations: 36666
errors: 0
throughput_ops_per_sec: 609.6
Daemon status deltas:
Inter-thread max queue depth: before=95 after=435
Subject deferred events: before=0 after=1
Subject defer max depth: before=0 after=1
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=0 after=3 delta=3
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=0 after=3 delta=3
Subject collisions: before=89 after=36881 delta=36792
Subject evictions: before=113 after=36905 delta=36792
Object collisions: before=207 after=216 delta=9
Object evictions: before=207 after=216 delta=9
Allowed accesses: before=63638 after=672670 delta=609032
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 608940
Max queue depth during timing: 435
Timed throughput: 10122.9 decisions/sec
Active decision rate: 10159.5 decisions/sec
Decision latency: avg=98.4 us max=19.9 ms p95_bucket=<=500us
There is also an ld-so-regression preset intended for comparing builds with and without subject deferral:
[run the following command: sudo src/tests/stress/fapolicyd-stress --preset ld-so-regression --timing]
./fapolicyd-stress --preset ld-so-regression --timing
fapolicyd stress harness
workload: fork-exec
roots: 32
fanout: 8
depth: 1
estimated leaf processes: 256
iterations per leaf: 0
seconds: 60
workdir: /tmp/fapolicyd-stress.On1G0Q
Workload summary:
wall_seconds: 60.145
operations: 32529
errors: 0
throughput_ops_per_sec: 540.8
Daemon status deltas:
Inter-thread max queue depth: before=435 after=453
Subject deferred events: before=1 after=2
Subject defer max depth: before=1 after=1
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=3 after=3 delta=0
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=3 after=3 delta=0
Subject collisions: before=36901 after=70158 delta=33257
Subject evictions: before=36925 after=70182 delta=33257
Object collisions: before=295 after=298 delta=3
Object evictions: before=295 after=298 delta=3
Allowed accesses: before=673924 after=1215792 delta=541868
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 541778
Max queue depth during timing: 453
Timed throughput: 9007.2 decisions/sec
Active decision rate: 9041.6 decisions/sec
Decision latency: avg=111 us max=1.63 ms p95_bucket=<=500us
The strongest evidence for the early-eviction problem is:
- Subject collisions increased
- Early subject cache evictions increased
- The workload was wide enough to create many concurrent process startups
After deferral is working, a healthy run should show fewer early subject cache evictions and fewer unexpected denials under the same daemon configuration. Subject deferred events and Subject defer max depth may increase. That is fine. That means events were parked instead of evicting a BUILDING subject. But if Subject defer fallbacks keeps rising, the fixed defer array filled and the daemon fell back to the older eviction path.
 |
| The early-evict preset is meant to prove whether subject deferral reduces premature BUILDING evictions. |
Cache and Hash Workloads
For object cache pressure, run the churn workload and watch object misses, collisions, and evictions:
./fapolicyd-stress --workload churn --roots 8 --fanout 4 --depth 1 --seconds 60 --timing
fapolicyd stress harness
workload: churn
roots: 8
fanout: 4
depth: 1
estimated leaf processes: 32
iterations per leaf: 100
seconds: 60
workdir: /tmp/fapolicyd-stress.SqRIGt
Workload summary:
wall_seconds: 0.101
operations: 3200
errors: 0
throughput_ops_per_sec: 31531.9
Daemon status deltas:
Inter-thread max queue depth: before=453 after=453
Subject deferred events: before=2 after=2
Subject defer max depth: before=1 after=1
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=3 after=3 delta=0
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=3 after=3 delta=0
Subject collisions: before=70180 after=70217 delta=37
Subject evictions: before=70207 after=70244 delta=37
Object collisions: before=724 after=743 delta=19
Object evictions: before=724 after=743 delta=19
Allowed accesses: before=1218087 after=1221409 delta=3322
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 3228
Max queue depth during timing: 32
Timed throughput: 30256.9 decisions/sec
Active decision rate: 34689.6 decisions/sec
Decision latency: avg=28.8 us max=4.32 ms p95_bucket=<=50us
For integrity cost, use the hash workload and look for hash_sha or hash_ima timing:
./fapolicyd-stress --workload hash --roots 8 --seconds 60 --timing
fapolicyd stress harness
workload: hash
roots: 8
fanout: 1
depth: 0
estimated leaf processes: 8
iterations per leaf: 100
seconds: 60
workdir: /tmp/fapolicyd-stress.1NYq01
Workload summary:
wall_seconds: 1.603
operations: 800
errors: 0
throughput_ops_per_sec: 498.9
Daemon status deltas:
Inter-thread max queue depth: before=453 after=453
Subject deferred events: before=2 after=2
Subject defer max depth: before=1 after=1
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=3 after=3 delta=0
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=3 after=3 delta=0
Subject collisions: before=70230 after=70242 delta=12
Subject evictions: before=70257 after=70269 delta=12
Object collisions: before=755 after=757 delta=2
Object evictions: before=755 after=757 delta=2
Allowed accesses: before=1221624 after=1222544 delta=920
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 830
Max queue depth during timing: 8
Timed throughput: 515.9 decisions/sec
Active decision rate: 100841.2 decisions/sec
Decision latency: avg=9.92 us max=138 us p95_bucket=<=50us
For broad coverage, run all workloads:
./fapolicyd-stress --workload all --roots 8 --fanout 4 --depth 1 --seconds 60 --timing
fapolicyd stress harness
workload: all
roots: 8
fanout: 4
depth: 1
estimated leaf processes: 32
iterations per leaf: 100
seconds: 60
workdir: /tmp/fapolicyd-stress.2rexsD
Workload summary:
wall_seconds: 16.156
operations: 76800
errors: 0
throughput_ops_per_sec: 4753.7
Daemon status deltas:
Inter-thread max queue depth: before=453 after=453
Subject deferred events: before=2 after=35
Subject defer max depth: before=1 after=3
Subject defer fallbacks: before=0 after=0 delta=0
Subject defer oldest age: before=0 ns after=0 ns
Early subject cache evictions: before=3 after=3 delta=0
Subject BUILDING tracer evictions: before=0 after=0 delta=0
Subject BUILDING stale evictions: before=3 after=3 delta=0
Subject collisions: before=70253 after=115846 delta=45593
Subject evictions: before=70280 after=122273 delta=51993
Object collisions: before=1416 after=1471 delta=55
Object evictions: before=1416 after=1471 delta=55
Allowed accesses: before=1224770 after=1751558 delta=526788
Denied accesses: before=3 after=3 delta=0
Kernel queue overflows: before=0 after=0 delta=0
Reply errors: before=0 after=0 delta=0
Decision timing:
Full report: /run/fapolicyd/fapolicyd.timing
Decisions: 526698
Max queue depth during timing: 129
Timed throughput: 32591.0 decisions/sec
Active decision rate: 33450.5 decisions/sec
Decision latency: avg=29.9 us max=12.5 ms p95_bucket=<=100us
Use all after you understand the individual workload costs. If you start with everything, the output may show several things moving at once and you will not know which workload caused which effect.
Reading the Results
After a stress run, collect both reports:
fapolicyd-cli --check-statusFor queue pressure, compare Inter-thread max queue depth with configured q_size, and then look at the timing report's Queueing section. If both queue depth and queue wait are high, requests are backing up before evaluation.
fapolicyd-cli --check-metrics
For subject-cache pressure, look at:
- Subject collisions
- Subject evictions
- Early subject cache evictions
- Subject deferred events
- Subject defer max depth
- Subject defer fallbacks
- Subject BUILDING tracer evictions
- Subject BUILDING stale evictions
For object-cache pressure, look at:
- Object misses
- Object collisions
- Object evictions
For policy decisions, look at:
- Allowed accesses
- Denied accesses
- Allowed by rule
- Allowed by fallthrough
For daemon health, any non-zero value in these fields deserves attention:
- Kernel queue overflow
- Reply errors
- Subject defer fallbacks
The stress harness does not replace real production observation. It gives you a controlled way to move specific parts of the daemon. That is useful for regression testing, sizing experiments, and proving whether a change affected the counter or timing gate you intended to move.
The pattern is:
- Pick one workload.
- Run a bounded test.
- Read status for health.
- Read metrics for counters.
- Read timing for latency.
- Change one variable.
- Run it again.
That is how the stress harness fits with the rest of the reporting work. Status tells you whether the daemon stayed healthy. Metrics tell you what moved. Timing tells you where time went. The stress tool gives you a repeatable way to make those questions concrete.
