Performance tuning

r7's defaults aim for high throughput, a short tail and a small footprint at the same time. Most deployments never need to change them. When you do, each setting below trades one resource for another, and the right side of that trade depends on where r7 runs. Each entry says what to tune, when it helps, what it costs, and how to check the result on your own hardware.

Numbers on this page were measured on one machine (20 cores, JDK 25, ZGC, an nginx backend on the same host) and are there to show direction and size, not to promise throughput. Measure before and after with benchmark/run.sh (see Benchmarks); only the passthrough-relative deltas carry over to other hardware.

Journal storage: shard_count, pre_fault and where segments live

Symptom

With journaling on at high request rates, throughput falls well below what the same gateway does with journaling off. p99 latency climbs into tens or hundreds of milliseconds, and it often gets worse the longer the load lasts. Header-heavy traffic at HEADERS or FULL shows it first. It grows with the number of open connections, since every connection has a thread of its own that may be writing.

Why it happens

The journal is split into shard_count shards (by default one per CPU the JVM sees, so a container's CPU limit counts, rounded down to a power of two, from 2 to 16). Each request goes to one shard, chosen by its request id. Each shard writes into one memory-mapped segment at a time, and writers take turns: encoding happens in parallel, but placing each entry in the segment is done one writer at a time per shard. That step is a short copy, cheap unless something stalls it. Two things do:

• Page faults. Segments are mapped lazily: the first write to each 4 KB page asks the kernel for the page, and every other writer to that shard waits meanwhile. At a few hundred MB/s of journal that is tens of thousands of stalls a second.
• Writeback throttling. On a disk, once enough journal pages are dirty, the kernel makes the writing thread wait for the disk to catch up. That happens at the same moment, and it builds up as dirty pages accumulate, which is why the slowdown grows over a run.

How much a stall costs depends on how many writers are waiting. The gateway runs each connection on a virtual thread of its own, so there can be a writer for every open connection, all queued on the same shard.

What r7 does about it: fault-ahead

On Linux 5.14 and later, r7 handles page faults without any configuration. A background thread per shard keeps the next 8 MB past the write position faulted in, using madvise(MADV_POPULATE_WRITE), so writers almost never fault. That costs about 8 MB of memory per shard beyond what has been written, and nothing up front. The call prepares pages without writing to them, so it never touches journal data.

Fault-ahead switches itself off where it can't work: on other operating systems, on older kernels, or when pre_fault is on. Writers then fault as before. It uses the JDK's native interface, which the gateway jars allow in their manifest (Enable-Native-Access). If you start r7 some other way than java -jar, add --enable-native-access=ALL-UNNAMED (already part of the recommended JVM flags), or the JVM prints a warning.

Fault-ahead does nothing about writeback throttling, which depends on the disk and on how quickly a tailer consumes sealed segments.

The settings

What they did, measured

HEADERS journaling, header-heavy workload, 200 connections at saturation. On tmpfs, which removes disk writeback and isolates the writer lock:

On a local disk, one shard: 63k req/s without fault-ahead, 77k with pre_fault, 84k with fault-ahead.

Under a container memory limit (-m 8g), journals on a disk volume, with cgroup v2 memory.stat:

In the container, the journal's page cache fills the memory limit within seconds at full load, and reclaiming it, not the writers' lock, sets the pace. That is why every setting is slower there than on the host. pre_fault makes it worse, because the pages it maps up front are the ones the kernel can least easily reclaim.

Recommendations

• Leave pre_fault off. Fault-ahead does its job at a fraction of the memory. Turn it on only on a platform without fault-ahead (not Linux, or a kernel older than 5.14), with memory to spare and outside a memory-limited container.
• Keep the default of one shard per CPU. Every connection's writer queues on its shard, for the placement with compression: none and for a copy into the shard's stage with zstd, and each shard's batches are compressed by one writer thread, so the count has to grow with the cores doing the writing: 4 shards halved p99 compared with one. Each shard is one more open segment, a fault-ahead thread and 8 MB kept faulted in, and without pre_fault costs no other memory of note. Past 16 CPUs the default stops at 16; set shard_count explicitly to go higher.
• Give a container headroom for page cache, not just heap. The journal is written through the page cache, which counts toward the container's memory limit. Up to the limit, that cache is reclaimable. How much stays dirty depends on how fast the disk and your tailer keep up.

Never put journals on tmpfs (emptyDir: {medium: Memory}, --tmpfs, /dev/shm) unless something deletes sealed segments as fast as they are written. tmpfs pages cannot be reclaimed and count fully against the container's limit, and r7 does not delete its own journals. A container with journals on tmpfs was OOM-killed within 30 seconds of full load in these tests.

Reading the numbers in a container

The kernel's per-container memory counters (cgroup v2 memory.stat) break the usage down:

• file_mapped and file: journal segments. Mapped segments count as active, so they show up in the working set Kubernetes and docker stats report, even though most of it can be reclaimed.
• shmem: with ZGC, this is mostly the Java heap (ZGC backs it with shared memory), not journals, unless the journals are on tmpfs.
• file_dirty: journal data not yet written to disk. If it stays high, the disk is the bottleneck, not r7.

A working set near the limit is normal under journaling load and is mostly page cache. An OOM kill is not: check shmem for journals on tmpfs, and the heap settings.

How to check yours

benchmark/run.sh --scenario journal --journal-levels HEADERS,FULL --workload headers --repeat 3

Run it once with your current storage settings and once with the change, against the same backend. Compare the vs r7 column, and the p99 at the rate you actually serve, not only saturation throughput. In a container, also watch memory.stat while it runs.

Journal compression: compression and compression_level

What it does

With compression: zstd, the default, the request thread does not compress. It copies its entry into its shard's stage, a 32 KB buffer, and a writer thread per shard compresses the stage as one zstd frame and places it in the segment as one batch: when the stage is full, or 10 ms after its first entry. A whole batch compresses far better than any entry would on its own, because the entries in it repeat each other's headers and bodies, and it costs a fraction of the CPU of compressing them one at a time.

What it costs

• A tailer sees an entry when its batch is placed: at low traffic, 10 ms after the write plus the time to compress and place the batch; sooner as traffic fills the stage.
• A crash loses what is staged, up to 64 KB of entries per shard, the last 10 ms of them at low traffic. A graceful shutdown places them first.
• Damage costs a batch, about 32 KB of entries before compression, where uncompressed it costs the rest of a 32 KB block.
• One writer thread per shard, busy in proportion to the bytes journaled.
• An entry larger than the stage is compressed by its request thread, as a batch of its own; in practice a body chunk over 32 KB, journaled at FULL.
• A full disk stops the shard until the gateway restarts, rather than failing the requests whose entries were staged; gateway health shows it as ERROR (see the management paths).

What it saves is everything downstream of the write: most of the disk, of the page cache charged to a container's memory limit, of the writeback, and of what a tailer reads.

Recommendations

• Leave it on. At any load a real deployment sees, the cost does not show, and the journal is a fraction of the size.
• Turn it off (compression: none) only where an entry must be on disk the moment its request completes, or where disk and page cache are cheap and every core is busy. Without compression each entry is placed by its own request thread and committed before the write returns.
• Leave compression_level at 1.

Latency at saturation: the JDK's I/O pollers

Symptom

Driven past the request rate it can sustain, a gateway on virtual-thread I/O pollers serves the connections it already has but leaves new ones waiting: a connection opened while every core is busy can wait seconds for its first request to be read. A load generator that opens all of its connections at once against a saturated gateway shows this as a burst of very late responses (wrk counts them as timeouts). The gateway started through R7Helidon.main (the jar and the images) uses platform-thread pollers, which do not do this.

Why it happens

Every connection runs on a virtual thread, and when it waits for the network the JDK parks it and hands the socket to an I/O poller. With virtual-thread pollers (the JDK default, mode 2) or per-carrier pollers (mode 3), a connection woken by a poller is queued on that poller's carrier. A new connection's thread is started by Helidon's acceptor, a platform thread, and waits in the scheduler's shared queue instead. A carrier runs its own queue until it is empty before looking at the shared one, and at saturation it is never empty, so new connections wait until the load drops. Platform-thread pollers (mode 1) wake every thread through the shared queue, in order.

The measurements are in the design history (upstream-client.md, "New connections starved at saturation"): with per-carrier pollers, a burst of late responses showed up in about a quarter of the runs; with platform-thread pollers, in none. Per-carrier pollers keep a shorter tail for connections that are already open, which is why the mode stays selectable.

What to do

• Leave the default (mode 1) unless every client holds long-lived connections and you have measured the difference.
• R7_POLLER_MODE (1, 2 or 3) chooses the mode; any other value stops startup. An explicit -Djdk.pollerMode replaces both. The setting is internal to the JDK, which may change or drop it. An embedder that does not start through R7Helidon.main sets -Djdk.pollerMode=1 itself.
• Do not run the gateway at saturation. Size it so that peak load stays below the rate it sustains.

Startup: the AOT cache and the garbage collector

What the image does

The gateway image starts from an AOT cache (the JDK's ahead-of-time cache, JEP 483): while the image is built, a training run starts the gateway, sends requests through it, and records the classes it loaded and linked. The image then starts from that record instead of reading and linking those classes from the jars again, which is most of what a JVM does while it starts. The first requests after a start benefit too, as the classes they need are in the cache. Nothing needs configuring: the cache is at /app/r7.aot and the entrypoint uses it.

A cache only fits the exact JDK, jars and JVM flags it was made with. A JVM that finds it does not fit prints a warning and starts without it, as it would with no cache.

Choosing a garbage collector

The image names no collector, so the JVM picks one for the container: G1, or SerialGC with fewer than two CPUs or less than about 2 GB of memory. To choose one, set it in JAVA_TOOL_OPTIONS, for example -XX:+UseZGC for the shortest pauses under load. The throughput and latency numbers on this page were measured with ZGC.

ZGC cannot use the AOT cache at all: the cache is made with compressed object pointers, which ZGC does not use, so the JVM says the cache does not fit and starts without it, noticeably slower than with G1, ParallelGC or SerialGC. Where startup matters more than pause times (an autoscaled or frequently restarted gateway, a gateway with little CPU), leave the collector to the JVM.

Running the jar yourself

java -jar r7-helidon-<version>.jar needs the lib/ directory beside the jar, as the build leaves it. To make a cache for it, run the training once with the JDK and flags you will run with, then start with the cache:

R7_ROUTES_CONFIG=docker/aot-training/routes.yaml R7_JOURNAL_DIR=/tmp/aot-training \
  R7_FINGERPRINT_KEY=aot-training-only-fingerprint-key \
  java -XX:AOTCacheOutput=r7.aot -Dr7.aot.training=true -jar r7-helidon-<version>.jar
java -XX:AOTCache=r7.aot -jar r7-helidon-<version>.jar

The training run exits by itself once it has served its requests. Its throwaway key is only for the training requests; the second command needs your deployment's own fingerprint key.