Journal Tailing and Observability

r7 writes request/response traffic straight to memory-mapped binary files (journals) on disk instead of serializing logs to text or making network calls on the request thread — that's what keeps the hot path fast. To get that data into Grafana, ELK, ClickHouse or a WARC archive, you run a Tailer: a separate process/sidecar that reads the journals asynchronously.

graph LR
    Client -->|HTTP| R7[r7 gateway Container]
    R7 -->|Binary Write| Vol[(Shared Volume: /journals)]
    Vol -->|Binary Read| Tailer[r7 Tailer Container]
    Tailer -->|JSON / Batch Insert| Ops[Observability Stack]

Quick start

The common setup: the gateway journals headers and bodies, the tailer writes the metadata and headers as JSON lines to its standard output, and the bodies go to WARC files. The tailer needs no configuration for this; its defaults are exactly that.

1. Journal the route in full. In routes.yaml:

routes:
  - id: my-route
    match:
      - PathPrefix:
          prefix: /api
    upstream:
      targets:
        - url: http://backend:8080
    journal:
      request:
        level: FULL
      response:
        level: FULL

2. Run the gateway and the tailer on one volume. The gateway writes the journals, the tailer reads them. Add the two services to the compose file that runs your upstream (backend in the route above):

services:
  r7-api:
    image: ghcr.io/ethlo/r7-gateway:main
    ports:
      - "8888:8888"
    environment:
      # Required: the key behind the fingerprints in place of redacted values.
      R7_FINGERPRINT_KEY: ${R7_FINGERPRINT_KEY:?set it in .env}
    volumes:
      - ./config:/app/config:ro
      - r7-journals:/journals:rw

  r7-tailer:
    image: ghcr.io/ethlo/r7-tailer:main
    volumes:
      - r7-journals:/journals:ro
      - r7-warc:/warc:rw
      - r7-tailer-checkpoints:/checkpoints:rw

volumes:
  r7-journals:
  r7-warc:
  r7-tailer-checkpoints:

Next to it, create the gateway's fingerprint key once, and keep the file. Only the gateway needs it. If config/server.yaml sets fingerprint_key instead, drop the environment block.

(umask 077; set -C; echo "R7_FINGERPRINT_KEY=$(openssl rand -base64 32)" > .env)

3. Read the result. Each exchange is one JSON line on the tailer's standard output, which Promtail, Fluent Bit, Vector or your Docker logging driver picks up. The line holds the timing, status, sizes and the headers of every leg (redacted values appear as fingerprints), and, for a complete exchange with a body, a warc pointer to its records in /warc. The line is one physical line; this one is abbreviated and pretty-printed for reading. Every field is listed under The JSON line:

{
  "request_id": "01JA0Q6R8X4V2N7M3K5T9B1C0D",
  "route_id": "items",
  "start": "2026-10-01T12:00:00.000120Z",
  "duration": 0.004290,
  "client_request": {"method": "POST", "path": "/items", "headers": {"host": ["api.example.com"]}, "body_bytes": 212},
  "client_response": {"status": 201, "headers": {"content-type": ["application/json"]}, "body_bytes": 1274},
  "warc": {"file": "r7-1759320000000-6f1c….warc.zst", "offset": 48211, "length": 1873}
}

The /warc files are standard WARC; see the tailer. Journals grow until something deletes them: before sustained use, add the reaper from Retention. Everything else on this page is reference for tuning this setup: the journal levels, redaction, every tailer property, other output layouts and retention.

1. Journal levels and redaction

Tailers only see what the gateway actually writes. Set the levels per route, group or the whole file, and raise or lower them with rules, in routes.yaml:

routes:
  - id: my-route
    match:
      - PathPrefix:
          prefix: /api
    upstream:
      targets:
        - url: http://backend:8080
    journal:
      request:
        level: METADATA
      response:
        level: METADATA
      on_response:
        - id: failures
          match:
            - Status:
                include: [5xx]
          response: { level: HEADERS }   # more detail when things break

What each level records, and how rules resolve the levels, is in Configuration §7. Each rule that fires is recorded in the gateway.journal.rule attribute, by its id or its path, and a rule that could not be evaluated in gateway.journal.rule_error, and counted in the management view's rule_errors whether or not the exchange was journaled; where and how journals are written on disk is the storage block. Requests refused before any route matches (no route, ambiguous path, TRACE and similar) are journaled only when configured, see Unrouted Requests.

Capping bodies

At FULL the journal keeps every body whole unless told otherwise. Request bodies are bounded by limits.max_entity_size; response bodies by nothing, so a route that serves a 2 GB download writes 2 GB to the journal, and a tailer holds every fragment of an exchange in its heap until the exchange ends. truncate_body_at on a direction keeps the body's first bytes instead; the traffic itself streams whole:

journal:
  request:  { level: METADATA, truncate_body_at: 64KB }
  response: { level: METADATA, truncate_body_at: 64KB }
  on_response:
    - id: failures
      match:
        - Status:
            include: [5xx]
      response: { level: FULL }      # the first 64KB of each failure's body

A capped body is marked in the record: the end record carries a truncation flag per direction, the body's checksum covers the prefix that was kept, the JSON line adds body_truncated: true to the leg, and the WARC record says WARC-Truncated: length with the digest of the prefix. A body exactly truncate_body_at long is kept whole and not flagged. How the cap is inherited and which rules may set it is in Configuration §7.

Route ids and upstream target URLs are recorded in every exchange's journal attributes (gateway.route.id, gateway.target; the JSON line has them as route_id and upstream_request.targets), and the journal stores ISO-8859-1 text only. A route declared in a group is also recorded with the group's id as gateway.route.group (in the JSON line's attributes). A route or group id or target URL with a character outside ISO-8859-1 is refused at startup, naming the field.

Redacted header and query parameter values

Headers. At HEADERS and FULL, every header name is journaled, but a value is written as sent only when its name is on the safe list for its direction (request or response). The built-in lists hold names such as host, user-agent, content-type, etag, x-forwarded-for and sec-fetch-mode; everything else, such as authorization, cookie, set-cookie or a custom API key header, is written as a fingerprint.

Query parameters. The query is part of the request line, which every level records, METADATA included. Each parameter name is journaled as sent, and its value is written as sent only when the name is on safe_query_parameters. There is no built-in list, because no parameter name is safe everywhere, so with nothing configured every value is fingerprinted:

GET /search?page=2&api_key=s3cret&q=shoes HTTP/1.1                       # as sent
GET /search?page=2&api_key=fp:R5XU2m_VAik&q=fp:0uCr61zmqQQ HTTP/1.1      # journaled, with page safe

• Case. Names on safe_query_parameters match regardless of case, as header names do. Query parameter names are case sensitive on the wire, so when your upstreams tell id from ID, set safe_query_parameters_case_sensitive: true.
• Encoding. Names and values are compared after percent-decoding, as the upstream reads them: p%61ge and page are the same parameter, and q=a+b and q=a%20b get the same fingerprint.
• Repeats. Each occurrence of a repeated parameter is redacted on its own; a safe name is safe every time it appears.
• Bare parameters. A parameter without = (?s3cret-token) is fingerprinted whole unless its name is safe. Empty values (?q=) stay empty.

Both request lines are redacted the same way: the one the client sent and the one forwarded upstream, after filters such as SetQueryParameter have changed it.

The safe lists are set in storage.journal_security; see Journal Redaction for the keys. Redaction applies to the journal only; it never changes what a client or upstream receives.

Fingerprints. A fingerprint is fp: followed by 11 base64url characters, for example fp:R5XU2m_VAik: the first 64 bits of HMAC-SHA-256 of the value, keyed with storage.journal_security.fingerprint_key.

The same value always gives the same fingerprint under the same key, so you can see that two requests, two replicas, or records from before and after a restart carried the same value. Without the key, nobody who reads the journal can check a guess, which matters for low-entropy secrets such as Authorization: Basic with a known user name and a common password, a short API key or a role=admin cookie. The characters are URL-safe, so a fingerprint stands in a query string, a JSON string or a WARC header without escaping.

The key. r7 refuses to start without it. Set it in server.yaml, or set only the R7_FINGERPRINT_KEY environment variable, which is the default when the file does not:

storage:
  journal_security:
    fingerprint_key: ${R7_FINGERPRINT_KEY}   # at least 32 characters, e.g. `openssl rand -base64 32`

Keep the key out of anything a journal reader can see: anyone who has it can confirm a guess. Give every replica of a deployment the same key so their fingerprints correlate. To rotate it, change it: records already written keep the fingerprints they have, and values journaled before and after the change no longer correlate. Nothing reads a fingerprint back, so there is no old key to keep.

Journals written by earlier versions keep the forms they were written with, id:sha256: (an unkeyed, truncated SHA-256) and id:hmac:; nothing rewrites them.

2. Mount the shared volume

Gateway and tailer share the journal_dir volume: the gateway writes, the tailer only reads.

3. The tailer

Image: ghcr.io/ethlo/r7-tailer:main

The tailer reads the journals once and writes each exchange to one or both of two outputs:

• WARC files: WARC 1.1 records, the standard web-archiving format used by the Internet Archive and national libraries, in rotating .warc.zst files (one independent Zstandard frame per record, so zstd -d alone decompresses a file back to plain WARC). This is the archive: durable, replayable and readable by any WARC tool, such as pywb, for compliance archiving, incident forensics or replaying traffic against a new backend.
• JSON lines: one JSON object per exchange, to standard output for Promtail (Grafana Loki), Fluent Bit, Vector or a Docker logging driver, or to rotating files. This is the log.

With both on, a body is stored once, in the WARC file: the JSON line carries a warc pointer to the exchange's records instead of the bodies. A body the WARC output does not store (one of an incomplete exchange, or any body with warc.bodies: false) goes into the JSON line when json.bodies is true. With both bodies settings false, no body is stored anywhere; the records and lines still say there was one, and which, by its size and checksums.

If writing the JSON line fails, the exchange's WARC records are taken back before the tailer retries it, so a retry never archives an exchange twice.

Configuration: like the gateway, the tailer reads a YAML file: tailer.yaml in the working directory by default (/app/config in the image), or the path in the TAILER_CONFIG environment variable. It supports the same ${VAR:default} interpolation as routes.yaml/server.yaml. Without a file, every field takes its default, so a bare container tails /journals to standard output as JSON lines without bodies, and archives the exchanges that have a captured body in WARC files under /warc.

At least one output must be enabled.

warc

The WARC profile specifies what an exchange becomes in the files: the records, the WARC-R7-* fields and the rules a reader can rely on. design/warc.md explains why they are shaped this way.

CDXJ index. With cdxj_index: true, every sealed r7-….warc.zst gets an r7-….cdxj beside it: one line per exchange, sorted by SURT key and timestamp, holding the URL, status, mime type, payload digest and the file name, offset and length of the client response record. This is the index format pywb and OutbackCDX read, so a lookup by URL and time goes straight to the record without scanning the archive. The index is built from the finished WARC file and renamed into place before the WARC file is, so a sealed .warc.zst always has its index; one sealed after a crash is indexed on the next start. Requests other than GET carry __wb_method=<method> in their key, as pywb does; request bodies are never copied into the index.

json

The JSON line

The output is JSON Lines: one compact JSON object per exchange, one object per line, UTF-8. A proxied GET with headers journaled, archived in a WARC file under exchanges: all, looks like this (pretty-printed here):

{
  "request_id": "01JA0Q6R8X4V2N7M3K5T9B1C0D",
  "route_id": "items",
  "start": "2026-10-01T12:00:00.000120Z",
  "end": "2026-10-01T12:00:00.004410Z",
  "duration": 0.004290,
  "remote_address": "10.0.0.7",
  "remote_address_source": "SOCKET",
  "client_request": {
    "level": "HEADERS", "method": "GET", "path": "/items", "query": "page=fp:b9ZtS0R2qzY", "protocol": "HTTP/1.1",
    "headers": {"host": ["api.example.com"]}, "header_bytes": 142, "body_bytes": 0
  },
  "upstream_request": {
    "level": "HEADERS", "method": "GET", "path": "/v1/items", "query": "page=fp:b9ZtS0R2qzY", "protocol": "HTTP/1.1",
    "targets": ["http://backend:8080"], "headers": {"host": ["backend:8080"]}
  },
  "upstream_response": {
    "level": "HEADERS", "protocol": "HTTP/1.1", "status": 200, "reason": "OK",
    "headers": {"content-type": ["application/json"]},
    "start": "2026-10-01T12:00:00.000300Z", "first_byte": "2026-10-01T12:00:00.004100Z",
    "end": "2026-10-01T12:00:00.004380Z", "duration": 0.004080
  },
  "client_response": {
    "level": "HEADERS", "protocol": "HTTP/1.1", "status": 200, "reason": "OK",
    "headers": {"content-type": ["application/json"]}, "header_bytes": 98, "body_bytes": 1274
  },
  "warc": {"file": "r7-1759320000000-6f1c….warc.zst", "offset": 48211, "length": 1873}
}

The rules every field follows:

• A field with no value is left out, never written as null. Without an end event (incomplete is TIMED_OUT, CAPACITY_EVICTED or SHUTDOWN), timing, status and sizes are unknown, and are left out rather than written as zero.
• Timestamps are RFC 3339 in UTC with microseconds. Durations are seconds, as a number with six decimals.
• Header and attribute values are always arrays of strings, in the order they were sent or set, so a header sent once and one sent twice have the same type. Values outside the safe lists are fingerprints.
• Enumerated values are upper case.

Fields of the exchange:

Fields of the four leg objects (client_request, upstream_request, upstream_response, client_response):

A consumer should ignore fields it does not know.

The caller's identity

When an auth filter verified the caller, the line carries who they are, in plain text:

"identity": {"scheme": "jwt", "issuer": "https://auth.example.com/realms/acme",
             "subject": "6f1c2b8e-4d3a-4f0e-9a51-2c7d8e9f0a1b", "username": "alice"}

The journal is an audit record, and who made a request is the question it exists to answer, so the identity is recorded verbatim, like the client address, in any script. The credentials themselves stay fingerprinted: the Authorization header's value is never written in plain text. If two filters on a route set the identity, the last one wins.

Other layouts

Each of these is a tailer.yaml and replaces the defaults only where it says.

Every exchange in the WARC archive, with a CDXJ index, so every complete exchange's JSON line has a warc pointer:

warc:
  exchanges: all
  cdxj_index: true

JSON to rotating files for a loader instead of standard output:

json:
  output: file
  output_dir: /json
  max_file_age: 5m

JSON lines alone, bodies included:

warc:
  enabled: false
json:
  bodies: true

4. Retention: The Reaper

Image: ghcr.io/ethlo/r7-reaper:main

The tailer is read-only by design (see §2's warning) — something else has to delete a sealed segment once it is no longer needed, or journal_dir grows without bound. That something is the reaper: a small standalone process whose only job is deleting old segments from a shared journal volume.

Configuration: same YAML config mechanism as the tailers — reaper.yaml by default, overridable via the REAPER_CONFIG env var.

Example config/reaper.yaml:

journal_dir: /journals
ttl: 7d
# Delete as soon as the tailer is done with a segment (and it is an hour old), not after 7 days
tailers:
  - /checkpoints/tailer

Who may write what. Each component writes only what it owns and reads the rest, and the mounts enforce it:

So the reaper can delete what the gateway wrote, a tailer cannot, and the reaper cannot touch anything a tailer wrote. The one thing that crosses a boundary, the reaper trusting a tailer's checkpoint file, can only keep a segment longer: anything short of "done" leaves it for ttl.

Example Docker Compose Integration:

services:
  r7-api:
    image: ghcr.io/ethlo/r7-gateway:main
    volumes:
      - r7-journals:/journals:rw

  r7-tailer:
    image: ghcr.io/ethlo/r7-tailer:main
    volumes:
      - ./config/tailer.yaml:/app/config/tailer.yaml:ro
      - r7-journals:/journals:ro                # reads only; retention is the reaper's job
      - r7-warc:/warc:rw
      - r7-tailer-checkpoints:/checkpoints:rw   # its own progress

  r7-reaper:
    image: ghcr.io/ethlo/r7-reaper:main
    volumes:
      - ./config/reaper.yaml:/app/config/reaper.yaml:ro
      - r7-journals:/journals:rw                         # the only container that deletes from it
      - r7-tailer-checkpoints:/checkpoints/tailer:ro    # read to learn what the tailer is done with; never its output

volumes:
  r7-journals:
  r7-warc:
  r7-tailer-checkpoints:

5. Where the data goes next

The JSON lines go to any log shipper and on to ClickHouse, Loki or an OpenTelemetry backend; the WARC files go to any WARC reader. Where r7 fits shows how, with a minimal configuration for each.