The WARC profile

The tailer's WARC output is standard WARC 1.1, so any WARC reader can open it. This page is the contract for what r7 puts in those files on top of the standard: which records an exchange becomes, the WARC-R7-* fields, and the rules a reader can rely on. Every file states the profile version it follows in its warcinfo record. How to run the tailer and configure the output is in Journaling.

Files


Name	`<file_prefix>-<millis>-<uuid>.warc.zst`, where `<millis>` is when the file was opened
While written	`<name>.open`. Never read a `.open` file: it is renamed once finished, and a file left `.open` by a crash is cut back to its last complete exchange and renamed on the tailer's next start
Sealed	Renamed to `<name>` only after its contents are flushed to disk. A sealed file never changes again
Compression	Each record is one independent Zstandard frame with its content size and checksum, and the frames are concatenated. `zstd -d` turns a file into a plain WARC file; a reader with an offset can decompress one record without touching the rest
Index	With `cdxj_index: true`, a sorted `<file_prefix>-<millis>-<uuid>.cdxj` beside it, sealed before the WARC file is. See CDXJ index

A file rolls over between exchanges, never inside one: the records of one exchange are always in the same file, and an exchange larger than max_file_size makes that file larger than the limit.

The `warcinfo` record

Every file starts with a warcinfo record, Content-Type: application/warc-fields:

software: ethlo-r7-tailer-warc
format: WARC File Format 1.1
conformsTo: https://iipc.github.io/warc-specifications/specifications/warc-format/warc-1.1/
r7-profile: 1

r7-profile is the version of this page the file follows. A reader that finds a version it does not know should stop rather than guess. Every other record names the file's warcinfo in WARC-Warcinfo-ID.

One exchange, up to four records

An exchange becomes up to four records, written together and in the order the messages happened:

#	`WARC-Type`	Message	`WARC-Target-URI`	Holds the body
1	`request`	client to r7	as the client asked	the request body
2	`request`	r7 to the upstream	as r7 forwarded it	no, see below
3	`response`	upstream to r7	as r7 forwarded it	no, see below
4	`response`	r7 to the client	as the client asked	the response body

A record is written for each message whose start line was journaled. A route with journal level NONE for the response has no response records; an exchange with nothing journaled has none at all. A request at NONE still gets its request records when the response is journaled, since the journal then records it at METADATA.
Records 2 and 3 exist only for an exchange that was proxied. A request that a filter answered itself (a rate limit, a redirect) has records 1 and 4.
Record 3 exists only when an upstream sent a response head. A proxied request whose upstream refused the connection, timed out or broke the protocol before answering has records 1, 2 and 4, and record 4 is r7's own 502, 503 or 504. A head r7 refuses to relay, such as a 101 the client never asked for, is still record 3, followed by r7's 502 as record 4.
Every record of an exchange names all the others in WARC-Concurrent-To, and carries the same WARC-R7-Request-Id.
An incomplete exchange (the gateway stopped before it finished, or it was evicted while being reassembled) gets no records. Its JSON line still says what is known about it.
With exchanges: with_body, only exchanges with a captured request or response body get records; with exchanges: all, every exchange does.

Each record's block is an application/http message: the start line and headers as journaled, a blank line, and the body when this record stores it. Header values are ISO-8859-1, as in the journal; redacted values appear as their fingerprints.

r7 streams bodies without changing them, so the body r7 forwarded is the body the client sent, and the body the client got is the body the upstream returned. Each is stored once: the request body in record 1, the response body in record 4.

Where the body is

WARC-Payload-Digest is always the digest of the message's body, wherever the bytes are. A record says plainly when it does not hold them:

Case	Record	Fields
The body is in this record	`request` / `response`	`WARC-Payload-Digest`
Records 2 and 3, always: any body is in record 1 or 4 of the same exchange	`request` / `response`	`WARC-Truncated: unspecified`, and `WARC-Payload-Digest` when there is a body
Record 3 for a head r7 refused to relay: its body was never read, and record 4 is r7's own answer	`response`	`WARC-Truncated: unspecified`, no digest
`warc.bodies: false`: the body was captured, but this output leaves it out	`request` / `response`	`WARC-Truncated: unspecified`, `WARC-Payload-Digest`. The JSON line holds the body when `json.bodies` is `true`
A body crossed the wire, but the journal level was below `FULL`	`request` / `response`	`WARC-Truncated: unspecified`, no digest
The journaled body failed its checksum when read back	`request` / `response`	`WARC-Truncated: unspecified`, `WARC-R7-Checksum-Mismatch: true`, no digest. The damaged bytes are never archived
Records 1 and 4: the journal kept only the body's first `truncate_body_at` bytes	`request` / `response`	`WARC-Truncated: length`, and `WARC-Payload-Digest` of the prefix the record holds. Never a `revisit`, and never the target of one: a prefix is not the payload. Records 2 and 3, and a body left out by `warc.bodies: false`, also carry `length` in place of `unspecified`, since their digest is of the same prefix
The same body is already stored by an earlier exchange	`revisit`	The WARC 1.1 `identical-payload-digest` profile: `WARC-Refers-To`, `WARC-Refers-To-Target-URI`, `WARC-Refers-To-Date`, `WARC-Truncated: length`
Records 1 and 4: no body	`request` / `response`	No digest, no `WARC-Truncated`

A record without WARC-Truncated holds its whole body. Its headers are what the journal level kept. Records 2 and 3 never store a body, so they carry WARC-Truncated even when the message had none. The body of record 2 is in record 1, and the body of record 3 is in record 4, found through WARC-Concurrent-To, except for a head r7 refused to relay, whose body is in no record. When that record is a revisit, its WARC-Refers-To leads to the earlier record that holds the bytes; when it carries WARC-Truncated: unspecified, this output does not hold the body, and WARC-Truncated: length on a request or response record means it holds the body's first bytes only.

Deduplication across exchanges remembers the last dedup_cache_entries digests per tailer process. A body seen again after that, or after a restart, is stored again in full; a revisit always points at an earlier record written by the same tailer.

Fields

Standard fields with r7-specific values:

Field	Value
`WARC-Date`	When r7 began capturing that message, to the precision the journal recorded: the client request when it arrived, the forwarded request when r7 started sending it, both responses when the upstream's response headers had arrived (or, for a response r7 produced itself, when the request arrived). Not when the tailer wrote the record, so it matches the exchange's JSON line. When the journal lacks that timestamp, a forwarded message falls back to the next one recorded: the upstream's end (responses only), then the client request's arrival, and with none at all the record is dated when the tailer wrote it
`WARC-Target-URI`	`http://` + the request's `Host` header + the request target. r7 does not journal whether the client connection used TLS, so the scheme is always `http`. Without a `Host` header, `urn:r7:request:<request id>`
`WARC-Payload-Digest`	`sha256:` and the base32 SHA-256 of the body
`WARC-Record-ID`	`urn:uuid:` and a random UUID

The r7 fields. A reader that does not know them can ignore them, as WARC requires:

Field	On	Meaning
`WARC-R7-Request-Id`	every exchange record	The exchange's request id, the same as `request_id` in its JSON line
`WARC-R7-Client-IP`	records 1 and 4	The client's address as r7 resolved it (see `trusted_proxies`). `WARC-IP-Address` is not used for this: it means the server a record was fetched from
`WARC-R7-Sequence`	every record	The record's number within its file: 0 for the `warcinfo` record, then one more for each record, with no gaps
`WARC-R7-Group-End`	the last record of each exchange, and the `warcinfo` record	`true`. The records from just after one group end up to the next one belong together
`WARC-R7-Checksum-Mismatch`	a record whose body failed its checksum	`true`

What a reader can rely on

A gap in WARC-R7-Sequence means records are missing. A file that lost data in the middle can otherwise read as a shorter file that looks whole. A reader that cares about completeness checks the sequence.
Groups are whole. A sealed file never ends partway through an exchange, and an exchange is never split across files.
A failed write is taken back. If the tailer fails partway through an exchange, it takes the records back before trying again, so a retry never leaves a second copy.
A crash can archive an exchange twice. An exchange is checkpointed only once both outputs have it. A crash after its WARC records are written but before the checkpoint replays it on the next start, and it is written again: delivery across a crash is at least once. A reader that counts exchanges deduplicates on WARC-R7-Request-Id.
The JSON line points here. With both outputs on, the JSON line of every exchange this output archived carries warc: {file, offset, length}, the sealed file name and the byte range of the exchange's records. Decompress from offset to read them. A line without warc is an exchange this output left out: incomplete, or without a body under exchanges: with_body. See The JSON line.