Open Q Language

Storage

On-disk formats: serialized files, splayed tables, enumeration, partitions

Required One chapter of the Open Q Language specification. README.txt states what must be implemented, how conformance is defined, and how rule IDs work.

This chapter specifies what a q process writes to disk and expects to read back. Two runtimes conform if either can load a database the other wrote.

The layout rules below were observed by writing a database with the reference implementation and reading the bytes. Where a header field's meaning could not be determined by observation, it is marked OPEN rather than guessed: see section 6.

STOR-00011. The two file shapes

Every file a q process writes is one of two shapes.

SERIALIZED VALUE   2-byte header  ff 01, then the value in exactly the
                   IPC encoding (ipc.txt 3) with no IPC frame header.
                   Used for any value written with set to a plain path,
                   and for the sym and .d files of a splayed table.

COLUMN VECTOR      16-byte header, then the elements back to back.
                   Used for one column of a splayed table.

A reader distinguishes them by the first byte: ff is a serialized value, fe or fd is a column.

STOR-00022. The column file

offset  size  meaning
0       1     fe  simple column   /  fd  nested or enumerated column
1       1     OPEN (observed 0x20 in every file written)
2       1     TYPE CODE of the vector (core.txt 4)
3       1     ATTRIBUTE (0 none, 1 `s, 2 `u, 3 `p, 4 `g)
4       12    OPEN (observed all zero for simple columns)
16      ..    the elements, fixed width, no separators

STOR-00032.1 The element count is not stored

There is no count field. A reader computes it:

count = (file_size - 16) / width_of(type)

Verified: a 1000-element int column is exactly 4016 bytes, 16 + 1000*4. A 4-element float column is 48 bytes, 16 + 4*8.

This is why a column file can be appended to by appending bytes, and why truncation is a valid delete of trailing rows. It also means a corrupt length is unrepresentable — but a partial write silently shortens the column instead of failing, so writers must write atomically.

STOR-00042.2 Simple columns

Byte 0 is fe. The element width is the type's fixed width: 1 for boolean, byte and char; 2 short; 4 int, real, date, month, minute, second, time; 8 long, float, timestamp, timespan, datetime; 16 guid.

4 floats  ->  fe 20 09 00 <12 zero bytes> then 4 * 8 bytes
4 ints    ->  fe 20 06 00 <12 zero bytes> then 4 * 4 bytes

STOR-00052.3 Nested and enumerated columns

Byte 0 is fd. Two cases share this marker:

ENUMERATED (a symbol column). The name of the enumeration domain is written as a NUL-terminated string at offset 16, and the column holds integer indices into that domain. A symbol column of a splayed table is always stored this way — see section 3.

NESTED (a column whose elements are themselves lists, e.g. a column of strings). The data lives in the column file and a COMPANION INDEX FILE named <column># holds the offsets. Both files are required; a reader that finds one without the other cannot reconstruct the column.

The internal layout of fd files is OPEN (section 6).

STOR-00063. Splayed tables

A splayed table is a DIRECTORY, written with a trailing slash:

`:splay/ set .Q.en[`:.; t]

produces

splay/.d        serialized symbol vector: the column names, IN ORDER
splay/<col>     one column file per column
splay/<col>#    index file, for each nested column only
sym             at the DATABASE ROOT, not inside the table directory

STOR-00073.1 .d gives the column order

The on-disk file order is a directory listing and therefore arbitrary. .d is authoritative: it is a serialized symbol vector naming the columns in their table order. A reader MUST take column order from .d and MUST NOT infer it from the filesystem.

STOR-00083.2 Enumeration is mandatory for symbol columns

A symbol column cannot be splayed as symbols. .Q.en enumerates it against the sym file at the database root, appending any new symbols to that file and writing the column as indices into it. Writing a splayed table with a raw symbol column is an error.

The sym file is a serialized symbol vector, SHARED by every table in the database. It is append-only: an existing symbol keeps its index forever, because every column already written refers to it by position.

STOR-00093.3 Attributes are stored, not recomputed

A column's attribute travels in header byte 3, so `s# survives a write and reload and the reader does not re-sort or re-verify. (Contrast a runtime that discards attributes: it does not conform, per core.txt 5.)

STOR-00104. Partitioned databases

A partitioned database is a directory of partitions, each holding splayed tables:

<root>/sym                       the shared enumeration domain
<root>/<partition>/<table>/.d
<root>/<partition>/<table>/<col>

The partition name is the partition column's value: a date (2024.01.01), a month, a year, or an integer. The partition column is NOT stored as a column inside the table — it is the directory name, and is materialized as a virtual column when the database is loaded.

.Q.dpft[`:pdb; 2024.01.01; `sym; `t]

writes pdb/2024.01.01/t/ with .d and one file per column, applies the parted attribute to the named column, and maintains pdb/sym. Verified layout:

pdb/sym
pdb/2024.01.01/t/.d
pdb/2024.01.01/t/{sym,px,n,s,s#}

STOR-00114.1 par.txt

A root may contain par.txt, whose lines are paths to the directories that actually hold the partitions, spreading one logical database across filesystems. NOT EXERCISED by this specification yet — the layout above was verified against a single-root database only.

STOR-00125. Compression

A compressed file begins with the 8-byte ASCII magic

6b 78 7a 69 70 70 65 64        "kxzipped"

followed by the compressed stream. With .z.zd set to (17;2;6) the stream that follows begins 78 9c, a zlib/deflate header. A 1000-element int column that is 4016 bytes uncompressed is 102 bytes compressed.

.z.zd is (blockSize; algorithm; level) and applies to subsequent writes.

A reader MUST check for this magic before interpreting a file as either shape in section 1. A runtime that ignores .z.zd and writes plain files is not merely slower — it produces files a compressing reader accepts but a compression-expecting toolchain may not, so silent non-compression is a conformance failure worth testing for.

STOR-00136. What is not pinned here

Stated plainly rather than guessed:

  • Header byte 1 of a column file. Every file observed carries 0x20; its meaning is unknown and a reader should not depend on it.
  • Bytes 4..15 of a column header. All zero in every simple column observed.
  • The internal layout of fd (nested/enumerated) files beyond the domain name at offset 16. Observed sizes suggest a reserved region of about 4096 bytes before the payload, but the structure was not decoded.
  • par.txt (section 4.1).
  • Compression algorithms other than 2 (zlib), and the block framing inside a kxzipped stream.

These are gaps in verification, not statements that the features are absent. Closing them requires either more observation or a reference that documents them; until then a runtime targeting byte-level file compatibility must test against the reference directly.

Source: spec/storage.txt