Serde & Base64

Serde is buffer serialization for storage, built on the same Types engine that Packet uses on the wire. A schema codec stores zero keys and zero type bytes — the schema implies the names, order, and widths — so a record that JSON-encodes to ~80 bytes packs into 5–20. Base64 is the buffer-native escape hatch for stores that cannot hold raw buffers. Together they are the serialization layer under every persistence path in the framework.

Where Serde sits in the stack

One engine, four consumers:

Consumer	How it uses Serde
DataDriver	Codec = Serde.schema({...}) packs profile data into buffers before the backend write; Codec = "Auto" infers the schema from Defaults with extras enabled.
MemoryDriver	Every value is buffer-serialized — through the configured codec, or adaptively without one. Falls back to base64 when a store rejects raw buffers.
MessageDriver	Cross-server payloads pack through Serde, then base64 (MessagingService carries text), under a ~1KB guard.
Replica	Serde.struct codecs power replication: full snapshots for late joiners, field-indexed deltas per tick — one changed NumberU16 is 4 bytes.

The wire and storage split deliberately: Packet ships Instance references out-of-band as remote-event arguments, but a DataStore has no analog — so Serde rejects Instances everywhere, at schema compile time ("Instance" in a spec) and at encode time (an Instance inside an Any), with an error instead of corruption.

Mental model

The schema is the contract. A dictionary spec’s keys are sorted once at compile time; fields are then read and written in that fixed order with the exact widths the schema names. Nothing about the shape travels in the buffer — which is why encoding is so small, and why the same schema must decode what it encoded. Two guardrails cover schema drift:

• Version — the codec writes one leading U8; decoding a buffer whose version differs errors (Serde version mismatch: buffer is v1, codec is v2) instead of misreading fields. Pair with DataDriver migrations for actual data evolution.
• Extras — fields present in the data but absent from the schema are preserved adaptively instead of silently dropped (the default drops them: typed fields are zero-overhead precisely because the schema is the contract).

Adaptive mode (Serde.encode/Serde.decode) is the schemaless twin: every value carries 1 type byte and stores at the best-fitting representation — numbers auto-pack into the smallest width that holds them (5 costs 2 bytes total, 300 costs 3, 70000 costs 4), fractions store as F32 when exact there and F64 otherwise, strings and buffers switch to long prefixes past 255 bytes automatically.

Underneath, the engine path shares one scratch cursor with the Types engine — encode and decode are strictly synchronous, complete in one shot, and grow the scratch buffer as needed (a 5KB blob is fine).

Usage

local ReplicatedStorage = game:GetService("ReplicatedStorage")
local Serde = require(ReplicatedStorage.ChloeKernel.Serde)
local Base64 = require(ReplicatedStorage.ChloeKernel.Base64)

local Codec = Serde.schema({
  Coins = "NumberU32",
  Position = "Vector3F24",
  Inventory = { "NumberU16" },          -- list of item ids, 2-byte length prefix
  Title = Serde.optional("String"),     -- 1 byte when absent
  Pouch = "Any",                        -- untyped field: best fit per value
}, { Version = 2, Extras = true })        -- un-schema'd fields ride along

local Packed = Codec:encode({
  Coins = 1250,
  Position = Vector3.new(12.5, 3, -40),
  Inventory = { 7, 12, 99 },
  Pouch = { Note = "hello" },
})
local Back = Codec:decode(Packed)         -- errors on version mismatch

-- Schemaless, for one-off blobs
local Blob = Serde.encode({ Anything = true, Nested = { 1, 2.5, "x" } })
local AsText = Base64.encode(Blob)        -- for stores that need strings
local Again = Serde.decode(Base64.decode(AsText))

-- Typed spec from a sample — tighten by hand where you know the real bounds
local Spec = Serde.infer({ Coins = 0, Name = "x", Tags = { "a" } })
-- { Coins = "NumberU32", Name = "StringLong", Tags = { "StringLong" } }
local Inferred = Serde.schema(Spec)

API reference

Member	Description
Serde.schema(spec, { Version?, Extras? }?) → codec	Compiles a spec into a codec. Version: number? writes/checks a leading U8. Extras: boolean? preserves un-schema’d dictionary fields adaptively (and disqualifies the fast engine).
codec:encode(value) → buffer / codec:decode(buffer) → value	Round trip through the schema.
Serde.optional(spec) → spec	Wraps any spec node: 1 presence byte, then the payload when present. nil round trips.
Serde.encode(value) → buffer / Serde.decode(buffer) → value	Adaptive (schemaless): 1 type byte per value, best-fit widths.
Serde.infer(sample) → spec	Typed spec from a plain table — safe wide widths, never narrowed from the sample’s magnitude.
Serde.struct(spec) → struct	Field-indexed codec for replication-style full/delta encoding. Dictionary specs only, at most 255 fields.
struct:encodeFull(data) / struct:decodeFull(buffer)	Whole-record snapshot in schema order (no version byte, no count).
struct:encodeDelta(changes) / struct:decodeDelta(buffer)	Only the changed fields: [U8 count] then [U8 fieldId][payload] per change. Unknown fields error.
struct.FieldIds	{ [name]: id } — ids assigned from sorted field names, so identical schemas always agree (spec-asserted).
Base64.encode(buffer) → string / Base64.decode(string) → buffer	RFC 4648 with padding, buffer-native. Decode asserts length is a multiple of 4.

Schema grammar

Node	Meaning	Cost
"TypeName"	A wire type from the table below	Exactly its width
{ elementSpec }	List — exactly one element type	2-byte U16 length prefix + elements
{ Key = spec, ... }	Dictionary — string keys, compiled in sorted order	Zero overhead; keys never written
Serde.optional(spec)	Present-or-nil	1 byte absent, 1 + payload present
"Any"	Untyped field: adaptive per value	1 type byte + best-fit payload

Nodes nest arbitrarily: lists of dictionaries, optional lists, dictionaries of lists of optionals. Invalid specs — unknown type names, multi-element lists, non-string dictionary keys, empty tables, "Instance" — error at Serde.schema time, not at first encode.

Type menu

Byte costs are exact and constant per type — the width you pick is the byte cost of that field on every save and every delta, forever. Verified widths from the Types engine:

Type	Bytes	Notes
NumberU8 / NumberS8	1	0–255 / −128–127
NumberU16 / NumberS16	2	0–65535 / ±32767
NumberU24 / NumberS24	3	0–16777215 / ±8388607
NumberU32 / NumberS32	4	0–4294967295 / ±2147483647
NumberF16	2	Half float: max ±65504, sub-normals flush to 0
NumberF24	3	Custom float: more mantissa than F16, 3 bytes
NumberF32 / NumberF64	4 / 8	Single / double precision
NumberVlq	1–8	Variable-length uint to 2^53: 1 byte under 128, 2 under 16384
Boolean8	1	One boolean per byte
Boolean1	1	Eight booleans packed into one byte (takes { boolean } of 8)
NumberU4	1	Two 0–15 numbers per byte (takes { number } of 2)
BooleanNumber	1	One boolean + one 0–127 number, bit-packed
String / StringLong	1 + len / 2 + len	Caps enforced on write: 255 / 65535 bytes — over-cap errors instead of desyncing the stream
Buffer / BufferLong	1 + len / 2 + len	Same caps, raw bytes
Characters	1 + packed	Bit-packed against a reduced character set — cheaper than String for plain text
Vector2S16 / Vector2F24 / Vector2F32	4 / 6 / 8	Integer studs / compact float / full float
Vector3S16 / Vector3F24 / Vector3F32	6 / 9 / 12	Same ladder in 3D
CFrameF24U8	12	F24 position + one byte per Euler axis
CFrameF32U8 / CFrameF32U16	15 / 18	F32 position + 8-bit / 16-bit rotation precision
Color3	3	8 bits per channel
BrickColor	2	Palette number
UDim / UDim2	4 / 8	Scale rides as S16 × 1000 (clamped ±32.767), offset as S16
Rect	16	Four F32
NumberRange	8	Min + Max as F32
Region3	24	Min + Max corners as F32
NumberSequence	1 + 3/keypoint	Time/value/envelope quantized to 8 bits each
ColorSequence	1 + 4/keypoint	Time + RGB quantized to 8 bits each
EnumItem	3	Enum type id (U8) + value (U16); unknown enums error on both sides instead of decoding to nil
Any	1 + payload	Adaptive; storable kinds only (nil, numbers, strings, booleans, tables, buffers, vectors, CFrame, Color3, BrickColor, UDim(2), Rect, NumberRange, sequences, EnumItem, Region3)
Instance	—	Rejected — Instances have no representation outside a running server

Coercion

Gameplay math should not have to track wire widths. Writers coerce:

• Integer types round (99.6 → 100) and clamp into range (300 → 255 for NumberU8, −5 → 0). Non-numbers error (cannot coerce Instance into NumberU8) — a silent 0 fallback would corrupt data.
• Boolean8 takes truthiness: 1 → true, nil → false.
• String types tostring non-strings: 42 → "42".

Coercion is byte-identical on the fast engine and engine paths (spec-asserted).

Inference and the “Auto” codec

Serde.infer(sample) builds a typed spec from plain data, with one hard rule: widths are never narrowed from a sample’s magnitude, because samples lie — Coins = 0 today is not Coins’ maximum.

Sample	Inferred
Non-negative integer	NumberU32
Negative integer	NumberS32
Fractional number	NumberF64
string / buffer	StringLong / BufferLong
boolean	Boolean8
Vector3 / Vector2 / CFrame	Vector3F32 / Vector2F32 / CFrameF32U16
Color3, BrickColor, UDim, UDim2, Rect, NumberRange, EnumItem	Their own wire types
Homogeneous array	{ elementSpec }
String-keyed table	Nested dictionary spec
Empty table, mixed keys, heterogeneous array	"Any" — the shape is unknowable from the sample

Edit the returned spec wherever you know real bounds (Health that caps at 100 belongs in NumberU8). DataDriver’s Codec = "Auto" is exactly:

Serde.schema(Serde.infer(Defaults), { Extras = true })

— typed fields from your Defaults at safe widths, with extras on so fields you add later persist adaptively before you type them. Tighten later; nothing is lost in the meantime.

Struct codecs

Serde.struct(spec) compiles a dictionary schema into a field-indexed codec — the primitive under replica delta replication:

• Field names sort once; ids 1..n assign in that order. Identical schemas on server and client produce identical ids with zero negotiation.
• encodeFull writes the whole record in schema order — the late-joiner snapshot.
• encodeDelta(changes) writes [U8 count] then [U8 id][payload] per changed field. A one-field NumberU16 delta is 4 bytes (count + id + value). Fields outside the schema error (field "Hacked" is not in this struct schema).
• Nested specs work as field values — a delta containing a whole sub-dictionary re-sends that field’s full payload.

local Struct = Serde.struct({ Wave = "NumberU16", BossUp = "Boolean8", Title = "String" })
local Snapshot = Struct:encodeFull({ Wave = 12, BossUp = true, Title = "Storm" })
local Delta = Struct:encodeDelta({ Wave = 13 }) -- 4 bytes
local Changes = Struct:decodeDelta(Delta)       -- { Wave = 13 }

The fast engine

Schema and struct codecs compile to a second implementation when every type in the spec supports it: byte widths resolve at compile time, so encode allocates one buffer at the exact final size and writers run with zero capacity checks. The techniques come from light/holy by @hardlyardi (MIT) — techniques adopted, no code copied.

Two properties, both spec-asserted:

1. Byte-identical output. The fast path and the engine path produce the same bytes for the same value, cross-decode each other, and share the same coercion — the format never changes, only the speed. The F16/F24 writers replicate the engine’s exact bit layout including its mantissa-carry rounding fix.
2. Automatic fallback. A spec using anything the fast engine does not cover falls back to the engine path silently. In a struct, one uncovered field sends the whole struct down the engine path.

Fast-engine coverage	Falls back to the engine
All fixed-width numbers (U8–S32, U24/S24, F16/F24/F32/F64), NumberVlq, Boolean8, Color3, all Vector2/Vector3 variants, String(Long), Buffer(Long), plus optional/list/dictionary composition of those	Any, CFrames, sequences, EnumItem, BrickColor, UDim(2), Rect, NumberRange, Region3, Characters, bit-packed types — and any codec with Extras = true

Measured on kernel hot paths (see Benchmarks): fixed-schema encode went 463ns → 285ns (38% faster) and decode 514ns → 367ns in a same-machine A/B against the engine path; dynamic schemas run 22–27% faster. Absolute throughput: 3.5M encodes/s and 2.7M decodes/s for a 3-field schema, 3.7M/3.4M ops/s for 2-field struct deltas.

Note

The F16/F24/U24/S24 and bit-packed codepaths use buffer.readbits/writebits — real Roblox APIs (2024) that selene 0.28’s bundled standard library predates. The repo ships buffer_bitops.yml as a selene std extension so linting recognizes them; nothing to configure at runtime.

Hostile decode

Stored bytes are still input. The decode paths are hardened and fuzz-covered (see Fuzz & Audit):

• A crafted huge length prefix (0xFFFFFFFF on a long string/buffer) errors instead of allocating.
• Unknown adaptive type ids error with a clear message rather than misreading the stream.
• Nested table bombs hit a depth cap, not the call stack.
• A NumberVlq running past 8 continuation bytes errors — beyond 2^53 a crafted stream would only accumulate garbage.
• Oversized String/Buffer values error on write — a wrapped U8 length prefix would silently desync everything after it.

Base64

RFC 4648 with padding, implemented buffer-to-buffer (--!native, lookup tables in buffers, no string concatenation churn):

local Text = Base64.encode(Codec:encode(Data)) -- buffer → string, +33% size
local Buf = Base64.decode(Text)                -- string → buffer; length must be % 4

Base64 exists because some stores only hold JSON-safe text. Where the framework uses it:

Consumer	Behavior
MemoryDriver	Buffer support is feature-detected per store by probing a raw buffer write; only a type error proves buffers unsupported (a throttle during the probe must not lock the driver into base64’s ~33% overhead). Fallback values carry a versioned CK1: prefix so raw strings written by other code are never misread. ForceBase64 = true skips the probe.
MessageDriver	Payloads are always Serde-packed then base64’d — MessagingService carries text. The ~1KB ceiling is checked against the packed length, with an error naming both sizes.
DataDriver	Codec’d profiles store as raw buffers on backends that support them (DataStores do, natively); on backends that do not, the save wraps as a { F = "B64", P = <base64> } envelope. See Backends.

Design notes

Danger

Never yield mid encode/decode. The engine path shares one scratch buffer and cursor with the Types engine — a yield with an encode in flight corrupts whatever resumes into it. Ordinary use cannot trigger this (codecs call no user code and never yield internally); it matters if you extend the Types engine with custom readers or writers — keep them synchronous.

Caution

Adaptive is not free. Every Any value, Extras field, and Serde.encode value pays 1 type byte, and extras additionally write their key strings. Adaptive mode is the safety net for shapes you have not typed yet — the schema is where the size wins live. The debug panel’s NET inspector shows serialized payloads as hex if you want to see exactly where bytes go.

Caution

Without Extras, unknown fields are dropped — encode a table with a field the schema does not name and the decode simply lacks it. This is the documented default, and it is the price of zero-overhead typed fields. If your data grows fields faster than you type them, turn Extras on (or use Codec = "Auto" in DataDriver, which does).

• Version is a single U8 — versions 0–255. The version byte guards format identity; use DataDriver SchemaVersion + Migrations for evolving the data itself.
• Serde.struct accepts dictionary specs only and at most 255 fields (field ids are one byte).
• Empty tables infer as "Any" — an inferred codec round-trips Bag = {} fine, but give real list fields at least one sample element (or hand-write the spec) to get a typed list.