Serde

Serde packs Lua values into buffers for storage and cross-server messages. It runs on the same Types engine that Packet uses on the wire. A schema codec writes no keys and no type bytes: the schema fixes the field names, their order and their widths, so a record that JSON-encodes to about 80 bytes packs into 5 to 20. That also defines the contract: only the schema that wrote a buffer can read it. Base64 turns the buffers into text for stores and transports that only carry strings.

Where Serde sits in the stack

One engine, several consumers:

Consumer	How it uses Serde
DataDriver	Codec = Serde.schema({...}) packs profile data before the backend write. Codec = "Auto" infers the schema from Defaults with Extras on (see Inference and the “Auto” codec).
MemoryDriver	Every value goes through the configured codec, or Serde.encode without one. Falls back to base64 text when a store rejects raw buffers.
MessageDriver	Payloads go through the codec or Serde.encode, optionally Zstd, then base64, because MessagingService carries text.
Replica	Serde.struct codecs: full snapshots for late joiners, field-indexed deltas per tick.
Unreliable state channels (NetDriver)	Serde.struct buffers over UnreliableRemoteEvents.

Packet carries Instance references as extra remote arguments. A DataStore has no equivalent, so Serde rejects Instances everywhere: "Instance" in a schema errors when the schema compiles, and an Instance inside an Any value errors on encode.

Mental model

The schema is the contract. A dictionary schema’s keys sort once at compile time, and fields are then read and written in that order at exactly the widths the schema names. Nothing about the shape travels in the buffer, which is why it is small and why the same schema must decode what it encoded. Two options cover schema drift:

• Version writes one leading U8. Decoding a buffer with a different version raises Serde version mismatch: buffer is v1, codec is v2 instead of misreading fields. It must be an integer from 0 to 255; anything else raises Serde Version must be an integer 0..255 when the schema compiles. Before 0.9.0 a larger version wrapped in the header byte and every later decode failed its check.
• Extras keeps fields that are in the data but not in the schema, written adaptively as key and value pairs after the typed fields. Without it they are dropped: typed fields cost nothing extra because the schema is the whole contract.

Adaptive mode (Serde.encode and Serde.decode) is the schemaless form. Every value carries one type byte and the smallest representation that holds it exactly: 5 costs 2 bytes, 300 costs 3, 70000 costs 4, 20000000 costs 5. A fraction stores as F32 when single precision round-trips it exactly and as F64 otherwise. Strings and buffers switch to a 4-byte length past 255 bytes.

The engine path shares one scratch buffer and cursor with the Types engine. Encode and decode are synchronous, finish in one call, and grow the scratch buffer as needed (a 5,000-byte string is fine).

Usage

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local Serde = require(ReplicatedStorage.ChloeKernel.Serde)

local Codec = Serde.schema({
  Coins = "NumberU32",
  Position = "Vector3F24",
  Inventory = { "NumberU16" }, -- list of item ids: 2-byte length, then 2 bytes each
  Title = Serde.optional("String"), -- 1 byte when absent
  Pouch = "Any", -- untyped field: best fit per value
}, { Version = 2, Extras = true }) -- fields the schema does not name ride along

local Packed = Codec:encode({
  Coins = 1250,
  Position = Vector3.new(12.5, 3, -40),
  Inventory = { 7, 12, 99 },
  Pouch = { Note = "hello" },
})
local Back = Codec:decode(Packed) -- raises on a version mismatch

-- Schemaless, for one-off blobs
local Blob = Serde.encode({ Anything = true, Nested = { 1, 2.5, "x" } })
local Again = Serde.decode(Blob)

-- A typed spec from a sample, then tightened by hand where you know the bounds
local Spec = Serde.infer({ Coins = 0, Name = "x", Tags = { "a" } })
-- { Coins = "NumberF64", Name = "StringLong", Tags = { "StringLong" } }
Spec.Coins = "NumberU32"
local Tight = Serde.schema(Spec)

API reference

Member	Description
Serde.schema(spec, { Version?, Extras? }?) -> Codec	Compiles a spec. Version writes and checks a leading U8 (integer 0 to 255). Extras keeps fields the spec does not name, and turns off the fast engine. Invalid specs raise here.
codec:encode(value) -> buffer	Encodes through the schema. Raises on values it cannot coerce and on lists over 65,535 elements.
codec:decode(buffer) -> value	Decodes through the schema. Raises on a version mismatch.
Serde.optional(spec) -> spec	Wraps any node: one presence byte, then the payload when present. nil round-trips.
Serde.encode(value) -> buffer	Adaptive encode: one type byte per value, smallest exact width.
Serde.decode(buffer) -> value	Adaptive decode.
Serde.infer(sample, { LegacyIntegers? }?) -> spec	A typed spec from a plain value. Every number infers as NumberF64; LegacyIntegers = true reproduces the pre-0.9.0 NumberU32/NumberS32 inference.
Serde.struct(spec) -> Struct	Field-indexed codec for full and delta encoding. Dictionary specs only, at most 255 fields.
struct:encodeFull(data) / struct:decodeFull(buffer)	Whole record in field order. No version byte, no count.
struct:encodeDelta(changes) / struct:decodeDelta(buffer)	Changed fields only: a U8 count, then a U8 field id and the payload per change. A field outside the schema raises.
struct.FieldIds	{ [name]: id }, assigned 1 to n from the sorted field names.

Schema grammar

Node	Meaning	Cost
"TypeName"	A type from the menu below	Exactly its width
{ elementSpec }	A list of one element type	U16 length, then the elements. At most 65,535 elements.
{ Key = spec, ... }	A dictionary with string keys, compiled in sorted order	No overhead; keys are never written
Serde.optional(spec)	Present or nil	1 byte absent, 1 plus the payload present
"Any"	Untyped: adaptive per value	1 type byte plus the best-fit payload

Nodes nest freely: lists of dictionaries, optional lists, dictionaries of lists of optionals. Invalid specs (unknown type names, lists with more than one element type, non-string dictionary keys, empty tables, "Instance") raise in Serde.schema, not on the first encode.

Type menu

Widths are fixed per type. The width you pick is the cost of that field on every save and every delta.

Type	Bytes	Notes
NumberU8 / NumberS8	1	0 to 255 / -128 to 127
NumberU16 / NumberS16	2	0 to 65,535 / -32,768 to 32,767
NumberU24 / NumberS24	3	0 to 16,777,215 / -8,388,608 to 8,388,607
NumberU32 / NumberS32	4	0 to 4,294,967,295 / -2,147,483,648 to 2,147,483,647
NumberF16	2	Half float, magnitudes up to about 65,504. Values under 2^-14 store as 0.
NumberF24	3	17-bit mantissa, 6-bit exponent
NumberF32 / NumberF64	4 / 8	Single and double precision. F64 holds integers exactly up to 2^53.
NumberVlq	1 to 8	Unsigned integer up to 2^53: 1 byte under 128, 2 under 16,384
Boolean8	1	One boolean per byte
Boolean1	1	Eight booleans in one byte (takes a list of 8)
NumberU4	1	Two numbers from 0 to 15 in one byte (takes a list of 2)
BooleanNumber	1	One boolean and one number from 0 to 127, bit-packed
String / StringLong	1 + length / 2 + length	Capped at 255 / 65,535 bytes. Longer values raise on write.
Buffer / BufferLong	1 + length / 2 + length	Same caps, raw bytes
Characters	1 + packed	Bit-packed against a reduced character set, cheaper than String for plain text
Vector2S16 / Vector2F24 / Vector2F32	4 / 6 / 8	Whole studs / compact float / full float
Vector3S16 / Vector3F24 / Vector3F32	6 / 9 / 12	The same ladder in 3D
CFrameF24U8	12	F24 position, one byte per Euler angle
CFrameF32U8 / CFrameF32U16	15 / 18	F32 position, 8-bit or 16-bit angles
Color3	3	8 bits per channel
BrickColor	2	Palette number
UDim / UDim2	4 / 8	Scale as S16 times 1000 (clamped to about 32.767), offset as S16
Rect	16	Four F32
NumberRange	8	Min and max as F32
Region3	24	Min and max corners as F32
NumberSequence	1 + 3 per keypoint	Time, value and envelope at 8 bits each
ColorSequence	1 + 4 per keypoint	Time and RGB at 8 bits each
EnumItem	3	Enum type index (U8) and value (U16). The enum must be listed in Net/Packet/Types/Enums.luau; others raise on write and on read.
Any	1 + payload	Adaptive. Stores nil, numbers, strings, booleans, tables, buffers, vectors, CFrame, Color3, BrickColor, UDim, UDim2, Rect, NumberRange, both sequences, EnumItem and Region3.
Instance	none	Rejected. An Instance has no representation outside a running server.

Coercion

Gameplay math should not have to track wire widths, so writers coerce:

• Integer types convert with tonumber, round (99.6 becomes 100) and clamp into range (300 becomes 255 for NumberU8, -5 becomes 0). A value tonumber cannot convert raises cannot coerce <type> into <TypeName>, because a silent 0 would corrupt data.
• Boolean8 takes truthiness: 1 is true, nil is false.
• String types tostring anything else: 42 becomes "42".

Coercion is identical on the fast engine and the engine path, and the specs assert it.

Inference and the “Auto” codec

Serde.infer(sample) builds a typed spec from plain data. One sample cannot bound a number: Coins = 0 is neither its maximum nor its sign. So every number infers as NumberF64, which holds any integer exactly up to 2^53 and keeps fractions and negatives.

Sample	Inferred
Any number	NumberF64
string / buffer	StringLong / BufferLong
boolean	Boolean8
Vector3 / Vector2 / CFrame	Vector3F32 / Vector2F32 / CFrameF32U16
Color3, BrickColor, UDim, UDim2, Rect, NumberRange, EnumItem	Their own types
Array whose elements infer alike	{ elementSpec }
Table with only string keys	A nested dictionary spec
Empty table, mixed keys, array of unlike elements, anything else	"Any"

Before 0.9.0 an integer sample inferred as NumberU32 (or NumberS32 when negative). A value above 4,294,967,295 was then clamped, a negative one saved as 0 and a fraction was rounded, with no error: 12e9, -25 and 1.5 reloaded as 4294967295, 0 and 2. Serde.infer(sample, { LegacyIntegers = true }) reproduces the old inference, for decoding buffers written by it.

Edit an inferred spec wherever you know real bounds: a Health that caps at 100 fits NumberU8.

DataDriver builds Codec = "Auto" as Serde.schema(Serde.infer(Defaults), { Extras = true }). Typed fields come from your Defaults, and Extras keeps fields you add before the schema catches up. Each Auto save also records its layout marker and the spec it was written with, and a load decodes with that writer’s spec, so adding a field to Defaults never makes older records unreadable. Records saved before 0.9.0 carry no marker; they decode with the LegacyIntegers spec and are rewritten in the F64 layout by their next load. Do not downgrade the framework after that.

Struct codecs

Serde.struct(spec) compiles a dictionary schema into a field-indexed codec, the primitive under Replica deltas:

• Field names sort once and ids 1 to n follow that order. Identical schemas on server and client agree on every id with no negotiation.
• encodeFull writes the whole record in field order: the late-joiner snapshot.
• encodeDelta(changes) writes a U8 count, then a U8 id and the payload per changed field. A one-field NumberU16 delta is 4 bytes. A field outside the schema raises field "Hacked" is not in this struct schema.
• A nested spec works as a field value. A delta that changes it resends that field’s whole payload.

local Struct = Serde.struct({ Wave = "NumberU16", BossUp = "Boolean8", Title = "String" })
local Snapshot = Struct:encodeFull({ Wave = 12, BossUp = true, Title = "Storm" })
local Delta = Struct:encodeDelta({ Wave = 13 }) -- 4 bytes: count, id, U16
local Changes = Struct:decodeDelta(Delta) -- { Wave = 13 }

The fast engine

When every type in a spec supports it, schema and struct codecs compile to a second implementation. Byte widths resolve when the codec compiles, encode allocates one buffer at the exact final size, and writers run with no capacity checks. The techniques come from light by @hardlyardi (MIT); no code was copied.

Two properties hold, and the specs assert both:

1. The output is byte-identical. The fast path and the engine path write the same bytes for the same value, decode each other’s output and coerce the same way. The F16 and F24 writers copy the engine’s bit layout, including its mantissa-carry rounding fix. Only the speed differs.
2. Unsupported specs fall back on their own. A spec that uses anything the fast engine does not cover runs on the engine path, and one uncovered field sends a whole struct there.

Fast engine covers	Falls back to the engine path
Every fixed-width number (U8 to S32, U24/S24, F16/F24/F32/F64), NumberVlq, Boolean8, Color3, every Vector2 and Vector3 type, String, StringLong, Buffer, BufferLong, and optionals, lists and dictionaries of those	Any, CFrames, sequences, EnumItem, BrickColor, UDim, UDim2, Rect, NumberRange, Region3, Characters, the bit-packed types, and any codec with Extras = true

In a same-machine A/B against the engine path, fixed-schema encode went from 463 ns to 285 ns (38% faster) and decode from 514 ns to 367 ns; dynamic schemas run 22 to 27% faster. The benchmarks list 3.5M encodes and 2.7M decodes per second for a 3-field schema, and 3.7M and 3.4M per second for 2-field struct deltas.

Hostile decode

Stored bytes are still input. The decode paths are hardened, and Serde.spec.luau feeds them crafted buffers:

• A huge length prefix (0xFFFFFFFF on a long string or buffer) raises instead of allocating.
• An unknown adaptive type id raises with a clear message instead of misreading the stream.
• Nested adaptive tables past 48 levels raise instead of exhausting the call stack.
• A NumberVlq that runs past 8 continuation bytes raises.
• A String or Buffer over its cap raises on write, because a wrapped U8 length would desync every field after it.
• A list over 65,535 elements raises on encode, on both engines. Before 0.9.0 the U16 length wrapped or clamped, which truncated the list and misaligned every later field.

Design notes

Never yield mid encode or decode

The engine path shares one scratch buffer and cursor with the Types engine. A yield with an encode in flight corrupts whatever resumes into it. Codecs call no user code and never yield, so ordinary use cannot trigger this; it matters only if you add custom readers or writers to the Types engine. Keep them synchronous.

Adaptive bytes cost

Every Any value, every Extras field and every Serde.encode value pays a type byte, extras also write their key strings, and a codec with Extras ends each dictionary with a terminator byte. Adaptive mode is the safety net for shapes you have not typed yet. The schema is where the size goes down.

Without Extras, unknown fields are dropped

Encode a table with a field the schema does not name and the decoded table lacks it. That is the price of typed fields with no overhead. If data grows fields faster than you type them, turn Extras on, or use DataDriver’s Codec = "Auto", which does. DataDriver also refuses a save whose decoded copy would differ from the data, and raises at boot when an explicit codec cannot hold the Defaults.

• Version guards the format. Evolve the data itself with DataDriver SchemaVersion and Migrations.
• A list over 65,535 elements in a DataDriver profile makes the encode raise, and DataDriver refuses that save with Data.SaveRefused(key, reason) while keeping the session lock.
• An empty table infers as "Any". An inferred codec round-trips Bag = {}, but give list fields one sample element, or write the spec by hand, to get a typed list.