Testing and QA

Recipes for game specs, expected warnings, the one-command check, the Studio spec gate from a plugin, benchmarks, hostile-payload fuzzing, security audits and simulated load.

Confidence in layers, cheapest first: specs check your logic, tools/check.py runs every gate CI runs, the Studio spec gate proves the build boots, benches measure costs, the fuzzer and the audit check your input handling and configuration, and the simulation modes put the kernel under load. The framework’s own Studio gate passes 1,244 tests on v0.9.1 with the Unreleased fixes.

Write tests

Specs are ModuleScripts whose names end in .spec. Game specs live in src/Server/Game/Tests/ (and src/Client/Game/Tests/ for client logic); python tools/scaffold.py service creates one next to each new service. Put game logic that can be pure in plain modules and test those:

--!nonstrict
local Wallet = require(script.Parent.Parent.Services.Economy.Wallet)

return function(T)
  local describe, it, expect = T.describe, T.it, T.expect

  describe("Wallet", function()
    it("starts empty", function()
      expect(Wallet.new():balance()).toBe(0)
    end)

    it("rejects negative deposits", function()
      expect(function()
        Wallet.new():deposit(-5)
      end).toThrow("negative")
    end)
  end)
end

Specs run on every Studio play-test before the kernel boots: the server runs the framework’s Shared and Server suites and then the Game suite, and the client runs its game specs. A passing suite prints one line, [TestKit] PASS: <passed>/<total> tests across <specs> specs in <ms>ms. A failure prints [TestKit] FAIL: <failed>/<total> tests failed (<ms>ms) and one line per failing test with its describe path and error, then stops the boot script; the kernel never starts, and the play session keeps running so Output stays readable. TestKit.runSpecs(container) returns { Total, Passed, Failed, Failures, DurationSeconds } for your own gates.

Each it body runs under xpcall, so a crashing test, a crashing describe body or a spec that fails to require is recorded as a failure instead of stopping the run. The runner yields through the deferred queue between spec modules, so whatever a spec deferred runs before the next one starts. Matchers: toBe, toEqual (deep), toBeTruthy, toBeFalsy, toBeNil, toBeType, toBeCloseTo, toBeGreaterThan, toBeLessThan, toContain, toHaveLength and toThrow(pattern?), each negatable with .never. Specs never load made-up asset ids: Studio logs an error for each one. For the same reason they make no EditableMesh or EditableImage call unless ReplicatedStorage has ChloeKernelMeshApis = true, since with “Allow Mesh & Image APIs” off Studio logs an error for every EditableMesh call, even inside pcall. Check that attribute in your own specs before touching either.

Go deeper: TestKit, Scaffolding.

Expect a warning in a spec

A passing run prints only its summary lines. Every Logger warning or error raised inside a test goes to that test instead of the Output window, and the test must declare each one it provokes:

--!nonstrict
local Pricing = require(script.Parent.Parent.Services.Economy.Pricing)

return function(T)
  local describe, it, expect = T.describe, T.it, T.expect

  describe("Pricing", function()
    it("refuses an unknown item with one warning", function()
      T.expectWarning("[Pricing] unknown item") -- a plain substring of the line, prefix included
      expect(Pricing.priceOf("Nope")).toBeNil()
    end)
  end)
end

The module under test warns through Logger.scope("Pricing"), which is why game code logs through the Logger rather than warn: its specs can then assert on it. A warning nothing expected fails the test, and so does an expected warning that never came. Output raised between tests fails the spec that ran last as (outside tests), and an uncaught error thrown through spec code (a spec’s Bus subscriber, a double on another thread) fails the test that was running. T.expectWarning must be called inside an it body. Underneath, Logger.capture(handler) routes console lines to the runner until its release runs; framework modules must warn through the Logger (directive CK016).

Go deeper: TestKit, Logger.

Run every gate before you hand off

python tools/setup.py          # once per machine: pinned Rojo, StyLua, Selene, luau-lsp, the Luau runtime and type definitions
python tools/check.py          # after every change; the last line must be CHECK PASSED
python tools/check.py --fix    # formats with StyLua first
python tools/check.py --quick  # inner loop: tools, format, lint, directives, catalog, types, core regressions
python tools/check.py --only types,lint

tools/check.py runs exactly what CI runs, in order: tool presence, StyLua, Selene, the directive linter (tools/directives.py), catalog freshness, both Rojo builds, strict luau-lsp type analysis against the baseline, and the core, network, gameplay, persistence, data safety and presentation regression runners, then the vendor and tooling tests. Each step prints PASS name or FAIL name with its time and the tail of its output, steps keep running after a failure unless a tool they need is missing, and the last line is CHECK PASSED (n steps) or CHECK FAILED: with the failed step names; the exit status is 1 when anything failed. Fix every failure and rerun until it passes. Never weaken a check, delete a test or add a ck-allow suppression without a reason a reviewer would accept.

The regression runners execute framework specs and regression cases in an isolated Luau runtime against engine doubles. They catch logic regressions offline; the doubles do not replace a Studio play-test or a published place.

Go deeper: check.py, Directives, Regression runners.

Run the spec gate from a plugin

Pressing Play runs the gate by hand. Tooling can start the same session and get the result back as data:

-- From a plugin or the Studio command bar
local Report = game:GetService("StudioTestService"):ExecutePlayModeAsync({
  Harness = "ChloeKernelSpecs",
  TimeoutSeconds = 120,
})
print(Report.Passed, Report.Failed, Report.TimedOut)

TestKit/Gate.luau collects the spec suites, the boot, the security audit and a one-session fuzz run into one serializable report: { Harness, Passed, Total, Failed, TimedOut, Suites, Steps }, where each suite has Name, Total, Passed, Failed, DurationSeconds and Failures, and each step (Boot, Audit, Fuzz) has a Name, a State and an optional Detail. When the harness arguments are present, the gate ends the session exactly once with StudioTestService:EndTest(report): on the first failure, once every expected step has settled, or at the deadline (120 seconds by default), so the caller always gets a report. ExecuteRunModeAsync works too and skips the fuzz step, because no player joins. A manual Play session never ends by itself. tools/plugin/ChloeKernelSpecs.server.luau is a one-button plugin that runs the harness and prints the report as one JSON line; it is not part of any Rojo project, because the plugin-only APIs it uses cannot run in games.

Go deeper: Studio spec gate.

Benchmark your own systems

Drop a <Thing>.bench ModuleScript into ChloeKernel/Benches:

-- Inventory.bench
local Inventory = require(game:GetService("ReplicatedStorage").Game.Inventory)

return function(bench)
  local Items = Inventory.new()

  bench("Inventory:add", function(index)
    Items:add(index % 50, 1) -- vary inputs so one warm branch is not all you measure
  end)
end

Benches never run at boot. Run them from the command bar during a play-test:

local Bench = require(game.ReplicatedStorage.ChloeKernel.TestKit.Bench)
Bench.runAll(game.ReplicatedStorage.ChloeKernel.Benches)

Bench.run executes a 1,000-call warmup batch first, so cold caches stay out of the numbers, then times 1,000-call batches until its deadline (0.25 seconds). It prints operations per second and p50 and p99 per-operation latency over the batch averages; multiply p50 by your per-frame call count to get a frame cost. runAll yields a frame between benches so they do not contend.

Go deeper: Bench, Benchmarks.

Fuzz every channel with hostile payloads

The audit (next recipe) reads configuration; the fuzzer drives input. Fuzz.run builds hostile arguments per schema type (empty and 10 KB strings, format specifiers, control bytes, 0, -1, type maxima, NaN, infinities, NaN vector components, and nil and nested tables for Any) and runs them through every intent and request validator chain:

local Report = Fuzz.run(kernel, {
  Session = session, -- a real session: fabricated ones make validators fail in ways no client could cause
  CasesPerChannel = 64,
  Channels = { "UseItem", "SetSign" }, -- optional filter
})
assert(#Report.ValidatorErrors == 0, `[Fuzz] {#Report.ValidatorErrors} validator error(s)`)

The stock Main.server.luau already runs this gate on the first session of every Studio play-test, validators only, and fails the gate when any validator throws instead of rejecting. The report carries Channels, Cases, Passed, Rejected, HandlersExecuted, ValidationOnly, ValidatorErrors and HandlerErrors. Handlers run only with ExecuteHandlers = true, and only on an isolated test kernel and session: handlers write to the session’s profile and world state, and the fuzzer is not a sandbox (validators can have side effects too). A validator that throws counts as a rejection, so the fail-closed gate holds, but the throw lands in ValidatorErrors because a validator must reject hostile input, not crash on it. CasesPerChannel defaults to 32, cases are deterministic per Seed (default 1), and IncludeRequests = false narrows the run to intents.

The fuzzer calls the chains directly, past the wire’s typed serialization, so it sends values real clients cannot encode (defense in depth) and a clean report says nothing about transport safety. CI separately drives the real Packet receive loop with hostile frames.

Go deeper: Fuzz and audit.

Audit the security surface in one call

local Findings = Kernel:securityAudit({ Silent = true })
for _, Finding in Findings do
  assert(Finding.Severity ~= "High", `security audit: {Finding.Kind} {Finding.Name}: {Finding.Detail}`)
end

Without Silent, the audit prints security audit: clean, or one warning listing every finding, most severe first, one line each in the form [High] UnguardedChannel (intent Grant): has no validator .... The stock boot runs this exact loop after every service has registered its channels, and a High finding fails the Studio gate.

Finding	Severity	Meaning
`UnguardedChannel`	High	An intent or request with no validator and no `Open`. It rejects every payload: fail-closed, but almost certainly not what you meant.
`FailOpenGate`	High	An `Intent.` or `Request.` hook point flipped to fail-open, so a crashing validator would pass the payload.
`OpenChannel`	Medium	A channel marked `Open` with a handler: any payload reaches the handler, so confirm each one survives hostile input.
`DefaultRateLimit`	Info	A channel with a handler on the default 30 per second; set a `RateLimit` that matches the gameplay.

The audit reads configuration only. Pair it with the fuzzer, which exercises the same channels. Net:auditValidators() is the narrow sweep for unguarded channels, and Kernel:start() warns about each one.

Go deeper: Fuzz and audit, Hooks.

Stress-test the kernel with simulation modes

The load harness drives real kernel code paths at game-shaped intensities, so the debug panel and hot-task profiler show where things strain before your game does. Set one attribute and press Play:

-- Command bar, or the Properties panel on ReplicatedStorage
game.ReplicatedStorage:SetAttribute("ChloeKernelSimulation", "HighIntensity")

Mode	What it is
`HighIntensity`	A busy shooter-scale server: 200 entity processes, 30 projectile arcs per second, 500 Bus messages and 300 hook chains per second, replicas with 30 viewers, and a deliberate 20 ms per second of CPU burn. A modern server clears it within its budget.
`SmallGame`	The same rigs at about a tenth of the intensity, as a baseline.
`Overload`	Deliberate torture: more scheduled work than a frame holds, so the budget pegs, `Deferred` goes above zero and the panel shows its red states. It models nothing real.
`Showcase`	A narrated tour that cycles six acts (processes, effects, projectiles, replicas, trade replay rejection, Serde sizes) into the panel log.

Change the attribute mid-game to swap modes: the old mode tears down completely (processes killed, recurring tasks cancelled, parts destroyed) before the new one starts, and an empty string stops. Open the panel with F8 while it runs; hot tasks name the simulation’s own origins, and a 10-second heartbeat logs achieved rates and GC cost so you can confirm the load is real. The SIM CONTROL window switches server modes from the client and runs client-side loads independently, so you can stress both sides at once.

Every rig runs the real subsystem (scheduler, processes, hook chains, Bus fan-out, replica interest and deltas, pools); only the wire is faked, with fake packet factories, so a connected client’s real Packet stream cannot be corrupted. The harness is Studio-only unless ChloeKernelDebug is set, and the SIM CONTROL intent honors the debug allowlist, so turning the flag on for a live server does not hand load controls to every player.

Go deeper: Simulation, Debug panels.