Spec gate

On every Studio Play, Main.server.luau runs the TestKit suites before the kernel boots, then the security audit and a fuzz pass once the server is up. TestKit/Gate.luau collects all of it into one report. Two decisions define it. A failure stops the server boot script, so the kernel never starts on a broken tree, while the play session keeps running so the Output window stays readable. And when a plugin starts the session as a harness run, the gate ends the session itself with StudioTestService:EndTest(report), so the plugin gets the whole result back as a value instead of reading Output. At 0.9.1 with the Unreleased fixes the gate passes 1,244 tests (435 in the shared suite, 809 in the server suite) with a clean Output window. The regression runners run many of the same specs offline in CI; Fuzz and audit documents the last two steps.

What runs on Play

Everything below sits behind RunService:IsStudio(). A production server runs none of it.

1. Gate.new() asks StudioTestService:GetTestArgs() for the session’s arguments. When they carry Harness = "ChloeKernelSpecs", the gate is in harness mode and arms a deadline of TimeoutSeconds from the arguments (default 120 s).
2. gate:suite("Shared", ReplicatedStorage.ChloeKernel.Tests) and gate:suite("Server", ServerScriptService.ChloeKernelServer.Tests), then gate:check().
3. When ServerScriptService.Game.Tests exists, gate:suite("Game", ...) and another gate:check().
4. The boot runs as the step Boot: ServerKernel.boot(), the Bootstrap module, kernel:start().
5. Booted:enableDiagnostics(1) feeds the debug panels.
6. The step Audit is declared, and on the next deferred resumption kernel:securityAudit({ Silent = true }) runs. Any finding of severity High fails it. The audit runs after every service has registered, because it reads channel configuration. Findings below High neither fail the step nor print; call securityAudit() yourself to see them.
7. Outside Run mode, the step Fuzz is declared. When the first session starts, Fuzz.run(kernel, { Session = session, Silent = true }) drives hostile payloads through every channel’s validators and fails the step if any validator throws instead of rejecting. Handlers never run, because they would write to the tester’s own profile. A passing run prints one line with the channel and case counts. Run mode has no players, so there is no Fuzz step there.
8. gate:seal() says no more steps are coming. The report settles once every declared step has finished.

Main.client.luau runs the client game specs (StarterPlayerScripts.Game.Tests) before the client kernel boots and asserts on the result. It does not use the gate and does not end harness sessions.

When the boot stops

gate:check() sums the failed tests across every suite recorded so far. With any failure it ends a harness session first, then raises <n> spec failure(s) in <suite names>: see the TestKit report above. That error stops Main.server.luau: the kernel never boots, no service starts, and no player data loads against a tree whose specs fail. The Studio session stays open with the TestKit failure lines in Output.

gate:run(name, fn) wraps each step. It runs fn under xpcall, records the step as Passed or Failed with the traceback as its detail, ends a harness session on a failure, and then re-raises, so a manual Play fails as loudly as before. A suite that crashes outside TestKit’s own protection is recorded as one failure named (suite) instead of taking the gate down.

Harness sessions

A plugin, or the Studio command bar, starts a harness session like this:

local HttpService = game:GetService("HttpService")
local StudioTestService = game:GetService("StudioTestService")

local Report = StudioTestService:ExecutePlayModeAsync({ Harness = "ChloeKernelSpecs", TimeoutSeconds = 120 })
print(HttpService:JSONEncode(Report))

ExecutePlayModeAsync yields until the session ends and returns whatever the server passed to EndTest. ExecuteRunModeAsync works the same way without a player, so its report has no Fuzz step. The gate calls EndTest exactly once, at the first of:

• the first failure (a failed suite at check(), or a failed step),
• the moment every declared step has settled after seal(),
• the deadline. Steps still pending become Skipped (“did not start before the deadline”), steps still running become Failed (“still running at the deadline”), and the gate warns that it ended the test with the report so far.

The report is a plain table of strings, numbers and booleans, so EndTest can carry it:

export type Report = {
  Harness: string?, -- "ChloeKernelSpecs" in harness mode
  Passed: boolean,
  Total: number, -- tests across every suite
  Failed: number, -- failed tests plus failed steps
  TimedOut: boolean,
  Suites: { { Name: string, Total: number, Passed: number, Failed: number, DurationSeconds: number, Failures: { { Spec: string, Test: string, Error: string } } } },
  Steps: { { Name: string, State: string, Detail: string? } }, -- State: Pending, Running, Passed, Failed or Skipped
}

Passed is true when nothing failed and the report either sealed or did not time out. A manual Play session is not a harness session: there is no deadline and no EndTest, and the session runs until you stop it.

The plugin

tools/plugin/ChloeKernelSpecs.server.luau is a one-button Studio plugin that runs the harness and prints the report. Paste it into a Script and use Save as Local Plugin, or copy it into the local Plugins folder as ChloeKernelSpecs.lua. It adds a ChloeKernel toolbar with a Run specs button. A click starts ExecutePlayModeAsync with the harness arguments and a 120 s timeout, then prints [ChloeKernelSpecs] followed by the report as one JSON line. It warns instead when the call fails, or when the session ended with no report because it was stopped by hand or the server never booted.

Plugins also load inside the client and server DataModels of a test session. The plugin checks StudioTestService.EditModeActive and acts only in the edit DataModel. It is not part of any Rojo project: it uses plugin-only APIs that directive CK015 rejects in game scripts.

Clean Output

A passing gate prints only summary lines: one [TestKit] PASS line per suite, the kernel’s boot line and the [Fuzz] line. Before 0.9.0 a passing run printed about 90 warning, info and error lines, and a real failure hid among them. Five rules keep it clean:

• TestKit holds each test’s Logger warnings and fails the test on a warning it did not declare with T.expectWarning, on a declared warning that never came, and on an uncaught error thrown through spec code. Output raised between tests fails the module that ran last as (outside tests).
• The hold works through Logger.capture, so it sees only Logger output. Framework modules warn through Logger.scope(<Module>) instead of raw warn, and directive CK016 enforces it.
• Specs never load made-up asset ids. Studio logs an error for every asset it cannot fetch. AnimKit’s real-loader spec plays animations registered with AnimationClipProvider:RegisterAnimationClip instead.
• Specs make no EditableMesh or EditableImage call unless ReplicatedStorage has ChloeKernelMeshApis = true. With “Allow Mesh & Image APIs” (Game Settings > Security) off, Studio prints a red “EditableMesh is not accessible” error for every EditableMesh call, even inside pcall. A place that turns the setting on and sets the attribute also runs the MaterialKit live-engine specs.
• The audit and the fuzzer run with Silent = true and report through the gate.

Cleaning the gate surfaced real bugs: a Leaderstats value the engine cannot show warned on every sweep, and the close-budget warning reported the wrong budget. It also surfaced spec bugs that raised errors on other threads.

Speed

0.9.0 made the gate about 3.4 s faster. Measured in Studio, the shared suite went from 4.46 s to 3.75 s and the server suite from 12.52 s to 10.08 s. Two changes did it:

• TestKit yielded one frame after each of the 104 spec modules, about 1.7 s per run. It now yields through the deferred queue, which costs no frame. It must still yield between modules: Roblox’s execution budget is per resumption, and one unyielded pass over every spec can exhaust it on a loaded host and kill the boot thread. Two deferred passes also let whatever the previous module deferred run before the next one starts.
• The Replica slot specs created 131,000 replicas to reach the 16-bit slot limit, about 1.7 s of CPU. ReplicaService now takes MaxLiveSlots (default and ceiling 65535), so the exhaustion spec fills 16 slots, and the recycle spec starts a slot’s generation just under 65535 to prove the generation still crosses 16 bits.

API

Member	Description
Gate.new(options?: { StudioTestService: any?, TimeoutSeconds: number? }): Gate	Reads the harness arguments and arms the deadline in harness mode. StudioTestService is injectable for tests; TimeoutSeconds overrides the arguments.
gate:suite(name: string, container: Instance): Suite	Runs TestKit.runSpecs(container) and records the result. A crash becomes one failure.
gate:check()	Raises when any recorded suite failed, after ending a harness session.
gate:expect(name: string)	Declares a step the report waits for before it can settle.
gate:run(name: string, fn: () -> any): any	Runs one step, records Passed or Failed, and returns fn’s result or re-raises its error.
gate:seal()	No more steps are coming. The report settles when every declared step has.
gate:report(): Report	A snapshot of the report so far.
gate:finish(): Report	Freezes the report and, in harness mode, calls EndTest with it. Idempotent.
gate.Harness	true when the session was started with the ChloeKernelSpecs harness arguments.
Gate.HarnessName	"ChloeKernelSpecs".

Status

The core regression runner checks the gate against a StudioTestService double: it ends a harness session exactly once with a serializable report when a suite fails, records a crashing suite and still ends, settles after every expected step and otherwise ends at the deadline, and never ends a manual Play. It also checks TestKit’s warning hold, stray output, uncaught-error tracking and deferred settling. The timings and the clean Output were measured in Studio. The regression runners do not replace the gate: four specs that passed offline failed in Studio before 0.9.0 because their doubles allowed things the engine does not (see Regression runners). The presentation runner models a place with the Mesh APIs off, so a spec that would print the EditableMesh error in Studio fails offline as well.