Regression runners

The scripts in tools/regressions/ read the current framework modules and selected spec files, replace every Roblox service with a local test double, and run the result in the standalone Luau runtime. They never connect to a game, a DataStore, a payment service or a teleport destination. The decision that defines them: they run the same spec files Studio runs, against the real module source, so CI catches a regression in most of the framework without Studio. Their limit is the doubles. A double models the engine only as far as the selected specs need, and four specs passed here while failing in Studio before 0.9.0. The Studio spec gate and published-place tests stay necessary. All six runners are steps of Check.

Running them

python tools/regressions/core.py
python tools/regressions/network.py
python tools/regressions/gameplay.py
python tools/regressions/persistence.py
python tools/regressions/datasafety.py
python tools/regressions/presentation.py

They need Python 3.10 or newer and Luau 0.737, with luau on PATH or LUAU_BIN set to the executable. Each runner finds the repository root from its own location, so it runs from any directory. It writes one generated Luau program into a temporary directory, streams the program’s output, removes the directory, and exits with the runtime’s exit code. A missing interpreter prints Luau was not found. Set LUAU_BIN or put luau on PATH. and exits 2. core.py and presentation.py accept --root <path>; core.py also understands a game’s vendored layout (Src/ReplicatedStorage/ChloeKernel), so it can check a vendored copy (see Releases and vendoring).

How a runner builds its program

• Each module body is pasted in as local Name = (function() <source> end)(), and its require(...) lines are rewritten to point at modules loaded earlier or at doubles. Where a runner lists exact replacement strings, a string missing from the source fails the build, so a renamed require cannot fall through to nothing.
• The Logger loads first, because framework modules warn through it. Then TestKit/Expect.luau and TestKit/Watch.luau, the same matchers and the same warning hold Studio uses.
• Spec files are pasted the same way and called with the runner’s own T. Some runners run every describe block; others keep an allowlist and skip the rest.
• Harness-only checks follow the specs. They cover engine behavior a Studio spec cannot force, such as an AssetService failure, a virtual clock past a 30 s timeout, or the exact event order of a UIDragDetector.

Holding warnings like TestKit

Every runner starts Watch.start({ Forward = warn }) and gives its T an expectWarning that calls watch:expect. Each it runs between watch:begin() and watch:finish(), so a Logger warning the test did not declare, or a declared one that never came, fails the test exactly as it would in TestKit. Forward passes every held line on to the runner’s own warn double, so harness checks can still count warnings.

The runners have no ScriptContext, so uncaught errors are caught through the doubles. In the network and presentation runners, task.spawn reports an error on the spawned thread instead of raising it into the caller, as the engine does, and the test that left one behind fails. The presentation runner also fails a thread error raised outside every test (in a spec body, a describe body or a leftover thread) as an entry of its own. The persistence and data safety runners drive each test on a coroutine scheduler and fail a test whose body never finishes or whose scheduler stalls.

Doubles follow the engine where a spec’s result depends on it: persistence value objects warn and keep their value on a type they cannot convert, gameplay models workspace collision groups, presentation loads only animations registered in the session, logs the engine’s load error for any other id and models a place with “Allow Mesh & Image APIs” off (see presentation), and the network runner repeats the NetDriver specs with Studio’s capture on.

The runners

Counts are what the source registers at 0.9.1 and the unreleased changes after it. Each runner prints its own totals on its last lines.

core

core.py with _core_doubles.luau: deterministic clocks, a RunService.Heartbeat signal it fires by hand, services, and a Stats double with the seconds properties (HeartbeatTime, PhysicsStepTime). A second Scheduler copy reads a Stats double without them, so the legacy millisecond path stays covered. Serde runs over the real Packet type engine.

It runs 13 spec files in full: Scheduler, Spatial, Pool, Scope, Process, Signal, Bus, HookRegistry, Logger, Preload, Curve, TablePool and Base64 (100 tests; an empty suite fails the run). It adds 41 harness checks: deadline-heap order and cancellation at every heap position, 1,000 future timers staying asleep across 120 frames, processes waiting on engine calls, Process.sleep, yields that must not leak into budget statistics, idle-gap budgeting, fractional priorities, constant-time Signal disconnect, NaN positions in Spatial, Spatial.characters sleeping with no roots, Serde number inference and the 65,535-element list limit, Mounts ordering and late MountType, fail-closed Rules keys, Logger sinks removed mid-emit, ComponentController over the real Mounts, diagnostics publishing total memory, Rng:index, Scope tweens, and the TestKit and Gate behavior behind the spec gate. 141 checks in total. The summary line is CORE REGRESSIONS | n/m passed.

network

network.py defines its doubles inline: Vector3, Instance and remote doubles, a task that reports thread errors, and scheduler doubles. It runs 19 spec files in full (TransportGuard, NetTap, SettingsClient, InventoryClient, Prices, ReplicaClient, Settings, TextFilter, Text, Replica, NetDriver, AntiExploit, Fuzz, BusBridge, NetGovernor, Registry, Prediction, RagdollClient, Consequences): 237 tests. It then runs NetDriver’s 31 tests again with NetTap.Active on and an allowlisted tester connected, because a Studio server boots that way.

Harness checks: 1 check of the ping-scaled prediction ack window on a virtual clock; 4 unreliable ordering checks over the real NetDriver and NetClient (stale and duplicate datagrams drop, sequences wrap, unordered channels keep everything, a missing channel errors after one bounded wait); 26 Packet receive and wire checks over a real server and client Packet loopback (frame parking and replay, varint lengths at the 1-, 2- and 3-byte boundaries, NumberVlq clamping, Instance slot validation, flood report reasons, request slot exhaustion, typed-array limits); 8 render checks driving HazardClient and ProjectileClient; and 1 check on a live-server NetTap that CryptedChloes (7241279999) keeps debug access after code empties NetTap.OwnerUserIds. 308 checks in total.

gameplay

gameplay.py defines flat doubles inline and a small DataModel in _world_doubles.luau (tag membership that follows the tree, ancestry signals, deferred threads, box and ball queries, CanTouch-guarded Touched). Its T.describe skips any block not on an explicit allowlist of 38 blocks drawn from 29 spec files: the combat, inventory, movement, leaderboard, NPC, lifecycle and kit suites, including CraftingKit, ShopKit, DialogueKit, PartyKit, RoundKit, QuestKit, LootKit with Paid tables, Policy, AchievementKit and kit teardown. A world phase runs 5 more blocks from 4 spec files against the DataModel double (InteractionKit, CheckpointKit pads, TeamKit, Zones, ProceduralKit generation failures) and loads the WizardDuel, GunArena and Obby templates.

Harness checks cover world-query eligibility (CanQuery counts only without CanCollide, RespectCanCollide reads CanCollide alone, exclude and include lists cover descendants), the default queries of WeaponKit, Projectiles and the Movement belt probe against those rules, a WeaponKit shot that Rewind rejects leaving the fire-rate window unused, Ragdoll’s engine calls in order, and the server kernel’s close phase. The ProceduralKit lifecycle suite needs real ProceduralModels and runs only in Studio. The summary line is RESULT | n/m passed in isolated source/spec harness.

persistence

persistence.py defines deterministic storage, HTTP and scheduler doubles. Base64, the Serde codecs and the Packet type engine run their real code, so the Auto codec spec runs and nothing is skipped. It runs 15 spec files in full (DataDriver, HttpBackend, Commerce, Pricing, TransactionRecovery, TeleportRetry, MemoryDriver, MessageDriver, Matchmaking, LiveConfig, Milestones, Analytics, Forensics, Leaderstats, SoftShutdown), about 235 tests, plus one harness check: a receipt that keeps waiting for a joining profile past the old 30 s cap on the harness clock.

JSON is a detached-value double whose encoder returns approximate text for realistic size checks. It does not establish compatibility with Roblox’s JSON or DataStore serializers. Timeout paths that rely on task.delay (a SaveBusy wait’s dead-owner re-check, SoftShutdown’s 25 s deadline) do not fire in this runner.

datasafety

datasafety.py reuses the persistence runner’s doubles, loads the real Session, Policy and kits, and runs 49 cases. Each case is a flow that once lost, rolled back or duplicated player data, asserting the safe behavior:

• _data_safety.luau, 19 game flows: declined and policy-gated receipts, items that do not fit, grants made before the profile loads, trades racing kit writes or leaving a participant blocked, timed crafts across a rejoin, an item in use being sold, the shop’s charge order, leave and close ordering, a second Receipts attach, MemoryDriver update retries.
• _data_safety_core.luau, 21 DataDriver cases: Auto codec layout changes in both directions, newer-schema records, removed default entries, codec round trips, fail-closed validation and migration errors, release save ordering, the 4 MB limit, the refused-save lock heartbeat, backup targets for a Memory primary, restoreAt staging, a staged restore that autosaves leave uncommitted and discard() drops, a staged restore whose lock another server took ending through the lost-lock path, a save() after a failed discard() that never commits the staging, a restore that must not bring back default entries the record had removed, hourly versions against a fresher Mirror, clock skew and the lock-loss kick.
• _data_handoff.luau, 9 multi-server cases for lock handoff: several servers share one backend, MessagingService double and MemoryStore double on a virtual clock whose timers fire. Same-server rejoins after a failed or lost leave save, a stale connection handing over, a crashed server’s lock, a server that crashed mid-trade (the trade rolls back and the trader loads on another server within 16 seconds), a stalled but live holder, servers without presence, late requests and a stuck autosave.

The summary line is DATA SAFETY | n passed, m failed.

presentation

presentation.py with _presentation_doubles.luau: a translation-only CFrame (rotations collapse to identity), Instances with ordered children, IsA inheritance and lifecycle signals, axis-aligned raycasts, render-step bindings by priority, EditableMesh and AssetService results, GUI events, and a fake frame clock on which task.wait steps engine frames. It runs 37 selected describe blocks from 17 spec files (AudioKit, AnimKit, Cinema, VfxSuite, Dissolve, Impact, Gait, InputDriver, Highlights, Time, the Time and Chronicle blocks of Foundations, the DeviceBench governor, MaterialKit, HairKit, BonePhysics, HookScope and DebugPanels), about 150 tests, in the client role (191 tests in all with the harness checks below). Every selected block must run at least one test or the runner fails.

The specs run in a place with “Allow Mesh & Image APIs” off, as a new place ships. There, both EditableMesh creation paths log the engine’s “EditableMesh is not accessible” error as a thread error and throw, so a spec that makes any EditableMesh call fails here, just as the same call prints a red error in Studio. A new MeshPart double has no mesh, as on the engine. MaterialKit.spec runs a second time opted in (ChloeKernelMeshApis = true) for its live-engine block alone: each spec entry runs only its own listed describe blocks, so the order tests do not run twice. The harness checks run with the Mesh APIs on.

Its 41 harness checks cover MaterialKit, HairKit and Dissolve agreeing on which MeshParts have a mesh (a part with no mesh opens nothing; a part baked from an earlier editable binds), render-step priorities and unbinding (CameraKit at Camera + 1 under a per-kit name, Impact at Camera + 2), CameraKit taking over a swapped CurrentCamera, AudioKit occlusion and layer starts, server-only Animator creation, EditableMesh failure handling, MaterialKit.supported() and HairKit.supported() probing again after a spent editable budget, HairKit fallbacks and the client character watch, UIController screen binding and gamepad selection, the ImGui drag detector and title-bar collapse (replaying the engine’s real DragStart and DragEnd order), ImGui sections, the HOOKS and NET inspectors keeping their slots, and the F8 SERVER window reading kernel attributes from ReplicatedStorage. The Studio suite runs on the server and skips client-only cases, so this runner is where those run. The DeviceBench benchmark suite does not run here: its measuring loop needs a real clock. The summary line is RESULT | n/m passed in client presentation harness.

To add specs, append module bodies to MODULES and (spec path, replacements, describe names) to SPECS in presentation.py; its docstring lists the steps.

What doubles cannot prove

The runners do not establish real-client replication ordering, network latency, physics accuracy, platform throttling, production HTTP behavior, successful purchase and teleport integration, render-step ordering against the real player camera, real EditableMesh error strings, or Animator replication. Run the Studio suite and multiplayer tests in Studio before shipping changes to those paths, and a published place for DataStores across servers, real purchases and teleports.

The doubles can also be wrong in the permissive direction. Before 0.9.0 the Studio gate stopped on four specs that passed offline, each leaning on something the engine does not do: a WrapLayer parented to a plain Part (the engine accepts only a MeshPart), table doubles read back from OverlapParams.ExcludeInstances (the engine keeps only Instances), NetTap capture assumed off (a Studio server starts with it on), and teleport options built with TeleportOptions:Clone() (which drops the teleport data). The doubles now model each of those rules, so every one of those specs fails offline on the old code too. When a spec passes here and fails in Studio, fix the double as well as the spec.

Status

The case counts above come from reading the runner and spec source at 0.9.1 and the unreleased changes after it. Each runner’s last lines print the exact totals when it runs. Framework changes need a regression test in these runners or a spec they execute, and it must fail on the old code (AGENTS.md).