Debug panels

Panel.attach(kernel) builds an immediate-mode overlay of separate windows: CLIENT and SERVER health, NET (a live wire inspector for kernel channels on both machines) and HOOKS (every hook point with its fire and rejection counts). The framework test place adds SIM CONTROL for load testing. Two decisions define it. First, it exists only in Studio unless a place opts in with an attribute and a per-user allowlist, because the NET window and the server log mirror carry every player’s decoded traffic and the full server log. Second, it is inert while hidden: nothing renders, samples or captures until F8 opens it. Chronicle is the queryable timeline counterpart to the HOOKS window, Simulation drives the loads SIM CONTROL starts, and the numbers come from the Scheduler, Logger and GcWatch.

Opening the panel

F8 toggles the windows (F9 is the engine’s own Developer Console). Every panel ScreenGui starts with Enabled = false, so a Play session does not open with an overlay over the game, and the first F8 press shows them. In Studio there is nothing to set up. Main.client.luau attaches the panel after the client kernel starts:

local Panel = require(ReplicatedStorage.ChloeKernel.Debug.Panel)
Panel.attach(Booted) -- Studio-only unless ChloeKernelDebug is true

The SERVER window needs the server to publish diagnostics. Main.server.luau does it in Studio and on any place with ChloeKernelDebug set:

if RunService:IsStudio() or ReplicatedStorage:GetAttribute("ChloeKernelDebug") == true then
  Booted:enableDiagnostics(1)
end

Without it the SERVER window says there are no server diagnostics and names the call to make. Panel.attach returns false when the player is not allowed, and otherwise a handle whose detach() removes every window, listener and task.

Each window drags by its title bar through a UIDragDetector, which works with mouse, touch and gamepad and needs no UserInputService connection. A press on the title bar that ends within 4 pixels of where it started collapses or expands the window; a longer press is a drag. The detector takes mouse and touch presses, so the title button’s own Activated never fires for them. Before 0.9.0 that meant a click never collapsed anything. Activated still serves input that bypasses the detector, and a press that reaches both toggles once.

What runs while hidden

Everything with a per-frame or per-tick cost is armed when the ScreenGui becomes enabled and disarmed when it is disabled:

Piece	While hidden	While shown
CLIENT and SERVER render (Scheduler:every(0.1), Priority.Low)	Cancelled	Runs
Frame sampling (RunService.Heartbeat connection)	Disconnected	Connected
GC sampling (GcWatch.attach)	Destroyed	Attached
Client log tail (Logger.addSink)	Removed	Added. The tail shows lines logged while the panel is open.
Server log mirror (CKDBG_LogTail)	Packets dropped without decoding	Decoded into the SERVER tail
NET capture on this client (NetTap.Active)	false	true
NET render (every(0.2)) and server wire mirror (CKDBG_NetTap)	Cancelled; packets dropped without decoding	Runs; packets ingested
HOOKS render (every(0.25))	Cancelled	Runs
HOOKS recording	Still counts every fire	Counts
SIM CONTROL render (every(0.15))	Cancelled	Runs

HOOKS keeps counting while hidden on purpose: dead gates and rejection totals since attach are the reason the window exists, and opening F8 should show them. The recording costs two os.clock() reads per fire and a few counter updates, with no table allocated per fire. What remains while everything is closed is the F8 listener, the Enabled change listeners and that HOOKS bookkeeping.

On the server the matching streams do nothing without a reader. The log mirror arms one flush when a line is logged and skips encoding when no allowlisted debugger is connected; the wire capture runs only while one is connected. See Production access control.

Mental model

Module	Role
Debug/ImGui.luau	The widget layer: immediate mode over retained Instances. See Building your own panels.
Debug/Panel.luau	Owns the ChloeKernelDebug ScreenGui (DisplayOrder 1000) and the F8 toggle, renders CLIENT and SERVER, reads and resets the Scheduler’s task profiler once a second, and attaches NET and HOOKS into the same ScreenGui.
Debug/NetTap.luau	A per-VM recorder of kernel channel messages: a 64-entry ring, running counters, byte estimates, hex previews and the access gate NetTap.debugAllowed.
Debug/NetPanel.luau	The NET window.
Debug/HookScope.luau	The HOOKS window. Decorates kernel.Hooks.fire.
Debug/SimPanel.luau, Debug/SimLoad.luau	SIM CONTROL and the client load rigs.

Server numbers reach the client two ways. Numeric health data rides attributes: enableDiagnostics writes short names (StepMs, BudgetMs, LuaHeapMb, NetAccepted, …) onto a Configuration named ChloeKernelDiagServer under ReplicatedStorage, while the kernel’s own ChloeKernelServer, ChloeKernelClient and ChloeKernelSessions attributes stay on ReplicatedStorage itself. Anyone can read those, and they hold nothing sensitive. The sensitive streams, the server log tail and the wire capture, travel as per-recipient packets to allowlisted debuggers and are never broadcast.

Note

Diagnostics moved onto the Configuration child in 0.6.2. About 29 long-prefixed attributes directly on ReplicatedStorage exceeded Roblox’s 1,024-byte cap on one instance’s total attribute payload, which errored the diagnostics task in real games. String attribute values cap at 50 characters, so the hot-task list rides three attributes (HotTasks1 to HotTasks3). If you read diagnostics directly, read them from ChloeKernelDiagServer. Until 0.9.0 the SERVER window also looked for the session count and the kernel versions on that child, so Sessions always showed a dash and the version-skew row never appeared.

The CLIENT window

Reads this machine directly. Every graded value is green, orange or red against thresholds chosen for that value, and most rows have a tooltip that says what the number means and where to look when it is wrong.

Section	Rows
Frame & network	Kernel version, uptime, FPS with frame time, ping, replication kbps in and out, physics solver rate.
Memory	Total (operating-system figure; in Studio it includes the editor), Lua heap, Instances, Script memory (only when category tracking is on), GC alloc in KB/s, GC cycles per minute with reclaim size.
Frame	A frame-compute bar (heartbeat phase plus physics against the frame target), then who owns it: engine and other scripts (heartbeat minus the kernel), physics, render CPU (when Stats.RenderCPUFrameTime exists) and the kernel share.
Scheduler	The live frame-budget bar, a step-time sparkline, tasks per second, Deferred, queue depths per priority (K/H/N/L/B), recurring tasks, live processes, and the five hottest task origins in ms/s.
Kernel internals	Booted services, distinct Bus topics (plus wildcards), hook points and their handler counts.
Log tail	The last 6 Logger lines captured while open. Hover a row for the full line. A dump button prints a one-line stats snapshot to the Output window, where text can be copied.

Grades for a few of the rows:

Row	Green	Orange	Red
FPS	55 or more	30 to 55	under 30
Ping	up to 100 ms	up to 200 ms	over 200 ms
Lua heap	up to 100 MB	up to 250 MB	over 250 MB
Instances	up to 350,000	up to 700,000	over 700,000
GC alloc	up to 1,024 KB/s	up to 4,096 KB/s	over 4,096 KB/s
Frame budget use	up to 60%	up to 90%	over 90%
Deferred	0	1 to 5	over 5

Hot tasks name the exact script:line that scheduled the work, so “something is lagging” becomes “Systems.Combat:131 is spending 4.2 ms/s”. The panel’s own render task appears in that list tagged (this panel) and is not graded, since it runs only while the panel is open.

Heap climbing with reclaims near zero is a leak; a stable heap with busy GC cycles is allocation churn. When Stats.MemoryTrackingEnabled is false, the category getters return 0 and warn on every call, so the panel hides the Script memory row and says tracking is off. Total, Lua heap, Instances and the GC rows do not depend on it.

The SERVER window

Mirrors the server through the diagnostics attributes, so it works in a real multi-machine session:

Section	Rows
Status	A VERSION SKEW row, only when the server and client kernels report different versions (a server mid-deploy or a stale client). Uptime since diagnostics started, sessions, services.
Memory	Server Lua heap, a Total bar against Roblox’s 6,400 MB server cap (the server crashes at the cap, taking every player with it), Instances, GC alloc and cycles.
Engine	Replication in and out (all players summed), physics replication, primitives and how many are moving, contacts. The physics replication counters read 0 once Improved Physics Replication is active.
Frame	The same decomposition as the client: compute against target, engine and other scripts, physics, kernel share.
Scheduler	Frame budget against the live budget, step-time sparkline (averaged over each diagnostics window), step max, tasks per second, and Deferred summed since the last diagnostics tick.
Hot tasks	The heaviest server origins in the last window, reassembled from the three attributes.
Log tail	Server Logger lines from the CKDBG_LogTail mirror.
Net	Intents accepted and rejected, rate-limited sends, transport floods with their byte total.

Deferred is summed because deferral is bursty and a point sample reads zero during real overload. Three diagnostics ticks in a row with deferred work publish Kernel.Overload on the Bus, and it keeps publishing each tick until a clean window. Replication CPU has no public timing stat, so when frame compute reads green and the server still lags, the replication row is the suspect.

The NET window

A live inspector for kernel channel traffic from both machines. Server entries arrive through the debug-only CKDBG_NetTap mirror. Rejected, rate-limited and dropped messages are rows of their own next to delivered ones.

Totals at the top come from counter deltas over 1-second windows:

Counter	Meaning
In / Out	Kernel channel messages per second, both machines combined.
Wire	Payload KB/s from each channel’s schema widths: exact for fixed-width wire types and buffers, measured with Serde.encode for Any. An Instance argument counts 0 bytes because it rides the remote’s own Instance argument, not the kernel buffer. Per-batch transport framing is not attributed.
Game total	This machine’s whole network traffic from the Stats service: engine replication, physics and every remote. Kernel Wire moving with it means your channels; Game total moving alone means something else.
Rejected	Validation returned false, the sender had no session, or the channel was Unguarded (no validator, so it rejects every payload).
Rate limited	Dropped by the channel’s token bucket before validation ran. Honest players hitting this means RateLimit is below real gameplay. See Rate limiting.
Dropped	Transport losses: Packet’s inbound flood drops (8,000 bytes per heartbeat per player by default, the RateLimitBytes attribute), recorded as one Flood row per player and reason per heartbeat window, and unreliable sends whose payload plus the 2-byte sequence footer passes 980 bytes.

A Shed row counts toward none of the three failure totals. It marks an unreliable send the NetGovernor skipped for a player on a poor connection.

The tail lists the 12 newest entries. Each row reads: an arrow for logical flow (up is client to server, down is server to client, the same on either machine), the channel, snt or rcv for which half of the wire recorded it, the decoded arguments, payload bytes, processing time and any verdict other than OK. A delivered message appears twice, as the sender’s snt row and the receiver’s rcv row. A snt with no rcv never arrived, or, for simulation replicas, was never on the real wire.

Pause capture freezes the tail (messages still flow; only recording stops). Clear resets the ring and counters, which discards boot-time spec traffic. A checkbox hides replica traffic; the window then over-fetches so the tail still fills.

Clicking a row pins it in the Inspector:

• channel kind and name, which machine recorded it and how long ago, the verdict,
• size and time: server times are gate plus validators plus handler, client request times are round trips,
• the wire types per argument, with Any expanded to the encoding it picked (Any:U8, Any:F32, Any:Str+U8len),
• the decoded values before serialization,
• the serialized payload as hex (first 32 bytes, then …+NB for the rest), plus the one channel-id byte that rides with every message.

Tests/NetTap.spec.luau pins the byte estimates, hex previews and Any descriptions to exact strings. NetTap.encodePreview({ "NumberU16" }, 258) is "02 01": the wire is little-endian.

When capture runs

NetTap.shouldCapture(channel) is NetTap.Active and not NetTap.Paused and the channel not muted. Kernel code checks it before building previews, timestamps or argument strings, so closed capture costs one check per message. NetTap.record itself keeps recording when called directly, whatever Active says.

• On a client, NetTap.Active starts false. NetPanel sets it true while its ScreenGui is enabled and back to false when hidden or detached. Code that relied on it being true at boot must open the NET window or set NetTap.Active itself.
• On the server, NetTap.Active starts true in Studio or when ChloeKernelDebug is set. Once the NetDriver exists it recomputes Active on every join and leave: true only while at least one connected player passes NetTap.debugAllowed. A Studio server has the tester connected, so capture is on there.
• The server mirror is dirty-driven. A NetDriver entry arms one flush a second later at Priority.Background. The flush lists the allowlisted receivers and returns without encoding when there are none, otherwise drains up to 40 new entries, JSON-encodes them once and fires them to each receiver. A full batch arms the next flush. ReplicaService calls NetDriver:markTapDirty() after recording its own entries.

The HOOKS window

Debug/HookScope.luau shows every hook point with its fire and rejection counts, chain cost, fail stance and handler count. On attach it saves the live kernel.Hooks.fire and installs a wrapper that times the original call with os.clock() before and after and books the result. A rejection is a chain whose first return value is false. Chain cost is the wall time of the whole validator chain per fire, as the caller feels it, shown as average and maximum.

The list is the union of two sources: every point that has fired, and every point Hooks:listPoints() returns ({ Name, FailOpen, Handlers }). A point that is defined but never fired shows dimmed as never fired. That is a signal worth checking: validation code nothing calls anymore, or a path nobody has exercised this session. Rows sort by most recent fire and the first 14 show.

Clicking a row pins the inspector: fail stance (fail-open or fail-closed) with the handler count, or fired, never defined (default fail-closed) for a point created by its first Hooks:on; fires and rejections; chain cost (orange when the maximum passes 1 ms); time since the last fire and the last rejection; and a fires-per-second sparkline (one sample per second while the window is open, last 40 samples). Pause capture stops the bookkeeping while the real chain still runs, and Reset counters clears every point and the pin.

HookScope and Chronicle decorate Hooks.fire independently. detach() restores the saved function only while its own wrapper is still installed; otherwise the wrapper stays in the chain as a pass-through, so detaching the two in any order leaves a working registry. Tests/HookScope.spec.luau covers both orders.

The SIM CONTROL window

Debug/SimPanel.luau is a remote control for load tests with the same gate and the same F8 key, in its own ChloeKernelSimControl ScreenGui. The framework test place attaches it from src/Client/Bootstrap.luau; delete those two lines in a real game.

• Server load: buttons for HighIntensity, Overload, SmallGame and Showcase. Each fires the SimControl kernel intent (RateLimit 5, validated with NetTap.debugAllowed), so the command runs the full validation pipeline and shows in the NET window. The running mode reads back from the ChloeKernelSimulation attribute, and Stop clears it.
• Client load on this machine: Debug/SimLoad.luau rigs. HighIntensity runs 150 entity processes, 20 pooled bolts/s, 400 Bus messages/s, 240 hook fires/s, 80 Serde round trips/s, a deliberate 2 ms burn per tick and 60 real SimNoise intents/s, above that channel’s 50/s limit so RateLimited verdicts appear in the NET window. SmallGame is about a tenth of that with no burn.

SimPanel.attach returns false when the player is not allowed and otherwise a handle with detach(), which stops a running client load and destroys the window. Before 0.9.0 it returned true; the handle is still truthy for callers that test it. Start a load, then watch CLIENT and SERVER react: hot tasks name the simulation’s origins, the budget bar climbs, Deferred grades.

Building your own panels with ImGui

Debug/ImGui.luau is a small immediate-mode layer for debug windows. Describe the whole window on every render call; the layer reuses Instances and writes only properties that changed. This window follows the kernel panels’ rules: it attaches only for allowed players, starts hidden, renders only while shown, and everything it creates belongs to a scope:

src/Client/Game/Controllers/CombatController/DebugWindow.luau

--!strict
-- DebugWindow: an F8 ImGui window for combat counters, gated like the kernel panels.
local Players = game:GetService("Players")
local ReplicatedStorage = game:GetService("ReplicatedStorage")
local UserInputService = game:GetService("UserInputService")

local ImGui = require(ReplicatedStorage.ChloeKernel.Debug.ImGui)
local NetTap = require(ReplicatedStorage.ChloeKernel.Debug.NetTap)

export type Counters = { Shots: number, Hits: { [string]: number } }

return function(kernel: any, counters: Counters): (() -> ())?
  if not NetTap.debugAllowed(Players.LocalPlayer) then
    return nil
  end
  local Scope = kernel:createScope()

  local Gui = Instance.new("ScreenGui")
  Gui.Name = "CombatDebug"
  Gui.ResetOnSpawn = false
  Gui.Enabled = false -- hidden until F8, like the kernel panels
  Gui.Parent = Players.LocalPlayer:WaitForChild("PlayerGui")
  Scope:add(Gui)

  local Window = ImGui.window("Combat", {
    Size = UDim2.fromOffset(300, 260),
    Position = UDim2.fromOffset(12, 560),
    Parent = Gui,
  })
  local History: { number } = {}
  local LastShots = counters.Shots
  local ShowEmpty = false

  local function render()
    table.insert(History, counters.Shots - LastShots)
    LastShots = counters.Shots
    if #History > 40 then
      table.remove(History, 1)
    end
    Window:render(function(ui: ImGui.Ui)
      ui:header("Weapons")
      ui:value("Shots fired", counters.Shots)
      ui:sparkline("shots per tick", History, nil, "Shots between renders, newest on the right.")
      ShowEmpty = ui:checkbox("show targets with no hits", ShowEmpty)
      -- The hit list changes length; inside a section the rows after it keep their slots
      ui:section("hits", function(ui: ImGui.Ui)
        for Target, Hits in counters.Hits do
          if Hits > 0 or ShowEmpty then
            ui:value(Target, Hits)
          end
        end
      end)
      ui:separator()
      if ui:button("Reset counters") then
        counters.Shots = 0
        LastShots = 0
        table.clear(counters.Hits)
      end
    end)
  end

  local RenderHandle: any = nil
  local function setVisible(visible: boolean)
    if visible and not RenderHandle then
      RenderHandle = kernel.Scheduler:every(0.1, render, kernel.Priority.Low)
    elseif not visible and RenderHandle then
      RenderHandle:cancel()
      RenderHandle = nil
    end
  end
  Scope:add(Gui:GetPropertyChangedSignal("Enabled"):Connect(function()
    setVisible(Gui.Enabled)
  end))
  Scope:add(UserInputService.InputBegan:Connect(function(input, processed)
    if not processed and input.KeyCode == Enum.KeyCode.F8 then
      Gui.Enabled = not Gui.Enabled
    end
  end))
  Scope:add(function()
    setVisible(false)
    Window:destroy()
  end)

  return function()
    Scope:destroy()
  end
end

How it stays cheap: each widget call claims a slot keyed by its call position. The same call order and the same widget type reuse the same Instance, and every widget compares against cached state before it writes a property. Slots past the last call of a pass are destroyed, so trailing conditional rows disappear. ui:section(key, build) holds its own slot list inside one slot of the parent, so a list that grows or shrinks shifts nothing after it; an empty section hides. The NET tail and the HOOKS point list both sit in sections so their inspectors keep their slots.

Buttons, checkboxes and selectables report a click on the next render pass, the usual immediate-mode handshake. A slot whose logical identity changes drops its pending click: for buttons and checkboxes the identity is the text, for selectable it is the optional id or else the text. Rows whose labels carry live counters pass a stable id (HOOKS passes the point name), so a relabel between the click and the next pass does not eat the click.

ImGui API

Member	Description
ImGui.window(title: string, options?: { Size: UDim2?, Position: UDim2?, Parent: Instance? })	Creates a draggable, collapsible window. Defaults: 380 by 440 at (16, 16). Drags through a UIDragDetector on the title bar.
window:render(build: (ui: Ui) -> ())	Runs one pass. Skipped entirely while the window is collapsed.
window:destroy()	Destroys the window’s Instances and forgets its slots.
ImGui.Theme	The shared palette (Good, Warn, Bad, Accent, Text, TextDim, …) and the Code font, for consistent health colors.

The ui object inside render:

Widget	Description
ui:header(text)	Section header row.
ui:label(text, color?, hint?)	One line of text, truncated at the end.
ui:value(label, value, color?, hint?)	Key and value on one row. The workhorse.
ui:separator()	Horizontal rule.
ui:button(text): boolean	true on the pass after a click.
ui:checkbox(text, checked): boolean	Returns the new value. Assign it back to your state.
ui:selectable(text, selected, color?, hint?, id?): boolean	List row with a selection highlight. true on the pass after a click. id keeps a relabeled row’s click.
ui:bar(label, fraction, text?, color?, hint?)	Progress bar with overlay text. fraction clamps to 0 to 1.
ui:sparkline(label, values, max?, hint?)	40-bar history of the last 40 values. Scales to the peak when max is nil. Bars turn orange past 60% and red past 85% of the scale.
ui:section(key, build)	A nested slot list for rows whose count changes. Hidden while empty.

A hint renders as a tooltip beside the window’s right edge. The built-in panels put their full diagnosis text there.

Production access control

Who can open any of this is decided in one function, NetTap.debugAllowed(player), and it fails closed:

1. In Studio: always allowed.
2. Live, with the ChloeKernelDebug attribute on ReplicatedStorage not true: denied. The panels do not attach, the mirrors never send, and client capture stays off.
3. Live with the flag on: CryptedChloes (7241279999) and the operators in NetTap.OwnerUserIds are always allowed. Everyone else is checked against the ChloeKernelDebugUserIds attribute: a comma-separated UserId list, or "*", "all" or "ALL" to open a test place to every joiner.
4. Flag on but the allowlist unset or empty: denied for everyone except owners and Studio, with one warning through the Logger that says so. Failing open here would hand every player’s decoded traffic and the full server log to anyone.

The sensitive mirrors check the same gate per recipient: CKDBG_NetTap and CKDBG_LogTail go to allowlisted debuggers one by one and are never broadcast. The SimControl and SimNoise intents validate their sender with the same function, so flagging a live server does not hand load controls to every player. The server log mirror keeps at most 32 pending lines and arms one flush 0.5 s after a line arrives; with no allowlisted receiver the flush clears the lines without encoding them.

NetTap.OwnerUserIds always includes CryptedChloes (7241279999), the framework’s only developer. debugAllowed also checks that id itself, so no edit to the table can drop it, and directive CK017 rejects code in any file that removes the entry, replaces the table or clears it. The table exists because a place edit can clear the ChloeKernelDebugUserIds attribute. A game adds further operators from shared bootstrap code, so the server (which sends the mirrors) and the clients (which attach the panels) agree:

local ReplicatedStorage = game:GetService("ReplicatedStorage")
local NetTap = require(ReplicatedStorage.ChloeKernel.Debug.NetTap)

NetTap.OwnerUserIds[123456789] = true -- a second operator, next to CryptedChloes

Member	Description
NetTap.debugAllowed(player: Player?): boolean	The gate above. Checks CryptedChloes (7241279999) directly, then NetTap.OwnerUserIds. Panel, NetPanel and SimPanel call it on attach, the server mirrors per recipient, and the SimControl and SimNoise validators per message.
NetTap.OwnerUserIds: { [number]: boolean }	Operators allowed whenever ChloeKernelDebug is true, whatever ChloeKernelDebugUserIds says. Starts as { [7241279999] = true }. Add ids with NetTap.OwnerUserIds[userId] = true; never replace or clear the table (CK017). Added in 0.9.1.

Attribute (ReplicatedStorage)	Meaning
ChloeKernelDebug: boolean	Turns the debug surface on outside Studio. Without it nothing attaches.
ChloeKernelDebugUserIds: string	Comma-separated UserIds, or "*", "all", "ALL". Missing or empty means owners and Studio only.
ChloeKernelSimulation: string	The current server simulation mode; empty means stopped. Set directly or through SIM CONTROL.

Gotchas

Caution

Owner ids pass the gate whenever ChloeKernelDebug is true, whatever the allowlist says, so every id in NetTap.OwnerUserIds sees every player’s decoded traffic and the server log on a flagged place. Add an operator on both sides: an id added only on the client opens windows that the server’s mirrors never feed, and one added only on the server gets mirror packets with no panel to show them.

Caution

ImGui identity is call position. Inserting a widget conditionally in the middle of a pass shifts every slot after it: those Instances are rebuilt and any pending click below the insertion drops. Put variable rows in a ui:section, keep conditional rows at the end, or render a placeholder in the other branch.

Note

The panel excludes itself from blame: its render task is listed in hot tasks as (this panel) and never graded. It runs only while the overlay is open.