Infrastructure and ops

The layer under the gameplay: per-player lifetimes that clean up after themselves, lifetimes for everything else, work scheduled under a frame budget, processes you can pause or kill, parallel dispatch, structured logs, deduplicated errors, fleet-wide config, and the instruments to watch it all on a live server. Server examples are services in src/Server/Game/Services/ (see Services and controllers).

Run code when a player joins or leaves

src/Server/Game/Services/PresenceService.luau

--!strict
-- PresenceService: per-visit state, join context and leave reasons.

local ReplicatedStorage = game:GetService("ReplicatedStorage")

local Logger = require(ReplicatedStorage.ChloeKernel.Logger)

local Log = Logger.scope("PresenceService")

local PresenceService = {
  Name = "PresenceService",
}

function PresenceService.init(self: any, kernel: any)
  self.Kernel = kernel
  self.Scope = kernel:createScope()
end

function PresenceService.start(self: any)
  self.Scope:add(self.Kernel:onSession(function(session)
    session.Data.JoinedAt = os.clock()

    -- JoinData is read once at join and verified by Roblox
    local Referrer = session.JoinData.ReferredByPlayerId
    if Referrer and Referrer ~= 0 then
      self.Kernel.Bus:publish("Referral.Joined", session, Referrer)
    end

    session:bind(session.Player.CharacterAdded:Connect(function(character)
      self.Kernel.Bus:publish("Presence.Spawned", session, character)
    end))
    session:bind(function()
      Log.info("played", session.Player.Name, os.clock() - session.Data.JoinedAt)
    end)
  end))

  self.Scope:add(self.Kernel.Bus:subscribe("Kernel.SessionEnd", function(_topic, session, exitReason)
    Log.info("left", session.Player.Name, exitReason)
  end))
end

function PresenceService.stop(self: any)
  self.Scope:destroy()
end

return PresenceService

onSession also runs for players already in the game, so a system that registers late misses nobody. Everything passed to session:bind (connections, Instances, functions, tables with a disconnect or destroy method) is cleaned up when the player leaves, in the order it was bound, before the DataDriver’s final save. That one habit is most of leak prevention: per-player state lives on the session, and the session’s teardown owns it.

session.JoinData holds Player:GetJoinData() read once at join; check SourcePlaceId before trusting TeleportData, and use ReferredByPlayerId and LaunchData for referral and share-link rewards. On leave, session.ExitReason is the Enum.PlayerExitReason from PlayerRemoving, also passed to Kernel.SessionEnd subscribers as (topic, session, exitReason) and to Session.End hooks as context.ExitReason, so analytics and anti-cheat can tell a creator kick from a disconnect. onSession fires before the profile loads: setup that reads saved data subscribes to Kernel.ProfileLoaded instead (see Set up a player when their data loads).

Go deeper: Sessions, Milestones.

Group cleanups with a scope

A scope gives related resources one lifetime: connections, timers, instances, tweens and cleanup functions close together, early or at shutdown. One scope per match:

src/Server/Game/Services/ArenaService.luau

--!strict
-- ArenaService: one scope per match; ending the match removes everything it created.

local ServerStorage = game:GetService("ServerStorage")

local MatchSeconds = 180

local ArenaService = {
  Name = "ArenaService",
}

function ArenaService.init(self: any, kernel: any)
  self.Kernel = kernel
end

function ArenaService.startMatch(self: any, arena: Model)
  self:endMatch()
  local Match = self.Kernel:createScope()
  self.Match = Match

  local Barrier = Match:add(ServerStorage.Arena.Barrier:Clone()) -- destroyed with the match
  Barrier.Parent = arena
  Match:tween(Barrier, TweenInfo.new(2), { Transparency = 0.5 }) -- cancelled if the match ends first

  Match:add(self.Kernel.Bus:subscribe("Combat.Kill", function(_topic, killer, _victim)
    self.Kernel.Bus:publish("Arena.Scored", killer)
  end))
  Match:add(self.Kernel.Scheduler:after(MatchSeconds, function()
    self:endMatch()
  end))

  -- A sudden-death phase that can close on its own, or with the match
  local SuddenDeath = Match:child()
  SuddenDeath:add(self.Kernel.Scheduler:after(MatchSeconds - 30, function()
    self.Kernel.Bus:publish("Arena.SuddenDeath")
  end))
end

function ArenaService.endMatch(self: any)
  if self.Match then
    self.Match:destroy()
    self.Match = nil
  end
end

function ArenaService.stop(self: any)
  self:endMatch()
end

return ArenaService

scope:add(resource, method?) returns the resource. It accepts cleanup functions, Instances, RBXScriptConnections and any object with destroy, Destroy, disconnect, Disconnect, cancel or Cancel; pass a method name for another convention (scope:add(handle, "close")). destroy() cleans in reverse registration order; a cleanup that errors is logged and the rest still run, and a second destroy() does nothing. A resource added to a closed scope is cleaned at once. scope:tween plays a tween the scope owns: closing cancels it, which leaves the properties where they are, and a tween that completes leaves the scope on its own. scope:child() makes a scope that closes with its parent or earlier, and removes itself from the parent when it does. scope:remove(resource) hands cleanup back to you without running it.

Scopes from kernel:createScope() (server or client) that are still open are closed during kernel shutdown, after the service stop phase. Treat that as a backstop and close shorter lifetimes yourself. Cleanup callbacks must not yield, and a raw thread or process needs an explicit cleanup function. Use a scope for anything that is not per-player; session:bind is the per-player equivalent, but it only understands disconnect, destroy and Destroy, so wrap a Scheduler handle there in a function that calls :cancel().

Go deeper: Scope, Sessions.

Schedule work without polling

Every timer is a Scheduler deadline. Pick the shape that matches the need:

-- Run once later: keep the handle and cancel it on cleanup
local Handle = Kernel.Scheduler:after(5, function()
  Kernel.Bus:publish("Shrine.Recharged")
end)

-- Coalesce a burst of changes into one piece of work: a dirty flag plus one pending deadline
local Pending = nil
local function markDirty()
  if Pending then
    return
  end
  Pending = Kernel.Scheduler:after(0.2, function()
    Pending = nil
    rebuildScoreboard()
  end, Kernel.Priority.Low)
end

-- Continuous work while something is active: start with the first item, cancel with the last
local Burning: { [any]: boolean } = {}
local Ticker = nil
local function ignite(session)
  Burning[session] = true
  Ticker = Ticker or Kernel.Scheduler:every(1, function()
    for Victim in Burning do
      burn(Victim)
    end
  end)
end
local function extinguish(session)
  Burning[session] = nil
  if Ticker and next(Burning) == nil then
    Ticker:cancel()
    Ticker = nil
  end
end

every, after and schedule(fn, priority?) return handles with :cancel(). The scheduler keeps timers in a deadline heap and wakes once for the earliest one, so an idle server spends nothing; it never rescans recurring jobs each frame. It drains queues in priority order (Kernel, High, Normal, Low, Background) under a per-frame budget. On the server the budget is frame-aware: each step spends the slack left under a 16.5 ms frame target after the engine’s measured share (Stats.HeartbeatTime and Stats.PhysicsStepTime, in seconds) and its own recent cost, and never less than a 1.5 ms floor; a frame already late falls back to the floor. The client uses a fixed 1.5 ms slice, because render cost does not show in that stat. Background runs only on spare budget, and Kernel priority always runs. Errors are isolated and reported on Scheduler.OnError.

Scheduled callbacks run inside the kernel step. A callback that must yield (an *Async call) wraps the yielding part in task.spawn with a busy flag. For expiries (buffs, cooldowns, invites) use one after per item or a timestamp checked when used, never a sweep. Per-frame camera or physics coupling uses kernel:phaseScheduler("PreRender") or "PreSimulation", only while active.

Go deeper: Scheduler, Event-driven execution.

Run long-lived behaviors you can pause or kill

A Process is a coroutine with a lifecycle (an id, a state machine, an exit signal) stepped by the scheduler. It suits scripted sequences that run across many frames:

local Process = require(ReplicatedStorage.ChloeKernel.Process)

local Siege = Kernel:spawnProcess(function(waves)
  for Index, Wave in waves do
    Kernel.Bus:publish("Siege.Wave", Index)
    spawnWave(Wave)
    Process.sleep(Wave.Delay) -- parks on a scheduler deadline; nothing runs meanwhile
  end
end, { Name = "Siege" }) -- shows in the hot-task profiler as "Process Siege"
Siege.OnExit:connect(function(finalState, result)
  -- Completed, Crashed (result is the traceback) or Killed (result is the reason)
end)
Siege:start(Waves)

Siege:suspend() -- stops stepping; state is kept
Siege:resume()
Siege:kill("arena closed") -- closes the coroutine and fires OnExit

Process.sleep(seconds) parks the body on a scheduler deadline instead of stepping it every frame, and Process.yield() hands control back until a later step, so a process spreads its work under the same frame budget as everything else. A body may also wait on an engine call (task.wait, WaitForChild, an *Async method); the engine resumes it when the call completes, and its next Process.yield() hands it back to the scheduler. kill() cancels a pending engine resume. Profiler time is attributed to the process, not to the kernel’s resume line. SpellKit channels run this way.

Go deeper: Processes.

Run heavy CPU work in parallel

ActorPool runs Parallel Luau workers and handles readiness, result routing and timeouts:

src/Server/Game/Workers/SightWorker.luau

--!strict
-- SightWorker: line-of-sight checks for the actor pool. Runs desynchronized.

local World = workspace -- resolve engine objects at the top level, which runs serially

return function(eye: Vector3, target: Vector3): boolean
  return World:Raycast(eye, target - eye) == nil -- WorldRoot:Raycast is thread safe
end

local Pool = ActorPool.new({ Size = 4, WorkerModule = script.Parent.Parent.Workers.SightWorker })
local Ok, Clear = Pool:dispatch(npcEye, targetPosition) -- yields

-- Fork-join: one batch split into balanced chunks; results come back in item order
local Grid = ActorPool.shareContext({ Cells = CoverCells, Size = GridSize }) -- one frozen SharedTable
local AllOk, Results = Pool:forkJoin(CoverRequests, { Context = Grid })

The worker is a pure function, arguments in and result out, which is the shape Parallel Luau runs safely. The pool waits for actors to signal readiness before the first dispatch, routes each result back to its caller and times out a hung worker (10 seconds by default). forkJoin splits a batch into chunks (4 waves per actor by default), and every actor pulls the next chunk as soon as it finishes, so uneven items balance themselves; the caller resumes once with (true, results) or (false, firstError). A caller cancelled while it waits is never resumed. Workers may only call members the engine marks thread safe (WorldRoot:Raycast, Blockcast, Spherecast, the GetPartBounds* queries, GetAttribute, FindFirstChild, CollectionService:GetTagged); GetService, SetAttribute and PathfindingService are unsafe there, so resolve services at the worker’s top level and keep engine pathfinding on the main thread. Each actor requires its own copy of the worker and everything it requires, so framework singletons are not booted inside it.

Go deeper: ActorPool.

Log with scopes and sinks

local Log = Logger.scope("Shop")
Log.info("purchase", player.Name, itemId)
Log.warn("stock low", itemId)

-- Ship every entry to your warehouse without touching call sites. Log entries have
-- no player, so they need a custom Destination: Roblox analytics drops playerless events
local LogEvents = Analytics.attach(kernel, { Destination = sendToWarehouse })
Logger.addSink(function(entry)
  LogEvents:track(entry.Scope, nil, entry.Level, { Message = entry.Message })
end)
Logger.setLevel(Logger.Level.Debug)

Scopes tag every entry with the system that wrote it, and sinks receive structured entries (Scope, Level, LevelName, Message, Args, Clock) instead of formatted strings. Sinks are isolated: a broken sink cannot break the caller, and one removed during an emit does not skip the next. addSink returns its remover. Framework modules warn through their own scopes ([DataDriver PlayerData], [NetDriver]), so sinks, setLevel and setConsoleEnabled cover framework warnings too. Logger.capture(handler) routes console lines to handler(line, level) until the returned release runs; TestKit uses it to hold spec warnings. Logger.setStructuredConsole(true) sends console output through LogService with { Scope, Level, Arg1..n } as structured context, which the Developer Console shows as data.

Go deeper: Logger, Analytics.

Capture and deduplicate script errors

local Watcher = ErrorWatch.attach(kernel)

-- Script errors have no player: send them to a custom Destination
local ErrorEvents = Analytics.attach(kernel, { Destination = sendToWarehouse })
kernel.Bus:subscribe("Kernel.ScriptError", function(_topic, message, trace, source, count)
  ErrorEvents:track("ScriptError", nil, count, { Message = message, Source = source })
end)

Identical errors log once per window (30 seconds) with an occurrence count, so a crash loop becomes one line with (x4000) instead of four thousand. Deduplicated errors publish Kernel.ScriptError(message, trace, sourceName, count), including crashes reported by the Scheduler, so the same stream feeds logs, a warehouse or an alerting sink. Analytics’ WatchErrors = true bridges the topic into ScriptError events itself, but with the default Roblox destination those events have no player and are dropped (Stats.DroppedPlayerless), so pair it with a custom Destination.

Go deeper: ErrorWatch, GcWatch.

Tune values live without redeploying

src/Server/Game/Services/TuningService.luau

--!strict
-- TuningService: live tunables and kill switches.

local ServerScriptService = game:GetService("ServerScriptService")

local LiveConfig = require(ServerScriptService.ChloeKernelServer.LiveConfig)

local TuningService = {
  Name = "TuningService",
}

function TuningService.init(self: any, kernel: any)
  self.Kernel = kernel
  self.Config = LiveConfig.new(kernel, {
    Defaults = { BossHealth = 100000, DoubleXP = false, NewShopEnabled = true },
  })
end

function TuningService.start(self: any)
  self.Connection = self.Kernel.Bus:subscribe("Config.Changed", function(_topic, key, value)
    if key == "DoubleXP" then
      self.Kernel.Bus:publish("Event.DoubleXP", value)
    end
  end)
end

-- Admin tooling: reaches every server at once over MessagingService. Yields
function TuningService.killShop(self: any): boolean
  local Ok = self.Config:set("NewShopEnabled", false)
  return Ok
end

function TuningService.stop(self: any)
  if self.Connection then
    self.Connection:disconnect()
  end
  self.Config:destroy()
end

return TuningService

Defaults is the source-controlled truth, and Config:get(key) reads the current value. set and update write an override into one MemoryStore document and publish an invalidation on the CKLiveConfig MessagingService topic, so every server re-reads at once; a poll reconciles missed messages (30 seconds, relaxed to 120 once invalidations arrive; Studio, without MessagingService, relies on the poll). Config.Changed(key, value) fires only on real changes, and a reverted key falls back to its default. Each server reads once at boot, so one that starts mid-event picks up the overrides at once. The document has a 7-day TTL that running servers refresh, so overrides expire only after seven days with no server online. The field __CKWrittenAt is reserved.

To edit values in the Creator Hub instead, pass Source = "ConfigService": LiveConfig reads Roblox Experience Configs once at boot and refreshes on ConfigSnapshot.UpdateAvailable, with no polling, MemoryStore or MessagingService. Only keys in Defaults are read, and a value whose type differs from its default is ignored with one warning. A failed read serves the defaults and retries at 5, 15, 45 and 135 seconds, then every 300 seconds. In this mode set and update return false and apply(key, value) still overrides one server. Config:forPlayer(session):get(key) returns that player’s value with their experiments applied; the first call per session yields.

Status

The logic is covered by specs with mock services. The MemoryStore source needs Studio Access to API Services in Studio, and cross-server invalidation only shows on a published game with several servers.

Go deeper: LiveConfig.

Watch kernel health in live servers

Kernel:enableDiagnostics(5) -- publishes every 5 seconds; Main.server.luau already calls it with 1 in Studio and debug places

Kernel.Bus:subscribe("Kernel.Overload", function(_topic, snapshot)
  Log.warn("tasks deferred for three windows in a row", snapshot.Scheduler.TasksDeferred)
end)

print(Kernel:stats()) -- { Version, Role, Scheduler = { LastStepSeconds, TasksRun, TasksDeferred }, ServiceTimings, Sessions, Net? }

Diagnostics write the kernel’s own numbers (scheduler health, service timings, session counts, net counters, total memory) as attributes on a Configuration named ChloeKernelDiag<Role> under ReplicatedStorage. The debug panel’s SERVER window mirrors them, and an external watchdog can read them without a network channel. Kernel.Overload fires once tasks have been deferred in three or more consecutive diagnostics windows, and again every window while that lasts, so alerting is a Bus subscription. Net appears in stats() once kernel:net() has been built.

Go deeper: Kernel and boot, Scheduler.

Watch everything live with the debug panel

Try it live

ChloKernel Test is the framework’s public test place with the debug panel on for every joiner. Press F8 in game to open the windows on a real server.

The panel is an immediate-mode overlay with four windows: CLIENT, SERVER, NET and HOOKS, plus SIM CONTROL wired by the client Bootstrap. CLIENT shows frame rate, ping, network rate, physics rate, memory, scheduler health with hot tasks, kernel internals and a log tail. SERVER mirrors the diagnostics attributes. NET is a live wire inspector for both machines: direction, channel, decoded arguments, payload bytes, processing or round-trip time and the verdict (Rejected, RateLimited, Dropped, Unguarded); click a row to pin it and see the decoded values next to the serialized bytes. HOOKS lists every hook point with fire and rejection counts. F8 toggles everything, and closed panels cost nothing per frame: no render tasks, frame or GC sampling, log sink or NET capture run until F8 opens them.

Reading it in brief: colors are health grades (green good, orange worth watching, red act now); the Frame sections split frame time into engine and other scripts, physics, the kernel share and client render, so a slow frame names whose problem it is; hot tasks name the offender by script:line; every stat has a tooltip that says where to look when it is wrong; and the GC rows tell a leak (heap climbing, reclaims near zero) from churn (heap stable, busy cycles).

The panel attaches in Studio and refuses to exist in production unless ReplicatedStorage has the ChloeKernelDebug attribute set to true. Access then fails closed: only Studio, the operators in NetTap.OwnerUserIds and the UserIds in ChloeKernelDebugUserIds (comma-separated; * or all opens it to everyone) get the panel, the NET and log mirrors fire per recipient, and the simulation controls reject everyone else.

Note

NetTap.OwnerUserIds is allowed whenever ChloeKernelDebug is true, whatever ChloeKernelDebugUserIds says. It always includes CryptedChloes (7241279999), the framework’s only developer: debugAllowed checks that id itself, and directive CK017 rejects code anywhere that removes or replaces the entry. Add your own operators from shared bootstrap code so the server and clients agree: NetTap.OwnerUserIds[123456789] = true.

Build your own windows with the same library:

local ImGui = require(ReplicatedStorage.ChloeKernel.Debug.ImGui)
local Window = ImGui.window("Combat", { Parent = screenGui })
-- Re-render as often as you like: widgets reuse their instances by call position
Window:render(function(ui)
  ui:header("Weapons")
  ui:value("Active projectiles", ActiveCount)
  -- A list that grows or shrinks goes in a keyed section, so later widgets keep their instances
  ui:section("targets", function(ui)
    for _, Target in Targets do
      ui:selectable(Target.Name, Target == Picked)
    end
  end)
  ui:sparkline("damage/s", DamageHistory, 500)
  DrawAimLines = ui:checkbox("Draw aim lines", DrawAimLines)
end)

Go deeper: Debug panels, Simulation.

Reuse instances instead of churning them

local Casings = Pool.new({
  Create = function()
    return ReplicatedStorage.Assets.ShellCasing:Clone()
  end,
  InitialSize = 16,
})

local function ejectCasing(at: CFrame)
  local Casing = Casings:acquire()
  Casing:PivotTo(at)
  Casing.Parent = workspace
  Pool.clearEffects(Casing) -- after the first CFrame write: drop trail segments and particles from its last life
  Kernel.Scheduler:after(2, function()
    Casings:release(Casing)
  end, Kernel.Priority.Low)
end

Instance.new and Destroy churn fragments memory and stutters frames; a pool turns both into table operations. acquire() returns a parked instance or creates one, and release() parks it again, so the instance count stays flat however fast effects fire. release errors on a double or foreign release. Pool.quietEffects(visual, hideBody?) stops a retiring visual’s emitters so it can fade out, and returns a function that restores exactly what it changed before you release it. forget, trim(keep) and destroy manage the pool itself. ProjectileClient and HazardClient pool their visuals this way.

Go deeper: Pool.

Ship gameplay analytics

src/Server/Game/Services/TelemetryService.luau

function TelemetryService.init(self: any, kernel: any)
  self.Events = Analytics.attach(kernel, {
    MaxPerMinute = 120, -- a framework cap on queued events; the engine documents no budget
    Sample = { PositionPing = 0.1 }, -- keep 10% of a chatty event
  })
end

function TelemetryService.roundFinished(self: any, winner: Player, roundNumber: number)
  self.Events:track("RoundFinished", winner, roundNumber)
end

function TelemetryService.bought(self: any, player: Player, price: number, balance: number, itemId: string)
  self.Events:economy(player, "Sink", "Coins", price, balance, Enum.AnalyticsEconomyTransactionType.Shop, itemId)
end

function TelemetryService.stop(self: any)
  self.Events:destroy() -- drains what is queued when the server closes
end

track(name, player?, value?, fields?) is a queue insert. A flush runs FlushSeconds (10) after the first queued event, off the scheduler step, and hands the batch to AnalyticsService (LogCustomEvent and the typed Log*Event methods) or to an injected Destination(batch) -> delivered?, which may yield. Over-cap, sampled-out and failed events are dropped and counted in Stats, never silently, and the default destination sends each event on its own, so one failure does not lose the batch. economy, funnel, onboarding and progression check the documented argument rules before queueing and drop invalid calls with one warning each (Stats.DroppedInvalid). With the default destination, custom field keys other than CustomField01 to CustomField03 warn once per event name, and the server warns once when it sees more distinct currencies (5), transaction types (20), SKUs (100), funnels (10) or field combinations (8,000) than Roblox keeps per experience. Every Roblox analytics event needs a player; playerless events need a custom Destination.

Go deeper: Analytics.