Cerberus

Cerberus — scope

Cerberus is Carino Systems’ self-hosted LLM runtime. It has three parts, which give it its name:

  1. Models. It runs local LLMs efficiently: it loads models on demand, unloads them when idle and shows each request’s speed and energy use.
  2. Gateway. It serves one OpenAI-compatible endpoint that routes each request to a local model or to a cloud provider (API keys).
  3. Agents. It runs coding-agent sessions (its own built-in agent, Claude Code, OpenCode and so on), each in its own git worktree. You can leave them running and steer them from a web UI on your phone or desktop.

Cerberus is a service plus a desktop client. The service is the Go binary cerberus. The desktop base is the existing Atrium LLM chat (llm-chat.py), which talks to Cerberus as an OpenAI-compatible server. The desktop client gets its own phase later.

Linux comes first. macOS and Windows are future targets, so the code must cross-compile today: no cgo, and platform-specific code goes behind build tags, with stubs where a platform isn’t supported yet.

Principles

Phases

Phase Contents
1 (this) Model hosting and gateway; agent sessions with worktrees, test gates, previews, a resource guard and queue, escalation and remote approval; web UI; CLI; systemd user unit
2 Usage and limit tracking per subscription, searchable session history and resume, voice tasks (local Whisper), secrets vault, fleet-wide tasks (one instruction → one session per repo)
3 GPU power-cap auto-tuner (tokens per joule), speculative-decoding and quantisation presets per GPU, vLLM backend, more than one GPU per box, several hosts (agent mode plus gateway mode)
4 (client edition) Plain chat for non-developers (Spanish first), RAG over documents, several users with roles, budgets and audit log (LFPDPPP), fully offline, Carino remote support (health alerts, updates, opt-in support tunnel)
5 (desktop) Cerberus desktop client grown from Atrium LLM chat, dynamic-island agent status, launcher “new task”, bar readout of GPU power and tok/s; macOS and Windows builds

Phase 1 runs on one host, but the data model already holds a host with a list of GPUs, so several GPUs and several hosts don’t need a rewrite later.

Phase 1 specification (the contract between packages)

Module path: carino.systems/cerberus. Binary: cerberus. Go 1.26 or later. CGO_ENABLED=0. These must all build: GOOS=linux, GOOS=darwin and GOOS=windows (go build ./..., go vet ./...).

Layout and ownership

cmd/cerberus/        main + subcommands (serve, agent, status, task, url, install-service, version)
internal/config/     Config types, defaults, load/save, token generation       (FROZEN — see config.go)
internal/platform/   paths per OS, process-group kill, meminfo, PTY (build tags)
internal/gpu/        GPU monitor (nvidia-smi, AMD sysfs, none) + energy integrator
internal/engine/     model backends + manager (load/unload/idle TTL/make room)
internal/usage/      usage records (JSONL) + aggregates
internal/gateway/    OpenAI-compatible /v1 proxy with streaming + metrics
internal/session/    agent sessions: PTY, scrollback, worktrees, checks, preview, queue/guard, escalation, accept
internal/agent/      built-in headless coding agent (tool loop) run as `cerberus agent`
internal/notify/     notifications (command hook, ntfy, notify-send)
internal/api/        /api/* JSON + SSE, auth, preview mount, embeds web/
web/                 the web UI (embedded with go:embed via web/embed.go)
packaging/           cerberus.service (systemd user unit)

Runtime paths (platform)

GPU (internal/gpu)

type GPU struct { Index int; Name string; Vendor string; MemTotalMiB, MemUsedMiB int; PowerW, PowerLimitW float64; UtilPct int; TempC int; PState string }
type Snapshot struct { Time time.Time; GPUs []GPU; EnergyJ float64 /* cumulative since start, all GPUs */ }
type Monitor struct{ ... }
func New(cfg config.GPUConfig, log *slog.Logger) *Monitor
func (m *Monitor) Run(ctx context.Context)          // poll loop
func (m *Monitor) Snapshot() Snapshot                // idle: re-queried on demand when older than IdleIntervalMs
func (m *Monitor) SetActive(active bool)             // poll every ActiveIntervalMs when true; no polling when false
func (m *Monitor) Refresh(ctx context.Context)       // query now (before make-room decisions)
func (m *Monitor) EnergyJ() float64                  // cumulative joules (trapezoid), extrapolated from the last sample
func (m *Monitor) FreeVRAMMiB() int                  // sum free; -1 if unknown (idle: on demand like Snapshot)

Backends: NVIDIA via nvidia-smi --query-gpu=index,name,memory.total,memory.used,power.draw,power.limit,utilization.gpu,temperature.gpu,pstate --format=csv,noheader,nounits; AMD via /sys/class/drm/card*/device/{mem_info_vram_total,mem_info_vram_used,gpu_busy_percent,hwmon/hwmon*/power1_average}; otherwise none. Run one query per tick, never a resident -lms process. A missing tool is not an error: the result is an empty snapshot.

Engine (internal/engine)

type ModelInfo struct { ID, Backend, Name string; Local, Loaded bool; SizeMiB int; Ctx int; LastUsed time.Time; InFlight int; Vision bool }
type Endpoint struct { BaseURL string /* ends in /v1 */; Model string /* id the backend expects */; Headers map[string]string; Local bool; Backend string }
type Backend interface {
    Name() string                                              // "llamacpp" | "lmstudio" | "ollama" | "openai"
    Available() bool
    List(ctx context.Context) ([]ModelInfo, error)
    Load(ctx context.Context, m ModelInfo) (Endpoint, error)   // idempotent
    Unload(ctx context.Context, m ModelInfo) error
    Endpoint(m ModelInfo) Endpoint
}
type Manager struct{ ... }
func NewManager(cfg *config.Config, gpu *gpu.Monitor, log *slog.Logger) *Manager
func (m *Manager) Run(ctx context.Context)                                  // idle reaper (per-model TTL), refresh lists, stops spawned processes on ctx done
func (m *Manager) Models(ctx context.Context) []ModelInfo                   // cached ≤5 s
func (m *Manager) Acquire(ctx context.Context, id string) (Endpoint, func(), error) // load if needed (making room), InFlight++; release func: InFlight--, LastUsed=now
func (m *Manager) Load(ctx context.Context, id string) error
func (m *Manager) Unload(ctx context.Context, id string) error
func (m *Manager) Shutdown(ctx context.Context)

Usage (internal/usage)

type Record struct { Time time.Time; Model, Backend, Client string; Local bool; PromptTokens, CompletionTokens int; TTFTMs, DurationMs int64; TokPerSec float64; EnergyJ float64; TokPerJ float64; Status int; Error string; SessionID string }
type Store struct{ ... }
func Open(path string, keep int) (*Store, error)   // appends JSONL; keeps last `keep` in memory
func (s *Store) Add(r Record)
func (s *Store) Recent(n int) []Record
type Agg struct { Model string; Requests, PromptTokens, CompletionTokens int; EnergyJ, AvgTokPerSec, AvgTokPerJ float64 }
func (s *Store) Aggregate(since time.Time) []Agg
func (s *Store) Close() error

Gateway (internal/gateway)

func New(eng *engine.Manager, mon *gpu.Monitor, store *usage.Store, log *slog.Logger) *Gateway
func (g *Gateway) Handler() http.Handler   // mounted at /v1 by api (paths below are relative to /v1)

Sessions (internal/session)

States: queued → starting → running ⇄ waiting → checking → passed | failed → (retrying | escalated) …, with terminal states accepted | discarded | stopped | error. waiting means the agent asked for input (the OSC marker below), or produced no output for IdleWaitSec while still running (interactive CLIs). When the process exits, checks run if the repo has any. With no checks configured, exit code 0 → passed.

type TaskSpec struct {
    Repo   string   // absolute path of a git repo (must be in cfg.Repos or under cfg.RepoRoots)
    Task   string   // the instruction
    Agent  string   // AgentProfile name (default cfg.DefaultAgent)
    Model  string   // model id for agents that take one (default engine default)
    EscalateTo string // profile to hand over to after MaxAttempts failed check runs ("" = none)
    MaxAttempts int // attempts with Agent before escalating (default cfg.Sessions.MaxAttempts)
    Count  int      // fan-out: N independent sessions with the same task (default 1), each its own worktree
    Title  string
}
type Info struct {
    ID, Title, Repo, Task, Agent, Model, EscalateTo string
    State string; Attempt int; ExitCode *int; Created, Updated time.Time
    Worktree, BaseCommit string; ChecksOutput string; PreviewURL string; Error string; Group string /* fan-out group id */
    WaitingReason string; Cols, Rows int
}
type Event struct { Type string /* "state" | "created" | "removed" */; Session Info }
type Manager struct{ ... }
func NewManager(cfg *config.Config, eng *engine.Manager, gpu *gpu.Monitor, notifier *notify.Notifier, log *slog.Logger) (*Manager, error) // loads existing sessions from disk (live ones become "stopped", reason "daemon restarted"; worktree kept)
func (m *Manager) Run(ctx context.Context)                       // queue scheduler (every 2 s + on signal)
func (m *Manager) Create(ctx context.Context, t TaskSpec) ([]Info, error)
func (m *Manager) List() []Info
func (m *Manager) Get(id string) (Info, bool)
func (m *Manager) Input(id string, data []byte) error
func (m *Manager) Resize(id string, cols, rows int) error
func (m *Manager) Attach(id string) (backlog []byte, ch <-chan []byte, detach func(), err error) // backlog = last ScrollbackTailKiB of output
func (m *Manager) FollowUp(ctx context.Context, id, text string) error  // running → typed into the PTY + "\r"; finished → new attempt of the same agent in the same worktree with text as the task
func (m *Manager) RunChecks(ctx context.Context, id string) error
func (m *Manager) Stop(id string) error
func (m *Manager) Diff(id string) (string, error)                      // unified diff of worktree vs BaseCommit, including untracked files
func (m *Manager) Accept(ctx context.Context, id string, o AcceptOpts) (string /*commit sha*/, error)
func (m *Manager) Discard(id string) error                             // kill, git worktree remove --force, state discarded
func (m *Manager) Subscribe() (<-chan Event, func())
func (m *Manager) Shutdown(ctx context.Context)
type AcceptOpts struct { Message string; Push bool }

Built-in agent (internal/agent, run as cerberus agent)

cerberus agent --base URL --model M --task-file F --cwd DIR [--key K] [--session ID] [--max-steps N] [--approve auto|ask]

Notify (internal/notify)

type Message struct { Title, Body, Urgency string /* low|normal|critical */; SessionID string; URL string }
func New(cfg config.NotifyConfig, log *slog.Logger) *Notifier
func (n *Notifier) Send(msg Message)    // non-blocking; never fails the caller

Channels, all optional: Command (argv, with {title}, {body} and {urgency} substituted; the default is Atrium’s ~/.config/hypr/scripts/notify when it exists), NtfyURL (a POST with a Title header), and notify-send as a fallback when Command is empty and the binary exists. Sends are rate-limited per session (at most one every 10 s per state).

API (internal/api)

type Deps struct { Cfg *config.Config; Engine *engine.Manager; GPU *gpu.Monitor; Usage *usage.Store; Gateway *gateway.Gateway; Sessions *session.Manager; Log *slog.Logger; Version string }
func New(d Deps) http.Handler

Web UI (web/)

CLI (cmd/cerberus)

Desktop integration (Phase 1 scope)

Acceptance (Phase 1)