Project 02 / Runtime observability

llmtop

A clearer view of local inference.

An nvtop-style terminal monitor that discovers local model runtimes, displays GPU and serving metrics, and records snapshots for replay and diagnostics.

My role
Creator & developer
Built with
Python · Textual · NVML
Requirements
Python 3.11+
License
MIT
llmtopLOCAL INFERENCE MONITOR● LIVE
GPU UTILIZATION
MEMORY PRESSURE
GPU / unified memory
ENGINETHROUGHPUTQUEUEKV CACHE
vLLM
llama.cpp
Ollaman/an/an/a
SNAPSHOT HISTORY
LIVE
p pause[ ] scrubq quitInterface illustration
Interface illustration · GPU telemetry, per-engine serving metrics, and snapshot history.

The problem

A busy GPU is only
part of the story.

GPU utilization alone doesn't explain which model is serving, how many requests are queued, or whether the KV cache is under pressure. With several local runtimes, that information is spread across processes and APIs.

I built llmtop to bring discovery, GPU telemetry, and serving metrics into one terminal view, with recorded history for understanding what happened.

What I built

From discovery to diagnostics.

01

Automatic discovery

Combines process scanning, port scanning, and read-only API fingerprinting to identify running engines and the models they serve.

02

Runtime visibility

Shows GPU utilization and memory alongside token throughput, queue depth, and KV-cache usage, where the backend exposes them.

03

Record and replay

Records normalized snapshots to a rotating log. Scrub through recent history or replay a recording later without a live server.

Engineering decisions

Observe without interrupting.

Read-only checks

Discovery and metrics collection use introspection endpoints. llmtop does not send generation requests or consume model tokens to inspect a runtime.

Handle unified memory honestly

On supported NVIDIA unified-memory devices, shared memory is labeled correctly. Per-process telemetry helps attribute memory use to engines.

Show what's known

Unavailable GPU fields and missing backend metrics appear as n/a. Discovery and serving metrics remain useful on hosts without an NVIDIA GPU.

Runtime coverage

Works across the local stack.

  • vLLM
  • llama.cpp
  • Ollama
  • TGI
  • SGLang
  • OpenAI-compatible APIs

Metrics depend on what each engine exposes. Auth-gated endpoints are recognized and can be inspected with a configured API key.

Try it locally

One command to look around.

Install from the GitHub repository, then launch the terminal monitor. Use --json for a snapshot or --replay to inspect a recording.

The name llmtop on PyPI belongs to a different project. Use the repository install command shown here.

Read the usage guide
Terminal
pip install git+https://github.com/rxxusp/llmtop.git

llmtop
llmtop --json
llmtop --replay recording.jsonl