Automatic discovery
Combines process scanning, port scanning, and read-only API fingerprinting to identify running engines and the models they serve.
Project 02 / Runtime observability
A clearer view of local inference.
An nvtop-style terminal monitor that discovers local model runtimes, displays GPU and serving metrics, and records snapshots for replay and diagnostics.
The problem
GPU utilization alone doesn't explain which model is serving, how many requests are queued, or whether the KV cache is under pressure. With several local runtimes, that information is spread across processes and APIs.
I built llmtop to bring discovery, GPU telemetry, and serving metrics into one terminal view, with recorded history for understanding what happened.
What I built
Combines process scanning, port scanning, and read-only API fingerprinting to identify running engines and the models they serve.
Shows GPU utilization and memory alongside token throughput, queue depth, and KV-cache usage, where the backend exposes them.
Records normalized snapshots to a rotating log. Scrub through recent history or replay a recording later without a live server.
Engineering decisions
Discovery and metrics collection use introspection endpoints. llmtop does not send generation requests or consume model tokens to inspect a runtime.
On supported NVIDIA unified-memory devices, shared memory is labeled correctly. Per-process telemetry helps attribute memory use to engines.
Unavailable GPU fields and missing backend metrics appear as n/a. Discovery and serving metrics remain useful on hosts without an NVIDIA GPU.
Runtime coverage
Metrics depend on what each engine exposes. Auth-gated endpoints are recognized and can be inspected with a configured API key.
Try it locally
Install from the GitHub repository, then launch the terminal monitor. Use --json for a snapshot or --replay to inspect a recording.
The name llmtop on PyPI belongs to a different project. Use the repository install command shown here.
pip install git+https://github.com/rxxusp/llmtop.git
llmtop
llmtop --json
llmtop --replay recording.jsonl