Project 01 / Inference orchestration

local-engine-router

One endpoint for your local models.

A Python reverse proxy with OpenAI- and Ollama-compatible APIs. It routes requests to local inference engines by model and swaps backends as GPU memory requires.

My role
Creator & developer
Built with
Python · asyncio · HTTP APIs
Requirements
Python 3.10+
License
MIT

The core idea

Let the client ask.
Let the router handle it.

A client sends a model name to one endpoint. The router resolves its engine, manages the memory transition, and forwards the request.

Compatible APIs. One active backend.

The problem

Different engines.
The same memory budget.

Local inference often means several backends, each with its own port and lifecycle. On memory-constrained hardware, keeping multiple large engines loaded can exhaust the available GPU or unified memory.

I built local-engine-router to manage that boundary: clients keep a stable API, while the router selects the backend and handles the transition.

What I built

Routing with the runtime in mind.

01

Compatible endpoints

OpenAI-compatible /v1/* and native Ollama /api/* endpoints let existing clients reach local engines through one proxy.

02

Task-aware selection

Smart routing scores local models for the task, context fit, observed reliability, and the cost of switching away from an already loaded engine.

03

Controlled backend swaps

The router drains requests, frees the outgoing engine, waits for memory reclaim, and starts the target before forwarding the request.

Engineering decisions

Make the transition explicit.

Memory reclaim is part of the lifecycle

Stopping an engine does not immediately make all its memory available. An explicit memory-settle step helps the next backend start after the operating system has reclaimed it.

Keep streaming clients connected

During a cold start, the proxy emits keep-alive frames for streaming requests. Non-streaming requests wait for the swap to complete.

Test without a GPU

I implemented a GPU-free automated test suite with mock backends. The same suite runs in continuous integration, without needing a live inference server.

Try it locally

Get started.

Install the router, detect your running engines, and start the proxy. The init wizard scaffolds a configuration from confirmed local backends.

For process-based engines, configure the start command so the router can restart them after a swap.

Read the full setup guide
Terminal
pip install local-engine-router

local-engine-router init
local-engine-router --config config.yaml