Private, multi-server LLM capacity with task-based model routing behind one internal API.
- 01Internal tools
- 02API gateway
- 03Task router
- 04Model pool
- 05GPU nodes
The challenge
Internal tools needed dependable language-model capacity. Third-party APIs meant handing over internal data, accepting variable latency, and paying per token for workloads that ran constantly.
Architecture
Several high-memory GPU servers host a pool of quantized models behind one OpenAI-compatible gateway, so every tool integrates once and never learns about the hardware. A router sends each request to the model suited to it, a queue bounds concurrency per node, and a hosted API stays configured as a fallback.
Technical decisions
Quantization was set per model against measured VRAM headroom rather than applied uniformly, because over-quantizing a reasoning model degrades output in ways that are expensive to debug downstream. Context length was derived from concurrency targets, since both compete for the same memory. A small fast model handles classification, so cheap work never occupies expensive capacity.
Responsibilities
- GPU and VRAM capacity planning
- Model evaluation and quantization strategy
- Gateway, routing and serving stack
- Monitoring, alerting and failover
Engineering considerations
- VRAM traded against context length and concurrency
- Power and thermal headroom under sustained load
- Quality regression from aggressive quantization
- Keeping the gateway stable as models change behind it
Technologies
Internal tools call one endpoint and get whichever model fits the task. Capacity grows by adding a node, and the data stays on infrastructure that is directly controlled.
