Solution 02 / AI Inference Engineering

Make production LLM systems fast enough to use and cheap enough to keep.

Serving-layer engineering for teams whose AI feature works but costs too much, responds too slowly, or falls over under real traffic.

Discuss AI Inference ↗

01The problem

Inference cost is usually treated as a model-selection problem when it is a systems problem. Teams pick a smaller model and accept worse output, when the real waste is elsewhere: batch sizes tuned for an empty queue, no prefix caching on prompts that are 80% identical, GPUs idling between bursts, and every request routed to the largest model regardless of difficulty. The result is a bill that scales linearly with usage and a latency profile nobody has actually measured end to end.

Typical symptoms

Inference spend growing in step with usage, with no unit-economics model to explain it
p50 latency looks acceptable; p95 and p99 are quietly unusable
GPU utilisation far below what the hardware could sustain
Every request hits the largest available model, regardless of how hard it is
Long, near-identical system prompts re-processed on every single call
Cost per request unknown, so no one can say whether a feature is profitable
Traffic spikes cause timeouts, and the fix so far has been over-provisioning

Decisions we help you make

Self-host versus managed API: decided on real load, latency targets, and total cost, not preference
Which serving stack fits the workload (vLLM, TensorRT-LLM, SGLang, or a managed endpoint)
Quantisation: what precision is acceptable, verified against your own evals rather than assumed
How to route by difficulty: small model first, escalate only when it fails a check
Batching and scheduling policy, given your real arrival pattern and latency SLO
What to cache (prefix, semantic, or exact), and where the correctness boundary sits
Autoscaling policy and how much cold-start latency the product can absorb

02How we work on it

Methods we apply.

End-to-end latency decomposition: queue wait, prefill (TTFT), decode (TPOT), network, post-processing

Continuous batching and paged KV-cache tuning against measured arrival distributions

Prefix and prompt caching for shared system-prompt and few-shot prefixes

Quantisation evaluation (FP8, INT8, AWQ, GPTQ), scored on your task rather than on public benchmarks

Speculative decoding and draft-model pairing where the workload rewards it

Model cascade and routing design, with confidence checks gating escalation

Tensor/pipeline parallel sizing and GPU memory budgeting for the target context length

Load testing at realistic concurrency, then unit-economics modelling: cost per request, per user, per feature

Throughput and latency move in opposite directions as batch size grows. The work is finding the largest batch that still clears the latency SLO. Past that knee, you buy throughput with user-visible delay.

Optimising the model alone addresses one band of this bar. Queue wait, prefill and decode each respond to different fixes: batching, caching, and parallelism respectively.

03What moves

Metrics this work is measured on.

Cost per 1M tokens, and cost per requestTime to first token (p50 / p95 / p99)Time per output tokenThroughput at target latency SLOGPU utilisationCache hit rateError and timeout rate under peak load

04What we need to start

Current serving setup: models, hardware or endpoints, and the framework in front of them
Traffic data: request volume, arrival pattern, prompt and completion length distributions
Latency and cost targets, and what the product genuinely needs versus what is aspirational
An existing eval set, if you have one. If not, building one is step one
Recent inference billing, broken down by model and endpoint

05Engagement path

How this becomes an engagement.

Inference Audit

2 weeks

Benchmark the current stack, decompose latency and cost, and quantify the achievable improvement before you commit to a build.

Optimisation Engagement

4–10 weeks

Implement the serving, caching, routing and quantisation changes, with before/after numbers on every one.

Load & Regression Harness

Ongoing

Leave behind the benchmark and eval harness so future model or prompt changes are measured, not guessed at.

Want to see what this looks like against your own systems?

Most engagements start with a short, fixed-scope assessment, enough to quantify the opportunity before anyone commits to a build.

Start a conversation ↗