Practical self-hosting · intermediate
29 · Measure the service people experience
Connect logs, metrics and traces to a repeatable benchmark.
What you can take away
- Create a benchmark card with quality and latency checks.
- Create the benchmark card and a results table before executing anything
Your trail through this idea
Checking saved progress…
A new model feels faster during one conversation. Was the prompt shorter, the cache warm, or the queue empty? Without a baseline, enthusiasm and regression look similar.
Prerequisite: complete the preceding lesson or bring its worksheet.
Understand the system
OpenTelemetry describes logs, metrics, and traces as distinct signals. Logs describe events, metrics summarize measurements, and traces connect work across a request path. Choose signals that answer operational questions without recording private prompts by default. A request identifier can connect stages without becoming a copy of the user’s content.
Measure time to first useful output, total completion time, errors, queue delay, and task quality. A fast wrong answer is not an improvement. Record workload, software version, model manifest, output cap, concurrency, and warm/cold condition so comparisons remain interpretable.
Work through the case
Original benchmark set: five short factual questions over synthetic documents, two no-evidence questions, and three bounded writing tasks. Score factual support separately from style. Run configurations A and B on the same set. Report every failure rather than selecting the nicest response. A small set teaches method; it does not establish universal superiority.
Make something useful
Try this
Original exercise
Create the benchmark card and a results table before executing anything. If using a test endpoint, run a small capped batch and stop if it affects other users. Keep prompts synthetic. Record median and slowest observed latency, along with sample size; avoid presenting a tiny sample as a reliable tail-latency estimate.
Check your reasoning
A quick check · for your own learning
What makes a benchmark comparison interpretable?
A self-study prompt. You can retry; this is not an exam.
Keep the result and follow the sources
Success is a comparison someone else could repeat. State the practical limits: these tasks, these conditions, this version. Expand the test only when the decision requires broader evidence.
Sources and scope
- Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
- Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
- Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.