Skip to content
Skip to this exploration
All explorations

Practical self-hosting · intermediate

29 · Measure the service people experience

Connect logs, metrics and traces to a repeatable benchmark.

10 min 5 sectionsv1

What you can take away

  • Create a benchmark card with quality and latency checks.
  • Create the benchmark card and a results table before executing anything

Your trail through this idea

Checking saved progress…

A new model feels faster during one conversation. Was the prompt shorter, the cache warm, or the queue empty? Without a baseline, enthusiasm and regression look similar.

Prerequisite: complete the preceding lesson or bring its worksheet.

Mark sections complete when you choose. Reading alone does not mark them.

Understand the system

OpenTelemetry describes logs, metrics, and traces as distinct signals. Logs describe events, metrics summarize measurements, and traces connect work across a request path. Choose signals that answer operational questions without recording private prompts by default. A request identifier can connect stages without becoming a copy of the user’s content.

Measure time to first useful output, total completion time, errors, queue delay, and task quality. A fast wrong answer is not an improvement. Record workload, software version, model manifest, output cap, concurrency, and warm/cold condition so comparisons remain interpretable.

Checking saved progress…

Work through the case

Original benchmark set: five short factual questions over synthetic documents, two no-evidence questions, and three bounded writing tasks. Score factual support separately from style. Run configurations A and B on the same set. Report every failure rather than selecting the nicest response. A small set teaches method; it does not establish universal superiority.

Checking saved progress…

Make something useful

Try this

Original exercise

Create the benchmark card and a results table before executing anything. If using a test endpoint, run a small capped batch and stop if it affects other users. Keep prompts synthetic. Record median and slowest observed latency, along with sample size; avoid presenting a tiny sample as a reliable tail-latency estimate.

Checking saved progress…

Check your reasoning

A quick check · for your own learning

What makes a benchmark comparison interpretable?

Choose one answer

A self-study prompt. You can retry; this is not an exam.

Checking saved progress…

Keep the result and follow the sources

Success is a comparison someone else could repeat. State the practical limits: these tasks, these conditions, this version. Expand the test only when the decision requires broader evidence.

Sources and scope

  • Primary documentation or archival source, reviewed 14 November 2026. Additional direct links appear beside the claims they support.
  • Editorial basis: Ian's historical self-hosting questions and current repository inspection informed the case selection. Historical requests do not establish current deployment. Private chat records, credentials and host details are excluded.
  • Examples and exercises are original educational synthesis. External sources support the stated technical or historical context, not every hypothetical scenario.
Checking saved progress…

These checks are for reflection and practice. Your answers and reflection notes are not sent anywhere.

Back to all explorations