Methods, frameworks, and benchmarks for evaluating AI systems.

Step-by-step evaluation techniques, testing strategies, and practical guidance, from benchmarking to monitoring to framework design.

Dec 2025 15 min

Root Cause Analysis for LLM Behaviour

A step-by-step guide to diagnosing why an LLM behaves unexpectedly. Covers observability, categorisation, hypothesis testing, and reporting, with a full worked example and templates you can use immediately.

Read →

Nov 2025 7 min

Statistical Risk

Key Idea: Use statistics to assess risk. Measure risk statistically by applying a risk rubric with scoring for probability and impact.

Read →

Nov 2025 6 min

Evaluation Over Testing

It's not 'is it right or wrong?', it's 'is it better or worse?' Testing is evolving into evaluation, shifting from Boolean pass/fail to analog better-or-worse.

Read →

Nov 2025 5 min

More is Better

Sending in more requests leads to better evaluation. Testing for stability and consistency, not just correctness, is fundamental to evaluating non-deterministic systems.

Read →

Nov 2025 7 min

Language Heuristics Part 1

Analysing text outputs for verbosity, language complexity and token performance. One of the simplest yet highly effective methods for checking LLM outputs.

Read →

Nov 2025 8 min

Confidence Scoring

Scoring how unsure the LLM is based on what you are asking it to do. Confidence is derived from logprobs and provides a direct insight into token-level certainty.

Read →

Oct 2025 10 min

Semantic Measurements

Key Idea: Scoring the meaning of natural language strings in a measurable way.

Read →