Capabilities
Evaluation, observability and control
Measuring how an AI system performs, detecting where it fails and controlling what it is allowed to do.
On one page
We design evals, adversarial testing, benchmarks, simulations, validators, traces, metrics and observability mechanisms for models, agents, knowledge systems, workflows, software and physical systems. We combine automatic evaluation, deterministic checks and specialist review according to what actually needs to be measured.
- 01Evaluating backwards from the result. We start from the expected result.
- 03Representative evals. An evaluation set must represent the use: ordinary cases, difficult cases, edges, exceptions, incomplete data, contradictions, regime changes, failures, different profiles and unequal consequences.
- 04Adversarial review. Adversarial review attempts to refute the claim that the system works.
An intelligent system can find its limits in many places.
The source may be out of date. Retrieval may select a similar-looking fragment. The model may misinterpret. A tool may run twice. A workflow may lose state. An interface may present a decision without context. A machine may act on an insufficient estimate.
We evaluate the complete journey.
Observability preserves the signals needed to answer afterwards for every execution. Control limits authority and makes it possible to stop, review, revert or escalate.
Evaluating backwards from the result
We start from the expected result. We then identify decisions, actions, policies, tools, models, retrieval, sources and data.
Each component has its own metrics and a relationship with the result.
A model may improve precision and increase latency. An agent may complete more cases and raise the cost. An automation may reduce steps and increase exceptions.
The evaluation of the system relates those consequences.
Layers of evaluation
- Information
- Coverage, currency, provenance, entity resolution, support, contradiction and abstention.
- Language
- Fidelity, terminology, omission, addition, naturalness, register and multi-format coherence.
- Machine learning
- Performance, calibration, temporal validation, cost of error, robustness, drift and policy.
- Agents
- Result, tool selection, policy compliance, state, cost, latency, recovery and abstention.
- Workflows
- Transition, durability, retries, events, idempotency, compensation and completion.
- Human interfaces
- Comprehension, load, corrections, time, decision quality and recurrence.
- Software
- Contracts, tests, availability, errors, security and observability.
- Physical Intelligence
- Perception, estimation, planning, control, safety, verification and recovery.
Representative evals
An evaluation set must represent the use: ordinary cases, difficult cases, edges, exceptions, incomplete data, contradictions, regime changes, failures, different profiles and unequal consequences.
Evals are versioned together with criteria, data, reference answers, reviews, severity, coverage and conditions.
Automatic metrics are complemented with human review when quality depends on meaning, judgement or context.
Adversarial review
Adversarial review attempts to refute the claim that the system works.
It asks:
- which source looks relevant and does not support the conclusion?;
- which plausible input breaks the schema?;
- which permission allows an improper action?;
- which event arrives twice?;
- which case makes an aggregate metric hide a failure?;
- which sentence is fluent and changes the meaning?;
- which technically correct action leaves the result unproduced?;
- which simulation excludes a relevant tolerance?;
- which exception would turn a rule into a bad policy?
This practice discovers conditions before operation turns them into incidents.
Simulation
Simulation makes it possible to reproduce loads, events, failures, sequences, environments, decisions and physical operations.
It can be used to test workflows, compare policies, examine recovery, study sensitivity, generate scenarios, validate control and prepare HIL.
Simulation preserves its assumptions. The result indicates behaviour within those assumptions.
Observability by trajectory
Traces are organised by case, question, decision or operation.
A trace can relate:
Each event includes identifiers, time, state and version.
Observability makes it possible to debug, explain, audit, compare, attribute, recover and optimise. Sensitive data is protected through permissions, minimisation and retention policies.
Evidence and traceability
A generated explanation can sound convincing. Traceability shows the actual journey.
We preserve what information came in, what rule was applied, what model and configuration were used, what tool acted, what person decided and what result was observed.
In informational intelligence, the claim is linked to the source. In operational intelligence, the action is linked to the state and to its effect.
Authority control
The system classifies actions by impact, reversibility, sensitivity, cost, risk and the authority they require.
Controls may include read-only, proposal, simulation, shadow mode, confirmation, authorisation, limits, dual control, bounded execution, stop and rollback.
Autonomy grows by journey and by action, supported by evidence of behaviour.
Gating and maturity
Each capability can advance through gates. For example:
The name and number can be adapted to each project.
The gate declares evidence, criteria, result, exceptions, blockers and approval authority. Maturity is communicated as state and scope.
Recovery
The system prepares a response to invalid output, an absent source, insufficient permission, tool failure, duplication, timeout, out-of-order event, inconsistency, loss of perception and ambiguous result.
Recovery may retry, change route, request information, reduce scope, compensate, escalate, stop or keep pending.
The trace records the failure and the remaining state.
Operational metrics
Beyond technical quality, we observe total time, waiting time, cost, human load, retries, referrals, reopenings, verified completions, availability, stability, incidents, adoption and improvement against the baseline.
A metric is interpreted together with the objective and the constraints.
Learning
Results may modify data, prompts, models, rules, thresholds, tools, interfaces, routes, permissions and evaluations.
The change goes through versioning and testing.
An exception is preserved with its context and analysed for recurrence, impact and similarity before becoming a rule.
Closing
Trust is built by making behaviour visible and by designing what should happen when the system finds its limits.
What evaluation answers
- 01what quality it reaches
- 02under what conditions
- 03against what baseline
- 04what error it produces
- 05how we detect it
- 06what consequence it has
- 07how it recovers
- 08what the system learns