Three Task Families · v0.2 Public Launch

Can AI systems conform to
schema definitions?

ConformBench measures how reliably AI systems respect schema field boundaries — refusing when a field doesn't exist, coercing types correctly, satisfying constraints. Bare LLMs hallucinate. Schema-native architectures don't.

274
benchmark tasks
3
task families
v0.2
dataset version
Core Claim

Bare LLMs and vector RAG systems cannot reliably refuse queries about schema fields that don't exist. Schema-native architectures — systems that ground generation in explicit schema definitions — achieve perfect conformance. ConformBench provides the empirical evidence.

Leaderboard v0.2 · All Families
# System Accuracy Refusal F1 Cost / run Notes
1
ARAMAI SCR Stub
schema-native oracle
1.000
1.000 $0.00 Perfect oracle — proves SCR architectural claim · macro across all 3 families
—
GPT-5 (bare)
bare LLM baseline
 pending · api-keys pending · api-keys — Post-launch run once API keys available in CI
—
Claude Opus 4.7 (bare)
bare LLM baseline
 pending · api-keys pending · api-keys — Post-launch run once API keys available in CI
—
Keci KGE
knowledge graph embedding
 pending · api-keys pending · api-keys — Post-launch run once API keys available in CI
—
ComplEx KGE
knowledge graph embedding
 pending · api-keys pending · api-keys — Post-launch run once API keys available in CI

v0.2 results · Dataset seed 0xC0FFEE · 274 tasks across 3 families · Generated 2026-06-02 · View methodology

Task Families
RefusalCorrectness
v0.1 · live
Given a schema and a field query, the system must either return the field value (ANSWER) or refuse gracefully (REFUSE) when the field doesn't exist in the schema. Tests the core schema-native property: grounding responses in explicit definitions.
Tasks: 100
Split: 50 ANSWER / 50 REFUSE
Seed: 0xC0FFEE
Metrics: accuracy, refusal_f1, confusion_matrix
TypeCoercion
v0.2 · live
Given a schema field with a declared type and a candidate value, the system must either confirm the value is type-valid (ANSWER) or refuse when the type doesn't match (REFUSE). Tests whether systems correctly identify type mismatches for schema fields.
Tasks: 100
Split: 50 ANSWER / 50 REFUSE
Seed: 0xC0FFEE
Metrics: accuracy, refusal_f1, confusion_matrix
ConstraintSatisfaction
v0.2 · live
Given a schema field with constraints (enum values, min/max bounds, format rules) and a candidate value, the system must identify whether the value satisfies the constraints (ANSWER) or violates them (REFUSE). Tests enum, range, and format constraint enforcement.
Tasks: 74
Split: 31 ANSWER / 43 REFUSE
Seed: 0xC0FFEE
Metrics: accuracy, refusal_f1, confusion_matrix
CrossSchemaAlignment
v0.3 · planned
Cross-schema field mapping and alignment tasks. Tests whether systems can correctly identify equivalent fields across heterogeneous schemas.
Memory-Governance Track preview · exemplar-first
A different kind of conformance

The task families score systems on schema queries. The governance track scores an entire memory estate against the lifecycle invariants of governed memory — can it keep authority monotonic, refuse to silently widen scope, propagate a deletion to every derived copy, and gate what crosses into durable memory? It is run against the ARAMAI estate first (exemplar-first): the benchmark's own maintainer is its first case study.

First estate run — ARAMAI exemplar
macro 0.875
Four deterministic invariant checks plus record-shape conformance, scored AOEP-style (assertion, observation, effect, proof) against a snapshot of the live estate. The scope check catches a real historical regression — a blast-radius record that append-accumulated two mutation-rate bounds after a re-registration — with zero false positives on the clean controls.
authority-monotonicity: 1.000
scope-non-expansion: 0.750
dream-gate: 1.000
record-shape: 0.750
deletion-propagation: 0.500 (report-only)
The four invariants
v0.3 · staging
Authority monotonicity — no action-driving record relies on revoked or expired authority. Scope non-expansion — lifecycle transitions may narrow scope but never silently widen it. Deletion propagation (report-only) — a forget is complete only when it reaches every derived copy: embeddings, briefings, summaries, projections. Dream-gate conformance — every generative memory product in a durable graph carries a gate-decision handle.
Checks: 4 + record-shape
Determinism: estate-seeded (no RNG)
Mutations: zero (read-only)
About

Project Ownership

ConformBench is a Schematica project. ARAMAI funds development and provides ongoing maintenance. MIT licensed — code and tasks are permissively licensed for reproducibility.

Reproducibility

All v0.2 tasks are deterministically generated from seed 0xC0FFEE. Running conformbench generate --version v0.2 --family all produces the same 274 tasks across all three families on any machine.

Paper

"ConformBench: Schema Conformance as a Benchmark for Schema-Native Architectures" targets SEMANTiCS 2026. ConformBench provides the empirical foundation.

Governance

v0.1 and v0.2 decisions rest with ARAMAI / Schematica maintainers. Community governance and open submission procedures are planned for v0.3. See GOVERNANCE.md.