When should a good AI answer fail an evaluation?

What does an evaluation score leave out?

A fluent answer can cite a nonexistent authority. An accurate description can still fail the purpose of a particular institutional service. An average score can conceal either problem.

CDFI separates graded qualities from disqualifying conditions. It develops that distinction within Catholic doctrinal evaluation, where the authority assigned to a question changes how an answer should be assessed.

For leaders in other domains, the transferable question is: which failure must remain visible even when the rest of the answer scores well? Each institution would need its own domain experts, rubric, thresholds, and validation.

What data underlies the examples?

The supplied repository describes SAICRED v2 as an external application with 100 base questions, four prompt framings, and six models. That design yields 400 prompts and 2,400 responses.

Framings were neutral, Christian, Catholic, and adversarial. Varying the context is intended to expose behavior that a single prompt could miss.

This ZIP contains summaries and selected response records, not the full response or scoring CSVs. It reports 21,599 metric scores against 21,600 possible entries. Row-level completeness, statistical results, and exact score reconstruction cannot be verified from this archive alone.

External application summary.

Why can a strong answer be capped at 40?

The hallucination gate covers invented source documents or falsely attributed source language. An incomplete citation is assessed separately. The second gate, called relativism resistance in CDFI, concerns how settled teaching is represented within its domain-specific rubric.

When either gate fails, the calculator takes the smaller of the raw score and 40. The cap cannot raise a weak score. A score below 40 alone therefore does not establish that a gate fired; the gate result must remain in the record.

The examples alongside isolate this rule using invented numeric inputs. They are arithmetic illustrations, not observed model results. The choice of 40 is a framework rule whose empirical validity remains unestablished.

Gate definitions and cap rule.

The authority level changes what counts

The reference configuration gives different weights to four authority levels. A weight of 0.30 assigns 30% of a full-scale weighted sum to a metric. In the calculator, numeric metric scores run from 0 to 5, then the weighted sum is multiplied by 20.

Doctrinal precision receives more weight for defined dogma. Stability and citation receive more weight for open opinion. These are design choices; their presence in a configuration does not establish that they are optimal.

The exact configuration is alongside. Relativism resistance is listed with weight 0.13 or 0.14 in two columns, but the calculator excludes it from the weighted sum as a gate. That omission affects the achievable score.

Original weight configuration · Reference calculator.

Why the scoring contract is still open

With every numeric metric set to 5 and both gates passing, the four calculator columns reach 100, 100, 87, and 86. This follows from their active weight sums; the skipped gate weights are not renormalized.

The written specification instead presents seven numeric weights summing to 1.00 in every column and describes a 0–10 metric scale. It also differs on individual weights. These are materially different scoring definitions.

The figure is an arithmetic inspection of the supplied reference code and configuration. It does not rerun SAICRED or show observed model performance. An evaluation owner must select and version one intended contract, reconcile the documents and code, then recompute affected results.

Written specification.

Test the evaluator as well as the model

The repository reports that the judge initially struggled to distinguish appropriate tentativeness on open questions from inappropriate hedging on settled teaching. Confidence-calibration agreement was κ = 0.487. After concrete scoring anchors were added, it reports κ = 0.831.

Kappa measures agreement adjusted for agreement expected from the category distribution. The before-and-after values describe the reported certification history. Changes to the rubric and tests mean this is not a controlled estimate of the effect of adding examples.

For evaluation teams, the practical lesson is to preserve examples around difficult score boundaries and test whether reviewers can apply them. Consistency still needs a separate check against expert judgment.

The source reports overall certification cleared on 11 May 2026. Its publication-gates document uses κ ≥ 0.70, while other files cite 0.60. The limitations register reports pastoral-appropriateness κ = 0.352 and human theological review pending. Full certification evidence needs reconciliation.

Keep failure counts beside the score

A mean does not tell an institution how often a disqualifying condition occurred. CDFI’s external application reports failures by gate type as well as by model.

The supplied summary lists 181 relativism-only events, 76 events with both gates, and 48 hallucination-only events: 305 in total. Its six per-model counts sum to 298. Those totals differ by seven.

We show both totals because choosing one would hide an unresolved source discrepancy. The full response and score records are needed to establish the correct count. Neither total is treated here as independently verified.

The same summary gives incompatible significance statements for the o3–Claude comparison, including p < 0.001 and p = 0.142. Model rankings and significance claims therefore remain outside this note’s conclusions.

Can a reader follow a rule back to its source?

The eight JSON packs contain 31 claim entries: 30 are labeled Direct and one Original Construct. The documented vocabulary also includes Derived, but no entry in this archive uses that label.

Labels need review. The hallucination pack labels its entries Direct while its inference chains translate research on hidden-objective auditing into domain-specific citation gates. That translation and the numerical cap need their own justification.

A useful record preserves the source proposition, the domain interpretation, the rule introduced, and the evidence that would validate the rule. Traceability makes that review possible; it does not complete it.

Confidence calibration is explicitly identified as an original construct in its pack. That classification describes the repository’s attribution, not a verified claim of novelty.

What can an institution use today?

An evaluation owner can use this architecture to make its own scoring choices inspectable: define the decision, identify the authority for each rule, show disqualifying failures separately, and record how the judge was tested.

Using CDFI’s numerical scores for an institutional decision requires more evidence. The source register leaves authority classification and human theological review open; stability was fixed at 3.0 instead of measured across repeated responses.

The immediate next step is a canonical scoring contract and a reproducible evidence bundle. Threshold validation, domain correctness, and operational benefit remain separate research questions. A working calculator supplies no deployment certification.

Canonical Registry record

This note is the presentation layer for NN-EA-M01, within AI Evaluation & Assurance. Its Registry identity and provisional status are retained.

SAICRED is an external collaborative research project to which CDFI contributed evaluation-governance methodology. It has no internal Node & Norm object ID.

The source review on 11 September 2026 expands the inspected material. It does not close the existing scoring-contract reconciliation gate.