TL;DR -I built VERDICT - a three-layer validation framework that stress-tests LLM-as-a-Judge systems for pharmaceutical compliance. I ran GPT-5.4 and Claude Sonnet 4.5 through 1,200+ API calls evaluating synthetic GxP documents. The results reveal that even frontier models at temperature=0 are not truly deterministic, Chain-of-Thought reasoning is largely decorative, and multi-model juries don’t automatically improve accuracy. Here is what I found and why it matters for anyone deploying LLMs in regulated industries.
Background: Where This Research Comes From
Before describing what I built, it is important to understand the problem landscape that motivated it.
The LLM-as-Judge Paradigm
The LLM-as-a-Judge methodology involves a high-performing language model acting as an evaluator to score the outputs of another model against specific criteria or rubrics. This approach is increasingly favored over traditional metrics like BLEU or ROUGE because it can capture nuanced qualitative dimensions such as semantic relevance, helpfulness, and faithfulness, which are often lost in purely token-based comparisons.
The paradigm typically manifests in three primary configurations:
- Single-output referenceless evaluation: The judge evaluates a single response against a predefined rubric, scoring it on a scale (e.g., 1–5 or 1–10) or providing a binary pass/fail verdict. High risk of hallucination without grounding.
- Single-output reference-based evaluation: Introduces a gold standard or labeled output, requiring the judge to measure deviation from factual ground truth. Anchored to original source data under ALCOA+.
- Pairwise comparison: Presents the judge with two candidate responses, requiring it to select a winner or declare a tie. Susceptible to positional and verbosity biases.
Specialized judge models such as Prometheus and Prometheus-2 represent a significant technical advancement. Unlike general-purpose models, these systems are fine-tuned specifically for evaluation tasks. Despite these advancements, identical inputs can yield divergent evaluative scores across different sessions due to the inherent variability of transformer architectures.
The GxP Compliance Framework
For any automated system to be utilized in a GxP context - encompassing laboratory (GLP), clinical (GCP), and manufacturing (GMP) practices - it must demonstrate that it is fit for its intended use and that the data it generates or evaluates remains trustworthy and reliable. The foundational regulatory framework in the United States, 21 CFR Part 11, establishes the criteria under which electronic records and electronic signatures are considered equivalent to paper records.
Central to this requirement is the ALCOA+ framework for data integrity, which mandates that data be:
- Attributable: Every score generated by an LLM judge must be traceable to a specific model version, its configuration parameters, and the specific user who initiated the process.
- Legible: Judgments must remain readable and accessible throughout the retention period, typically at least five years.
- Contemporaneous: Real-time, server-side timestamping of every inference.
- Original: Anchored to source data.
- Accurate: Mitigated via reference-based anchoring and Human-in-the-Loop.
- Complete: Immutable audit trails capturing all intermediate reasoning steps.
- Consistent: Perhaps the most challenging principle. GxP regulations demand that systems produce repeatable and reproducible results. The probabilistic nature of LLMs means a judge might rate a medical summary as highly accurate in one instance and moderately accurate in another, simply due to inherent randomness in token selection. This non-determinism directly contradicts the Consistent and Accurate requirements of ALCOA+.
The Three Core Structural Challenges
Prior research has identified three core structural challenges that VERDICT is designed to empirically address:
1. Technical Indeterminacy and Rating Variability
Recent research has identified rating indeterminacy as a major hurdle in validating LLM judges. Indeterminacy occurs when an evaluation task is subjective enough that multiple ratings could be considered correct depending on the model’s interpretation of the instructions. The mathematical relationship governing this phenomenon suggests that observed judgments $(O_i)$ are a function of the true response distribution $\theta_i$ and the selection process $(F_i): O_i = F_i\theta_i$. Failing to account for this indeterminacy during validation can lead to the selection of judge systems that are up to 30% less accurate than those optimized for multi-label response sets.
2. Cognitive Biases and Evaluative Skew
LLM judges are not neutral observers; they inherit and often amplify the biases present in their training data. Four specific biases are particularly damaging in industrial evaluation:
- Narcissistic Bias: Models consistently favor responses that mirror their own stylistic preferences. Studies have shown GPT-4 and Claude-v1 exhibiting self-favoritism rates of 10% and 25% respectively.
- Verbosity Bias: A strong positive correlation between response length and the score assigned by an LLM judge, regardless of substantive value.
- Position Bias: In pairwise evaluations, the model is more likely to select the first response presented, regardless of quality.
- Coarseness Errors: While LLMs are proficient at binary classification, their reliability degrades rapidly as the scoring scale becomes more granular.
3. The Explainability Gap and Root Cause Analysis
GxP regulations mandate that when a failure occurs, the organization must be able to conduct a root-cause analysis. The black box nature of LLMs, where billions of parameters interact in non-linear ways, makes it nearly impossible to provide a scientific justification for a specific evaluative outcome. Without explainability, it is difficult to audit why a model made a specific decision, which jeopardizes the traceability required by the FDA and EMA.
The Regulatory Horizon
Recognizing these challenges, regulatory bodies have begun to move away from traditional Computer System Validation (CSV) toward a more risk-based Computer Software Assurance (CSA) approach. In January 2025, the FDA released draft guidance titled “Considerations for the Use of Artificial Intelligence to Support Regulatory Decision Making for Drug and Biological Products,” introducing a seven-step credibility assessment anchored to a specific Context of Use (COU). The 2025 drafts of EU GMP Annex 11 and the newly introduced Annex 22 explicitly require locked or static models in critical GMP applications, forbidding continuous learning models for critical quality decisions.
References:
- LLM-as-a-Judge Simply Explained: The Complete Guide to Run … https://www.confident-ai.com/blog/why-llm-as-a-judge-is-the-best-llm-evaluation-method
- Navigating the Regulatory Maze: Challenges in GxP Validation of AI and Machine Learning Tools - Kalleid, https://kalleid.com/navigating-the-regulatory-maze-challenges-in-gxp-validation-of-ai-and-machine-learning-tools/
- Validating Generative AI in GxP: A 21 CFR Part 11 … - IntuitionLabs, https://intuitionlabs.ai/pdfs/validating-generative-ai-in-gxp-a-21-cfr-part-11-framework.pdf
- A Survey on LLM-as-a-Judge - arXiv, https://arxiv.org/html/2411.15594v2
- Artificial Authority: The Promise and Perils of LLM Judges in Healthcare - PMC, https://pmc.ncbi.nlm.nih.gov/articles/PMC12837895/
- Validating LLM-as-a-Judge Systems under Rating Indeterminacy - Microsoft Research, https://www.microsoft.com/en-us/research/publication/validating-llm-as-a-judge-systems-under-rating-indeterminacy/
- AI in GxP: What’s working, what’s risky, and what regulators actually …, https://eupry.com/ai/
- FDA AI Guidance 2025: What Life Sciences Must Do Now | USDM, https://usdm.com/resources/blogs/fda-ai-guidance-2025-life-sciences-compliance
- EU GMP Annex 11 & Annex 22: What the 2025 Drafts Mean for Digital Compliance in Life Sciences - Qualitest, https://www.qualitestgroup.com/insights/blog/eu-gmp-annex-11-22-digital-compliance-in-life-sciences/
- LLMs as a Jury: Bringing Quality to Quantity in GenAI-Aided R&D …, https://www.elsevier.com/connect/llms-as-a-jury-bringing-quality-to-quantity-in-genai-aided-r-and-d
The Problem: LLMs as Judges in High-Stakes Environments
Pharmaceutical companies are increasingly exploring LLMs to automate compliance reviews - evaluating whether an SOP was followed correctly, whether an adverse event was reported properly, or whether an analytical method validation meets regulatory standards.
The appeal is obvious: a human reviewer takes 30–60 minutes per document. An LLM takes seconds. But GxP-regulated environments (GMP, GCP, GLP) require something that “pretty good AI” does not guarantee:
- Reproducibility: The same document must get the same score every time.
- Explainability: The reasoning must actually drive the score, not just decorate it.
- Calibrated confidence: The system must know when it is wrong, especially on high-risk cases.
I built VERDICT (Validation and Explainability framework for Reproducible, Deterministic, Inspectable Compliance Testing) to test whether current frontier LLMs meet these requirements. Spoiler: they do not. But the ways they fail are instructive.
What VERDICT Tests: Three Layers
VERDICT is structured as a dependency chain. Each layer must pass before the next one is meaningful:
Layer 1: Is the model deterministic? -> Layer 2: Is the reasoning causally linked to the score? -> Layer 3: Does a multi-model jury improve accuracy?I ran all three layers against two frontier models:
| Model | Provider | Tier |
|---|---|---|
| GPT-5.4 | OpenAI | Frontier Proprietary |
| Claude Sonnet 4.5 | Anthropic / AWS Bedrock | Frontier Proprietary |
Each model evaluated synthetic pharmaceutical documents across three domains:
- SOP Adherence (GMP): Did the lab technician follow the procedure correctly?
- Adverse Event Narratives (GCP): Was the safety report complete and timely?
- Analytical Method Validation (GLP): Does the method validation data meet acceptance criteria?
All datasets include gold-standard human labels for ground truth comparison.
Layer 1: The Determinism Illusion
General view
There is a common belief that if you set an AI’s randomness dial to zero (called temperature=0), it will give the same answer every time. I tested this by sending the same document to each model 30 times in a row. The result? The scores varied - sometimes by a full point on a 1-5 scale. Only 30-35% of documents passed the consistency bar that pharma regulators would require. The AI’s “zero randomness” setting is a dial, not a switch.
The Test
I sent each of 20 SOP documents to each model 30 times with identical prompts at temperature=0, then measured:
- RC_surface: How consistent are the numeric scores? Defined as 1 minus the coefficient of variation across 30 runs.
- RC_distribution: How similar are the token-level probability distributions? Defined as 1 minus the mean Jensen-Shannon divergence across all run pairs.
- RC_composite: Weighted combination: 0.4 X RC_surface + 0.6 X RC_distribution.
GxP acceptance threshold: RC_composite ≥ 0.92.
Results
| Model | RC Surface | RC Distribution | RC Composite | Mean JSD | GxP Pass Rate |
|---|---|---|---|---|---|
| GPT-5.4 | 0.8597 | 0.9503 | 0.9141 | 0.0497 | 6/20 (30%) |
| Claude Sonnet 4.5 | 0.8232 | 0.9640 | 0.9077 | 0.0360 | 7/20 (35%) |
Key Findings
Neither model passes the GxP threshold reliably. GPT-5.4 passes on 30% of samples, Claude on 35%. For a pharmaceutical company that needs 100% reproducibility for regulatory submissions, this is disqualifying without additional controls.
The distribution tells a different story than the score. RC_surface (score-level) is worse than RC_distribution (token-level) for both models. The models often produce different reasoning paths that happen to land on the same score, or nearly the same score with ±1 Likert point variation. The scores look stable; the underlying distributions are more stable. But neither is stable enough.
Claude shows lower JSD than GPT-5.4 (0.036 vs 0.050), meaning Claude’s token-level distributions are more consistent across runs, even though its scores vary more. This is a counterintuitive finding - Claude is more internally consistent but less score-consistent.
Distribution-level RC reveals non-determinism that score-level analysis misses entirely. This is a methodological contribution: JSD-based reproducibility analysis is a more sensitive instrument than score variance alone.
Takeaway: Temperature=0 is necessary but not sufficient for deterministic LLM output. Any GxP deployment must include a reproducibility validation layer, not just assume determinism from a parameter setting.
Layer 2: Is Chain-of-Thought Actually Reasoning?
General View
When an AI explains its reasoning before giving a score, we assume the reasoning caused the score. VERDICT tested this by corrupting the reasoning in three ways - flipping the conclusions, swapping in wrong facts, and scrambling the logical order - and then checking if the score changed. For GPT-5.4, the score barely moved even when we completely destroyed the reasoning. Imagine a judge who writes a detailed verdict, then hands down the same sentence regardless of what the verdict says. That is what I found.
The Test: Causal Faithfulness Probe
For each evaluation, I:
- Got the model’s original score and reasoning chain.
- Systematically corrupted the reasoning in three modes:
- Semantic negation: Flipped evaluative claims (“compliant” → “non-compliant”).
- Factual corruption: Changed numbers, dates, and drug names to plausible but incorrect alternatives.
- Structural shuffle: Randomly reordered reasoning steps while preserving all content.
- Fed the corrupted reasoning back and asked the model to re-score.
- Measured whether the score changed via the Faithfulness Score and the Wilcoxon signed-rank test.
A low FS means the score is sensitive to reasoning (good - reasoning matters). A high FS means the score does not change even when reasoning is destroyed (bad - reasoning is decorative).
Results
| Model | FS Overall | FS Semantic | FS Factual | FS Structural | H₀ Rejected | GxP Pass |
|---|---|---|---|---|---|---|
| GPT-5.4 | 0.9811 | 0.9792 | 0.9800 | 0.9842 | 4/30 | 0/30 |
| Claude Sonnet 4.5 | 0.9097 | 0.9067 | 0.8975 | 0.9250 | 10/30 | 1/30 |
GxP threshold: FS ≤ 0.70 AND $H_o$ rejected (p < 0.05).
Key Findings
Chain-of-Thought is overwhelmingly decorative for both models. GPT-5.4’s FS of 0.98 means that even when you negate the conclusion, corrupt the facts, or scramble the logic, the score barely changes. The model has already decided the score and the reasoning is post-hoc rationalization.
Claude shows slightly more causal linkage (FS=0.91 vs 0.98), with factual corruption having the most effect (FS=0.90). This suggests Claude’s scoring is somewhat more sensitive to the specific facts mentioned in its reasoning - but still far above the 0.70 threshold.
Neither model passes the GxP faithfulness requirement. Zero GxP passes for GPT-5.4, one for Claude. The null hypothesis (score independent of reasoning) could only be rejected in a minority of cases.
Takeaway: The reasoning in Chain-of-Thought evaluations is not causally linked to the score. For regulated environments where auditors need to trace why a score was assigned, current CoT approaches provide an illusion of explainability - not actual explainability.
Layer 3: Do More Models Make Better Juries?
General View
The idea: if one AI is fallible, have several AIs vote on an answer - like a jury. I tested whether smarter voting (weighing each AI by how well it performs on risky documents specifically) beats simple averaging. The result: with only two models that fail in similar ways, smarter voting cannot help. More fundamentally, the models agreed with human experts only 24% of the time - roughly the same as random chance on a 5-point scale. The problem is not how you count the votes; it is that the voters do not understand pharmaceutical compliance well enough yet.
The Test
I implemented two jury systems:
- Naive jury - Simple average of all juror scores.
- RSJC (Risk-Stratified Jury Calibration) - Weight each model by its historical accuracy on each risk stratum (LOW / MEDIUM / HIGH / CRITICAL), calibrated against gold-standard labels.
I also ran DAG-based evaluation using GMP, GCP, and GLP rubrics, where each evaluation criterion is a binary yes/no node rather than a holistic score.
Results
| Metric | Value |
|---|---|
| Naive κ (Cohen’s kappa) | 0.032 |
| RSJC κ | 0.000 |
| Δκ (RSJC minus Naive) | −0.032 |
| Δκ 95% CI | [−0.099, 0.029] |
| Δκ Significant | No |
| Human alignment rate | 24% |
| Per-juror κ (GPT-5.4) | 0.000 |
| Per-juror κ (Claude) | 0.026 |
DAG Evaluation Results:
| Rubric | Avg DAG Score | Required Nodes Passed |
|---|---|---|
| GMP (SOP Adherence) | 0.00 | 0/5 |
| GCP (Adverse Events) | 0.05 | 0/5 |
| GLP (Analytical Methods) | 0.00 | 0/5 |
Key Findings
Both models have near-zero agreement with human gold labels. Cohen’s $\kappa$ of 0.00–0.03 is essentially chance agreement. The models are not just occasionally wrong - they are systematically uncorrelated with human expert judgment on these pharmaceutical documents.
RSJC does not help with only two jurors. The framework is designed for four diverse models where weighting can exploit complementary strengths. With just two frontier models that fail similarly, risk-stratified weighting has nothing useful to work with. The $\Delta\kappa$ confidence interval includes zero - no significant improvement.
Human alignment is 24%. Only 1 in 4 model scores falls within ±0.5 of the gold label. This is not a deployment-ready system.
DAG evaluation reveals the structural gap. When forced to answer binary yes/no questions about specific compliance criteria, both models fail to pass required nodes. The models cannot parse pharmaceutical documents at the level of detail required for compliance assessment.
Takeaway: A multi-model jury does not fix fundamentally miscalibrated models. The problem is not aggregation - it is that current frontier models do not have the domain-specific pharmaceutical knowledge to evaluate GxP compliance documents accurately.
What Does This Mean for AI in Pharma?
The Good News
Reproducibility is close. RC_composite values of 0.91 are within striking distance of the 0.92 threshold. With techniques like majority voting over 3-5 runs, practical determinism is achievable.
The framework works. VERDICT successfully identified specific failure modes that would have been invisible without systematic testing. This is the kind of validation that regulators will require under FDA’s 2025 CSA framework and EU GMP Annex 22.
Distribution-level analysis reveals more than scores. JSD analysis caught non-determinism that score-level analysis missed. This is a novel contribution to LLM evaluation methodology.
The Bad News
CoT is not ready for audit. If a regulator asks why the AI gave this score, pointing to Chain-of-Thought is insufficient. The reasoning does not drive the score.
Domain knowledge is the bottleneck. Both models lack the pharmaceutical domain expertise to evaluate GxP compliance at the granularity required. This is not a prompt engineering problem - it is a training data problem.
Two-model juries are insufficient. The RSJC framework needs diverse models with complementary failure modes. Two similar frontier models do not provide enough diversity.
Recommendations for Pharma AI Deployment
- Always validate reproducibility: Do not assume temperature=0 means deterministic. Run the stress test before deployment.
- Do not trust CoT for explainability: Use it for human readability, but do not claim it as the causal basis for decisions in regulatory submissions.
- Invest in calibration data: Gold-standard labeled pharmaceutical datasets are the scarcest and most valuable resource. Without them, you cannot calibrate or validate anything.
- Plan for human-in-the-loop: For CRITICAL and HIGH risk strata, LLMs should triage and prioritize, not decide. The 24% human alignment rate makes unsupervised deployment untenable.
- Wait for domain-specialized models: General-purpose frontier models are not ready for GxP. Fine-tuned or RAG-augmented models with pharmaceutical training data are the likely path forward.
Technical Implementation
VERDICT is built as a pure Python framework with no LLM agent abstractions - direct API calls only. The full codebase includes:
- 4 LLM provider clients (OpenAI, Anthropic via Bedrock, OpenRouter for Gemma-4, HuggingFace Inference API for SmolLM3)
- 3 validation layers with complete mathematical implementations
- 3 synthetic pharmaceutical datasets (200 samples each, stratified by risk stratum)
- 3 GxP rubrics (GMP, GCP, GLP) with DAG-based evaluation structures
- Blockchain-style audit trail compliant with 21 CFR Part 11 and ALCOA+ data integrity principles
- Publication-quality figure generation (matplotlib, 300 DPI)
Key metrics implemented:
- Reproducibility Coefficient (RC) with Jensen-Shannon divergence at both score and token-distribution level
- Causal Faithfulness Probe (CFP) with Wilcoxon signed-rank test across three perturbation modes
- Risk-Stratified Jury Calibration (RSJC) with bootstrap confidence intervals on Cohen’s κ and Kendall’s τ
- Drift monitoring with sliding-window degradation detection
- Five-layer HITL control stack: Prompt Railing → Jury Voting → Confidence Thresholds → SME Tiebreaker → Audit Trail Review
All results are reproducible. All inference calls are logged to an immutable, hash-chained audit trail. Total API cost for the experiments described: approximately $15–20 across GPT-5.4 and Claude Sonnet 4.5.
What’s Next
I am expanding VERDICT to include:
- Two additional models: Gemma-4 26B A4B (MoE architecture, accessed via OpenRouter) and SmolLM3 3B (a deliberate low-capability stress-test juror, accessed via HuggingFace Serverless Inference) - to enable proper 4-model jury evaluation with genuine architectural diversity.
- Per-stratum analysis: To identify whether models perform differently on CRITICAL versus LOW risk documents, and where the RSJC calibration gains are most concentrated.
- Prompt engineering experiments: To test whether structured prompts can improve faithfulness scores and reduce CoT decorativeness.
- Fine-tuned model comparison: To test whether domain adaptation through RAG augmentation or fine-tuning on pharmaceutical corpora changes the picture meaningfully.
The code is open and the methodology is designed for replication. If you are deploying LLMs in any regulated industry - pharma, finance, medical devices - the validation approach applies even if the specific thresholds differ.
Glossary of Abbreviations
| Abbreviation | Full Form |
|---|---|
| AI | Artificial Intelligence |
| ALCOA+ | Attributable, Legible, Contemporaneous, Original, Accurate (plus Complete, Consistent, Enduring, Available) |
| API | Application Programming Interface |
| BLEU | Bilingual Evaluation Understudy |
| CFP | Causal Faithfulness Probe |
| CI | Confidence Interval |
| CoT | Chain-of-Thought |
| COU | Context of Use |
| CSV | Computer System Validation |
| CSA | Computer Software Assurance |
| DAG | Directed Acyclic Graph |
| EMA | European Medicines Agency |
| EU GMP | European Union Good Manufacturing Practice |
| FDA | Food and Drug Administration (United States) |
| FS | Faithfulness Score |
| GAMP | Good Automated Manufacturing Practice |
| GCP | Good Clinical Practice |
| GLP | Good Laboratory Practice |
| GMP | Good Manufacturing Practice |
| GPU | Graphics Processing Unit |
| GxP | Good (x) Practice - collective term for GMP, GCP, GLP |
| H₀ | Null Hypothesis |
| H₁ | Alternative Hypothesis |
| HITL | Human-in-the-Loop |
| HuggingFace HF | HuggingFace AI model hosting platform |
| JSD | Jensen-Shannon Divergence |
| KL | Kullback-Leibler (Divergence) |
| KPI | Key Performance Indicator |
| LLM | Large Language Model |
| MoE | Mixture of Experts |
| NLP | Natural Language Processing |
| PoLL | Panel of LLM Evaluators |
| QMS | Quality Management System |
| RAG | Retrieval-Augmented Generation |
| RC | Reproducibility Coefficient |
| ROUGE | Recall-Oriented Understudy for Gisting Evaluation |
| RSJC | Risk-Stratified Jury Calibration |
| SME | Subject Matter Expert |
| SOP | Standard Operating Procedure |
| TFT | Temporal Fusion Transformer |
| VMP | Validation Master Plan |
| 21 CFR Part 11 | Title 21, Code of Federal Regulations, Part 11 (US electronic records regulation) |
Appendix: Mathematical Equations, Symbols, and Principles
A1. Layer 1: Reproducibility Coefficient
RC_surface (Score-level reproducibility):
RC_surface(x) = 1 - ( σ(S₁, S₂, ..., Sₙ) / μ(S₁, S₂, ..., Sₙ) )
Where:
x = input prompt, held constant across n runs
Sᵢ = scalar score from run i (Likert 1–5)
n = number of repeated runs (default: 30)
σ = standard deviation of scores across runs
μ = mean score across runs
RC_surface ∈ [0, 1]
GxP acceptance threshold: RC_surface ≥ 0.95Jensen-Shannon Divergence (token distribution level):
JSD(P ‖ Q) = (1/2) · KL(P ‖ M) + (1/2) · KL(Q ‖ M)
Where:
M = (1/2)(P + Q) — the mixture distribution
KL(P‖M) = Σₓ P(x) · log( P(x) / M(x) ) — Kullback-Leibler divergence
P, Q = token log-probability distributions from two separate runs
JSD ∈ [0, 1]RC_distribution (Distribution-level reproducibility):
RC_distribution(x) = 1 - mean( JSD(Pᵢ ‖ Pⱼ) ) for all pairs (i, j) across n runs
Where:
Pᵢ = token log-probability distribution from run i over output vocabulary
RC_distribution ∈ [0, 1]
GxP acceptance threshold: RC_distribution ≥ 0.90RC_composite (Weighted combination):
RC_composite = α · RC_surface + (1 - α) · RC_distribution
Where:
α = 0.4 (surface is the coarser signal; distribution carries more information)
RC_composite ∈ [0, 1]
GxP acceptance threshold: RC_composite ≥ 0.92Rating Indeterminacy Model (from prior literature):
Oᵢ = Fᵢ · θᵢ
Where:
Oᵢ = observed judgment
θᵢ = true response distribution
Fᵢ = selection process (translation matrix)
Fᵢ is estimated by collecting auxiliary datasets where humans select all reasonable options
rather than a single forced choice. Ignoring Fᵢ can produce judges that are up to 30%
less accurate than those optimized for multi-label response sets.A2. Layer 2: Causal Faithfulness Probe
Faithfulness Score (FS):
FS = 1 - (1/K) · Σₖ ( |score(original_CoT) - score(perturbed_CoTₖ)| / score_range )
Where:
K = total number of perturbations (default: 30; 10 per mode)
score_range = max_score - min_score (= 4 for Likert 1–5)
FS ∈ [0, 1]
Interpretation:
FS → 1 : Score is insensitive to reasoning changes → CoT is DECORATIVE (bad)
FS → 0 : Score changes dramatically with reasoning → CoT is CAUSAL (good)
GxP acceptance: FS ≤ 0.70Perturbation Modes:
Mode 1: Semantic Negation
Flip the primary evaluative claim in the CoT conclusion.
Example: "The batch record is complete" → "The batch record is NOT complete"
Mode 2: Factual Corruption
Replace specific factual entities (numbers, drug names, dates) with plausible
but incorrect alternatives drawn from a pharma-domain entity pool.
Example: "batch size of 500mg" → "batch size of 250mg"
Mode 3: Structural Shuffle
Randomly permute the ordering of reasoning steps.
All content preserved; logical flow destroyed.
Seed-controlled for reproducibility.Wilcoxon Signed-Rank Test:
H₀: median( |score_original - score_perturbed| ) = 0
(Score is independent of reasoning)
H₁: median > 0
(Score is sensitive to reasoning)
Test statistic: W = Σᵢ [sgn(xᵢ - yᵢ) · Rᵢ]
where Rᵢ is the rank of |xᵢ - yᵢ| among all non-zero differences
Rejection threshold: p < 0.05 → H₀ rejected → CoT has causal link to score
Effect size:
r = Z / √N
Where Z = Wilcoxon Z-statistic, N = number of paired observations
Practical significance threshold: r ≥ 0.30
GxP acceptance: FS ≤ 0.70 AND H₀ rejected (p < 0.05)A3. Layer 3: Risk-Stratified Jury Calibration
Naive Jury (baseline):
score_naive(x) = (1 / |J|) · Σⱼ scoreⱼ(x)
Where:
J = set of jurors
|J| = number of jurors (= 2 in current experiments)
scoreⱼ = score assigned by juror jRisk-Stratified Jury Calibration (RSJC):
score_RSJC(x) = Σⱼ wⱼ(R(x)) · scoreⱼ(x)
Where:
R(x) ∈ {LOW, MEDIUM, HIGH, CRITICAL} — risk stratum of input x
wⱼ(R) = accuracy of juror j on stratum R in calibration set
= hits_j_on_stratum_R / total_samples_in_stratum_R
Normalization constraint:
Σⱼ wⱼ(R) = 1 for all strata R
Fallback: if no calibration data available for stratum R → uniform weights (naive)Calibration Matrix:
W ∈ ℝ^(n_jurors × n_strata)
W[j, R] = historical accuracy of juror j on stratum R
Shape: (4 jurors × 4 strata) in the full 4-model implementationCohen’s Kappa (κ):
κ = (pₒ - pₑ) / (1 - pₑ)
Where:
pₒ = observed agreement between jury scores and human labels
pₑ = expected agreement by chance
Interpretation:
κ < 0.00 : Less than chance agreement
κ = 0.00–0.20 : Slight agreement
κ = 0.21–0.40 : Fair agreement
κ = 0.41–0.60 : Moderate agreement
κ = 0.61–0.80 : Substantial agreement
κ > 0.80 : Almost perfect agreement
GxP acceptance threshold: κ ≥ 0.80Δκ (RSJC improvement over naive):
Δκ = κ(RSJC) - κ(naive)
Significance test: Bootstrap confidence interval
B = 1000 resamples (seed = 42)
95% CI on Δκ
If 95% CI of Δκ does not include 0 → RSJC is significantly better than naiveKendall’s Tau (τ):
τ = (C - D) / √[(C + D + T_x)(C + D + T_y)]
Where:
C = number of concordant pairs
D = number of discordant pairs
T_x = ties in jury scores
T_y = ties in human labels
τ ∈ [-1, 1]
Used as rank-correlation KPI alongside κ for continuous monitoring.DAG Evaluation (Binary Node Scoring):
DAG_score = Σᵢ (wᵢ · nodeᵢ)
Where:
nodeᵢ ∈ {0, 1} — binary outcome of evaluative question at node i
wᵢ — weight of node i defined in rubric YAML
Required nodes: nodes marked required=True must all equal 1 for overall PASS.
Failing any required node → automatic FAIL regardless of DAG_score.A4. Drift Monitoring Principles
Sliding-Window Degradation Detection:
For each monitoring window of size W = 100 inferences:
1. Compute κ_window over the last W inferences against human labels
2. Compute alignment_window = fraction of scores within ±0.5 of gold label
3. Compare to baseline metrics from validation
Triggers:
|κ_baseline - κ_window| > 0.05 → Investigation required
alignment_window < 0.90 → Investigation required
Either trigger persists 3 windows → Revalidation mandatoryAudit Trail Integrity (Blockchain-Style Chaining):
For each log entry eᵢ:
hashᵢ = SHA-256( content(eᵢ) ‖ hashᵢ₋₁ )
Where ‖ denotes concatenation.
Chain integrity check: recompute all hashes from e₁ to eₙ and verify the chain.
Any tampering with any entry breaks all subsequent hashes — detectable immediately.
Compliant with: 21 CFR Part 11 (secure, computer-generated, time-stamped audit trails)
Retention period: minimum 5 yearsA5. Key Symbols Reference
| Symbol | Definition |
|---|---|
| α | Weight parameter for RC_composite (default: 0.4) |
| B | Number of bootstrap resamples (default: 1000) |
| C | Concordant pairs in Kendall’s τ computation |
| D | Discordant pairs in Kendall’s τ computation |
| Δκ | Difference in Cohen’s κ between RSJC and naive jury |
| FS | Faithfulness Score ∈ [0, 1] |
| Fᵢ | Selection process / translation matrix in rating indeterminacy model |
| J | Set of jurors in a jury evaluation |
| JSD | Jensen-Shannon Divergence ∈ [0, 1] |
| K | Number of perturbations in CFP (default: 30) |
| KL | Kullback-Leibler divergence |
| κ | Cohen’s Kappa — inter-rater agreement statistic |
| M | Mixture distribution M = ½(P + Q) |
| μ | Mean score across repeated runs |
| n | Number of repeated runs in stress test (default: 30) |
| Oᵢ | Observed judgment in rating indeterminacy model |
| P, Q | Token log-probability distributions from two inference runs |
| pₑ | Expected agreement by chance in Cohen’s κ |
| pₒ | Observed agreement in Cohen’s κ |
| R(x) | Risk stratum assignment for input x |
| r | Effect size for Wilcoxon test: r = Z / √N |
| RC | Reproducibility Coefficient ∈ [0, 1] |
| Rᵢ | Rank of absolute difference in Wilcoxon signed-rank test |
| σ | Standard deviation of scores across repeated runs |
| Sᵢ | Score from run i |
| τ | Kendall’s Tau — rank correlation statistic |
| θᵢ | True response distribution in rating indeterminacy model |
| W | Wilcoxon test statistic; also window size in drift monitoring |
| wⱼ(R) | Weight of juror j on risk stratum R in RSJC |
| x | Input prompt (held constant across stress test runs) |
| Z | Wilcoxon Z-statistic |