Caustic

A hallucination detector, a repair and a governor for language models, built from the orbit partition of a relation, with eleven proved bounds behind them, one a no-go result.

A model can know a fact and still be unable to reach it. When it cannot, distinct entities collapse onto one answer. That collapse is a topological invariant, it is measurable with no ground truth, and it is bounded from below by a theorem. The same bound then selects the intervention — including how much noise to add, and whether to intervene at all.

Invented by Teerth Sharma · github.com/teerthsharma/caustic · teerths57@gmail.com · MIT

Two models, and the distinction is load-bearing. Sections 1–13 are measured on Qwen/Qwen2.5-0.5B (D = 896, 24 blocks). Section 14 is measured on distilgpt2 (D = 768, 6 blocks). No figure from §14 may be combined with one from §1–§13. They describe different networks. Reconciling them is outstanding work, not a detail.

1 · The failure, measured

Retrieval is gated by the character of the surrounding text, not by its length. The prefix below is held at exactly 128 tokens in every row. Only what those tokens are changes.

The prefix contains none of the answers and is identical across every entity, so it carries no task information. The control is built into the design rather than argued for afterwards.

Same model, same entities, same prefix length. Only the character of the prefix varies.
capital, 20 entities	accuracy	distinct answers	largest orbit
no prefix	0.550	15	4
coherent prose, 128 tok	1.000	20	1
shuffled words, 128 tok	0.000	1	20
" the" × 128	0.000	1	20
random token ids, 128 tok	0.100	3	18
language, 12 entities	accuracy	distinct answers	largest orbit
no prefix	0.500	8	5
coherent prose, 128 tok	0.750	12	1
shuffled words, 128 tok	1.000	12	1
" the" × 128	0.000	2	7
random token ids, 128 tok	0.750	10	3

The same 128 tokens produce accuracy 1.000 or 0.000 depending only on whether they form language. Incoherent context does not merely fail to help: it merges all twenty countries onto a single answer, a largest orbit of 20, worse than the orbit of 4 with no prefix at all.

A prompt whose prefix is the word " the" one hundred and twenty-eight times is a well-formed input, a valid tokenisation, and an unanswerable question.

Caveat. capital and language disagree on the shuffled condition, 0.000 against 1.000. The boundary between coherent and merely lexically diverse is not settled by this measurement.

2 · The orbit partition

Under the failure the model does not become noisy. It becomes constant.

Distinct entities receive one answer. The object that records exactly that is the partition of entities induced by the model's answer map f, whose blocks are the classes of

e₁ ∼ e₂ ⟺ f(e₁) = f(e₂)

This is H0 of the answer-equivalence relation — its connected components. Three properties make it the right instrument:

It needs no ground truth. It is computed from the model's own answers over all entities, whether those answers are right or wrong.
It cannot be confounded by accuracy. Every entity is included regardless of correctness, so no class can collapse and no comparison inherits a moving base rate.
It is what the theorems constrain. Both n − m and 1/k below are statements about blocks of this partition — and, because the bound needs no answer key, it is also the objective every intervention on this page is scored against.

And it is stable. The adjusted Rand index between the partitions at different prefix lengths, over all entities regardless of correctness:

Partition stability against context growth, with the answer-churn column beside it.
capital	ARI	answers changed
0 → 32	−0.0243	0.600
0 → 128	0.0000	0.450
128 → 512	1.0000	0.150

Below the threshold the partition is destroyed and rebuilt. Above it the partition is exactly preserved while individual answers still move — structure held, labels free. The churn column is what makes an ARI of 1.0000 evidence of stability rather than evidence that nothing happened.

3 · Detection without ground truth

A fact carries a group action. A model that holds the fact must respect two halves of it; a model outside its retrieval regime respects only one.

invariance     paraphrase the prompt  ->  the answer must NOT change
equivariance   swap the entity        ->  the answer MUST change

The failing model is invariant where it should be equivariant: it returns the same answer whichever entity is named. Both quantities are computed from the model's own outputs under transformations of its own input. Neither consults a correct answer, which is the only condition under which a hallucination detector is of any use.

AUROC, oriented so that predicting *wrong* scores above 0.5. Bracketed figures are bootstrap intervals.
	invariance (self-consistency)	collision (equivariance)
capital	0.859	0.995 [0.97, 1.00]
language	0.942	0.950 [0.83, 1.00]
pooled, n = 32	0.920	0.945 [0.85, 1.00]

What carries the weight here is the certificate, not this AUROC. For an injective relation on n entities producing m distinct answers, at least n − m answers are provably wrong with no answer key consulted. Given the correct answers as a set — that Paris and Berlin are capitals, not which country each belongs to — the bound sharpens to n − |f(E) ∩ G|, which hit the true error count exactly in 16 of 16 measured rows across three model families while n − m was loose in all 16.

These are deterministic inequalities over a finite set: no sampling distribution, no minority class, nothing to be underpowered about. 60 bound checks, 3 models, 0 violations. The AUROC below is measured on 12 to 20 entities with minority classes of 1, 3, 4 and 8, and is the weakest claim on this page rather than the headline.

Scope. These figures are measured on relations whose errors take the form of collapse — distinct entities pooling onto one answer. The detector detects collapse, not error. Where errors disperse instead, pooled AUROC falls to 0.67–0.71 with bootstrap intervals spanning 0.5.

Those intervals are underpowered rather than null: at the relation sizes used here (n = 12–20) a 95% bootstrap interval has power 0.00–0.38 to separate a true AUROC of 0.70 from 0.50, and half-width 0.05 needs n ≈ 404. The minority classes behind the reported figures are 1, 3, 4 and 8 items.

Injectivity is necessary and not sufficient. currency is injective at the token level and its collision AUROC is 0.306, below chance, because the score averages five templates while the label comes from one. The precondition is also token-level: small_capital declares injectivity but Asmara/Asuncion share first token 1634 and Lusaka/Ljubljana share 444.

Precision is proved and recall is not. A model that shuffles the correct answers among entities pools nothing, emits nothing inadmissible, scores zero accuracy, and is invisible to the certificate. Measured recall is 0.65–0.79.

Five templates per relation means five forward passes per entity, and no derivative of anything. The invariance half is close to published self-consistency work and is included as the baseline the other half has to beat; the equivariance half — penalising a model for giving the same answer to different entities — is what reaches 0.995. Adding a third symmetry does not help: an inversion test scores 0.5686 pooled and exactly 0.5000 on capital, and the unweighted sum of all three scores 0.9451, identical to equivariance alone.

The precondition, and it is not optional. Equivariance requires the relation to be injective: distinct entities must genuinely warrant distinct answers. On a many-to-one relation — many countries share a continent — distinct entities should collide, and the signal inverts to 0.273. Point it at such a relation and you do not have a broken detector; you have a working detector with the sign reversed, which is worse.

RelationSpec.injective records the precondition so it cannot be forgotten. The certificate below returns zero whenever it is false, and select_prefix raises rather than optimising a score it knows is inverted.

4 · Five branches, and what each contributes

Each branch of mathematics answers one question about the collapse. They chain, and the chain is the argument.

Two branches feed topology, topology feeds game theory, and the no-go stands aside to explain why the differential route cannot be taken alone.

topology · what shape does the failure have?

The answer-equivalence relation partitions the entities; the failure is a merge of blocks. Because the true relation is injective, a merged block is already a proof of error. Topology is what turns an observation into a count — and the count into an objective a deployed system can optimise without labels.

game theory · how bad is a merge, downstream?

A block of size k mapped to one answer is a pooling equilibrium in the signalling sense. Pooling destroys the receiver's ability to infer the sender's type, and no downstream player repairs it. This is what makes collapse an unrecoverable error rather than a recoverable one.

differential geometry · what causes a merge?

If the logit of a candidate token does not vary along a path between two entity representations, the two entities receive the same logit — and share an orbit. Vanishing entity coupling is therefore a sufficient condition for pooling. Differential geometry feeds topology, and the partition sees the consequence whether or not the coupling itself is measurable.

chaos theory · what drives merges to happen?

Measured on a live model, the token-position Jacobian product contracts volume enormously: a strongly negative exponent sum on a positive leading exponent, the signature of dissipative dynamics on a low-dimensional attractor. Once the image volume falls below the resolution separating decision regions, distinct inputs must pool. Contraction is the driver, and depth is the clock.

partial-differential symmetry · why not just watch the Jacobian?

Because it provably cannot work. There is a smooth map, nonsingular at every point, that is still many-to-one, and its two preimages have Jacobians related by a rotation — so they share every spectral invariant. No determinant, smallest singular value, condition number or spectral decay can tell the injective case from the collapsed one. It is the only result here that tells you what not to build.

5 · The theorems, with proofs

Six theorems were proved after the original five below. caustic/theorems.py carries full statements, proofs and executable witnesses for all eleven; they are summarised here so a citation from the abstract finds a statement rather than a dangling number.

1* — given the correct answers as a set G, errors ≥ n − |f(E) ∩ G|. Never weaker than Theorem 1, and stronger by exactly one answer when anything inadmissible is emitted. Exact in 16 of 16 measured rows.
2* — a receiver seeing all T paraphrases recovers at most m_join / n. The join is discrete under coherent context and collapses under " the"×128, so the bound separates a repairable regime from an unrepairable one.
6 — the certified set has precision ≥ (n − m)/|S| = 1 − b/|S|, proved before it is evaluated. Realised 1.000 against floors of 0.667–0.950.
6* — tightened with G to (|S| − b_adm)/|S|, counting only orbits whose shared answer could be somebody’s truth. Takes the collapse rows from 0.950 to an exact 1.000.
7 — no-go. There is an observation at which no sound recall floor is positive: a model that permutes the correct answers among entities. So no constant c > 0 bounds recall everywhere.
8 — recall ≥ (n − m*)/n. Positive at 99.3% of observations, attained, and zero exactly at Theorem 7’s witness. Verified over 2,048,574 exhaustive configurations with 0 violations.

Theorem 8 corrects an error: Theorem 7 was first stated as “no recall floor exists”, which is false. The floor is certified_error_floor, already shipped and reported only as an error rate. Theorem 7’s witness marks where it is zero, which makes it tight rather than absent.

Theorems 1, 1*, 2, 2*, 6 and 6* are instances of one counting argument — a map constant on a block agrees with an injective truth on at most one member, and on none if the block’s value is nobody’s answer. They are separated because they are applied to different partitions and reported as different quantities, not because they need different proofs.

Every statement is elementary, and that is deliberate: the value is not in the difficulty of the proofs but in the fact that each is checkable against a measurement made in this repository. E is a finite set of entities, |E| = n. A relation R : E → A is injective when distinct entities have distinct correct answers. A model induces f : E → A, and P(f) is the partition of E by the value of f, with m = |P(f)| blocks called orbits.

Theorem 1 — topology — Orbit Error Bound

For injective R, the number of entities on which f errs is at least n − m.

err(f) ≥ n − m

Proof. The true map R is injective, so distinct entities carry distinct correct answers. If e₁ ≠ e₂ lie in one orbit then f(e₁) = f(e₂) while R(e₁) ≠ R(e₂), so f is wrong on at least one of them. An orbit of size s therefore contributes at least s − 1 errors, and summing over the m orbits gives Σ (sᵢ − 1) = n − m. ∎

Tight: attained exactly when every orbit contains one correct answer. Measured: capital with no prefix gave n = 20, m = 15, certifying at least 5 wrong answers with no ground truth consulted; observed accuracy 0.550, so 9 were actually wrong and the bound held with slack.

Why this one does more work than the others. It is the only theorem here that turns into an objective. Because the bound consults no answer key, it can be evaluated on a live relation where no answers are known — which is what makes the noise selection of §8 and the prefix competition of §11 possible at inference time.

Theorem 2 — game theory — Pooling Recovery Bound

If f maps a block of k entities to one answer, then for any downstream function h, the probability that h(f(e)) recovers e under a uniform prior on that block is at most 1/k.

Pr[ h(f(e)) = e ] ≤ 1/k

Proof. h ∘ f is constant on the block, so it takes one value there. It can therefore agree with the identity on at most one of the k entities. Under a uniform prior the success probability is at most 1/k. ∎

A block of size k > 1 is a pooling equilibrium, and no amount of downstream capability recovers it — not a larger model above that layer, not a longer chain of thought, not a better decoder. Measured: the " the" × 128 prefix pooled all 20 countries, bounding any downstream recovery at 0.05.

Theorem 3 — differential geometry — Zero Coupling Implies Pooling

Let z_c(h) be the logit of token c as a function of the entity representation h, continuously differentiable on a domain containing a path γ from h₁ to h₂. If the directional derivative of z_c along γ vanishes identically, then z_c(h₁) = z_c(h₂).

z_c(h₂) − z_c(h₁) = ∫_γ ∇z_c · dℓ = 0

Proof. The fundamental theorem of calculus along γ, as displayed. ∎

Consequence. If this holds for every candidate c, the two entities receive identical logit vectors, hence identical answers, hence share an orbit, and Theorem 1 applies. Differential geometry feeds topology. Measured: the token the model actually chose on wrong items coupled to the entity at 0.93 times its coupling to control tokens, against 1.33 for the correct token — the chosen token was closer to entity-independent, which is the finite-difference shadow of this statement. path_integral_change evaluates the integral numerically, and the suite pins it against linear and quadratic closed forms to 1e-6.

Theorem 4 — chaos theory — Dissipative Pooling

Let T be differentiable with characteristic exponents λ₁ ≥ … ≥ λ_D whose sum S = Σᵢ λᵢ is negative. Then for any bounded set A of positive Lebesgue measure, the volume of its image contracts.

vol(Tⁿ(A)) ∼ e^nS ⟶ 0

Proof. The change-of-variables formula gives vol(Tⁿ(A)) = ∫_A |det D(Tⁿ)|, and the exponents are defined so that (1/n) log |det D(Tⁿ)| → Σᵢ λᵢ = S. With S < 0 the integrand decays like exp(nS), so the volume does. ∎

Consequence. Once the image volume falls below the resolution separating decision regions, distinct inputs must land in one region and pool. Chaos theory feeds topology. Measured on distilgpt2: the token product at block 3 gave S = −226.74 over 768 dimensions with only 139 expanding directions, and D_KY = 29.57 at block 1 — a 26× compression against the width. What prevents total pooling in practice is that the trajectory is short, not that the map is volume-preserving.

Theorem 5 — partial-differential symmetry — No Local Criterion Detects Pooling

There exists a smooth map F whose Jacobian is nonsingular at every point of its domain and which is not injective. Consequently no function of the local Jacobian alone — determinant, smallest singular value, condition number, spectral decay, or any other pointwise invariant — can decide injectivity.

F(x, y) = (e^x cos y, e^x sin y)

Proof by witness. det DF = e^{2x} > 0 everywhere, yet F(x, y) = F(x, y + 2π). The Jacobians at the two preimages are related by a rotation and therefore share every spectral invariant, so no pointwise function of the Jacobian distinguishes the injective case from this one. ∎

This is the honest statement of a moral the refutation of the Jacobian Conjecture made vivid: local invertibility everywhere does not imply global injectivity, and the failure is invisible to local data. It is a no-go result, and it is the only theorem here that tells you what not to build. Measured corroboration: zero of 768 singular values fell below 1e-6 × sigma_max, so the observed collapse occurred with an everywhere-nonsingular Jacobian. The escape is global, and Theorem 1 is what takes it: compare two entities rather than examining one point — which is why the shipped detector is five forward passes and no Jacobian at all.

6 · What the method certifies

Theorem 1 turns the partition into a proof. For an injective relation on n entities producing m distinct answers, at least n − m answers are provably wrong, and the argument never consults an answer key.

Certified floor beside measured error. The theorem asserts floor ≤ measured, and it holds on every row with slack.
relation, condition	n	m	certified floor	measured error
capital, no prefix	20	15	0.250	0.450
capital, prose 128	20	20	0.000	0.000
capital, " the" ×128	20	1	0.950	1.000
language, no prefix	12	8	0.333	0.500
language, prose 128	12	12	0.000	0.250
language, " the" ×128	12	2	0.833	1.000

On capital, the choice of prefix moves the provable error floor across a 95-point range — from 0.950 under a degenerate prefix to 0.000 under coherent prose, at identical token count. The reverse move is real and is reported: a prefix can add 0.700 to the certified floor, and repair_by_context returns that case as WORSENED rather than quietly returning a number.

Downstream, Theorem 2 caps what anything else can do. A pooled block of size k limits any recovery of the entity from the answer to 1/k. For the fully collapsed 20-entity partition produced by the " the" × 128 prefix, the ceiling is 0.05. Nothing placed after that point — a bigger model, a verifier, a second pass, a retrieval step reading the answer — raises it, because what distinguished the twenty entities is not in the answer to be read.

What these numbers are, and are not. They are bounds on a certificate, measured on one model and two injective relations. +0.250 means a quarter of the answers were provably wrong before the intervention and none are provably wrong after — not that the true error rate fell by exactly that much. On language the true error fell from 0.500 to 0.250 while the certified floor went to zero, which is the bound behaving exactly as a bound should.

The certificate is one-sided by construction. It can prove a model wrong. It can never prove a model right.

7 · The decision is a near-tie

The decision the model gets wrong is a near-tie, and this is what makes everything after it possible.

Measured on wrong items. The model is not confidently wrong; it is marginally wrong.
quantity	value
vocabulary	151,936
wrong items	15 of 32
logit gap, chosen minus gold	mean 0.8338 · median 0.8085
logit standard deviation over vocab	mean 3.2942
gap as a fraction of one sd	mean 0.2526
p(chosen) / p(gold)	0.2042 / 0.1005 · ratio 2.76
mass on the other 151,934 tokens	0.6953
effective support exp(H)	wrong 30.0 · correct 27.5

The model is wrong by a quarter of a standard deviation, choosing between roughly thirty live candidates while the remaining 151,934 tokens absorb 69.5% of the probability mass. That explains the size of the coherence effect: a prefix has to move logits by about 0.83 to flip these decisions, and a positive leading exponent means a small change at the input grows along the expanding directions.

prefix character changes
  -> perturbation grows            lambda_1 = +0.1653
  -> logits shift by order 0.83
  -> flips a 0.2526 sd near-tie among about 30 candidates
  -> distinct entities land on one answer, which is orbit collapse
  -> certified error floor moves from 0.000 to 0.950

Chaos theory earns its place here as the explanation rather than as a detector: the exponents do not separate correct from wrong answers and are useless as a signal. What they explain is why an intervention this weak — 128 tokens about photosynthesis, containing none of the answers — produces an effect this large. The decision was never robust.

And it names the next experiment. A signal present but below the level at which the readout commits to it is, in the exact technical sense, a sub-threshold signal. Systems with sub-threshold signals have a known and falsifiable response to noise.

8 · Stochastic resonance

Stochastic resonance is the phenomenon in which a nonlinear system's response to a sub-threshold signal improves under moderate noise and degrades again under too much. It predicts a specific shape, and only one of the three possible shapes is evidence.

monotone increase   noise is smoothing something; not resonance
monotone decrease   no resonance; noise only destroys
inverted U          resonance, with an optimum at intermediate noise

The peak must also exceed the zero-noise baseline, or the curve is a decline with a flat start. Where the noise goes: into the input embeddings, scaled by their own standard deviation, so sigma is dimensionless. Injecting into the logits directly would be a different and much weaker claim — it would test whether adding noise to a ranking changes the ranking, which is trivially true. Each level is scored by majority vote over 16 draws, because resonance appears in the expectation and a single noisy pass is a worse estimator than a clean one.

language, 12 entities, Qwen/Qwen2.5-0.5B, seed 0, majority vote over 16 draws. The dashed vertical marks sigma = 0.80: accuracy is maximised and the certified floor is minimised at the same point, and the floor never saw an answer.

The full sweep. The certified-floor column is `certified_error_floor(12, m)` applied to the distinct-answer column beside it, so it is a function of the partition and of nothing else.
sigma	accuracy	vs baseline	distinct answers	certified floor
0.00	0.500	+0.000	8	0.333
0.40	0.583	+0.083	9	0.250
0.80	0.833	+0.333	11	0.083
1.20	0.667	+0.167	11	0.083
1.60	0.000	−0.500	2	0.833
3.00	0.000	−0.500	1	0.917

Inverted U confirmed. Accuracy rises to 0.833 at sigma = 0.80 and falls to 0.000 by sigma = 1.60, a gain of +0.333 over the zero-noise baseline from noise alone. No prefix was added, no answer was consulted, no parameter was changed. The only intervention is Gaussian noise on the input embeddings and a vote.

The distinct-answer column is the mechanism, not a diagnostic beside it. It runs 8 → 9 → 11 → 11 → 2 → 1. Moderate noise breaks orbit collapse — entities that were sharing an answer stop sharing it — and excessive noise causes orbit collapse, driving all twelve languages onto one answer. At sigma = 3.00 the model gives the same answer to every question, which is at least self-consistent, and is the sharpest available demonstration that an invariance-only detector would score that condition as healthy.

Theorem 1 selects the noise level with no ground truth

This is the result that makes the section a method rather than a curiosity. The accuracy series requires an answer key. The floor series does not — it is (n − m) / n computed from the model's own answers. The two agree on the argmin. A caller with no labels can sweep sigma, score each level by the certified floor, and land on the level that in fact maximises accuracy.

That is a closed loop: a quantity a deployed system can compute selects a hyperparameter a deployed system otherwise cannot tune. The same loop drives prefix selection in §11, where the empty prefix is always a candidate so the governor can decline.

The optimum is relation-dependent, and capital declines. Under identical noise on the identical model, capital shows no resonance at all: accuracy declines monotonically from 0.550 to 0.200. There is no universally good sigma, only a procedure for finding one — which is why the paragraph above is the load-bearing half of this section. Publishing sigma = 0.8 as a recommended setting would have been wrong on one of the two relations tested.

python -m caustic.experiments.stochastic_resonance

9 · System prompts are never neutral

A system prompt is a prefix, and §1 established that prefixes move the certified error floor by up to 95 points at fixed length. So "does this system prompt damage factual retrieval" is an empirical question with a cheap answer, and the answer is not automatically no.

Six prompts, `capital`, 20 entities. ARI is against the no-prompt partition; entropy is mean predictive entropy.
prompt	tok	acc	orbits	largest	ARI vs none	entropy
none	0	0.550	15	4	1.0000	3.5622
helpful	14	0.750	19	2	0.1923	3.0786
cautious	29	0.300	9	6	0.2721	3.2967
persona	27	0.900	20	1	0.0000	4.3754
json	22	0.950	20	1	0.0000	3.6461
long_policy	54	0.750	18	3	0.1465	4.0753

Three readings, in order of how uncomfortable they are.

Not one of six is neutral. A prompt is neutral when it leaves the partition alone, ARI = 1.0000. The highest achieved is 0.2721 and two are exactly 0.0000. Every one of these prompts restructured which entities share an answer. A prompt that improves accuracy while restructuring the partition has intervened on factual retrieval; it has not left it alone.

The prompt instructing care is the worst one. cautious — "if you are not certain of a fact, say so rather than guessing" — halves accuracy, 0.550 to 0.300, and merges orbits from 15 to 9. By Theorem 1 that moves the certified error floor from 0.250 to 0.550: it more than doubles the number of answers that are provably wrong. The instruction most likely to appear verbatim in a production system prompt did the most damage of the six.

Formatting instructions are not free either, and here the sign is positive. json, 22 tokens about output format and nothing about facts, reaches 0.950 with a fully separated partition. persona, which asks for confidence and wit, reaches 0.900. Neither prompt contains information about capitals, and both moved the partition further than helpful did.

from caustic.governor import prompt_cost

cost = prompt_cost(spec, top1, "You are a careful assistant. ", name="cautious")
print(cost)          # cautious: floor 0.250 -> 0.550 (+0.300), ARI 0.2721, costs
print(cost.neutral)  # False — neutrality is ARI >= 0.99, not "did not obviously hurt"

10 · The ensemble, and where it fails

A near-tie and a positive leading exponent together say a single forward pass sits on an unstable decision. The classical response is not to predict one trajectory but to average over many.

Here that means asking the same question under eight different neutral prefixes and taking a consensus. Two consensus rules, because they fail differently and reporting only one is choosing the flattering one after the fact.

single = no prefix, one forward pass. best-1 / worst-1 = the luckiest and unluckiest individual prefix of the eight.
relation	n	single	best-1	worst-1	mean-logit	vote
capital	20	0.550	0.800	0.000	0.100	0.600
language	12	0.500	0.917	0.417	0.917	0.917

Mean-logit averaging collapses on capital to 0.100 — below the 0.550 of a single pass with no ensemble at all. The ensemble was built to reduce variance and, under mean-logit, reduced accuracy instead. The cause is visible in the same row: individual prefixes span 0.000 to 0.800, and a single confident outlier dominates a mean of magnitudes. Averaging logits gives the most opinionated member of the ensemble a veto over the other seven.

Majority vote never goes catastrophic. It reaches 0.600 on capital, above the single-pass baseline, and 0.917 on language, equalling best-1 without knowing in advance which prefix that was. It does not always win — on language mean-logit ties it — but it does not lose by 45 points either.

The lesson, stated as a rule. With a positive characteristic exponent, average ranks, not magnitudes. A magnitude is a trajectory-dependent quantity in a system where trajectories diverge; a rank is not. Every consensus in this repository — the resonance vote of §8, the governor of §11 — is rank-based or partition-based, and none of them averages a logit.

The failure is the more useful half of the result, and it is the half a report showing only the language row would have omitted.

11 · Repair, and the governor

The detector finds collapsed orbits. repair_by_context applies the intervention the measurements point at, and reports the effect size rather than asserting it.

from caustic import NEUTRAL_PREFIX, repair_by_context

report = repair_by_context(spec, answer_fn, prefix=NEUTRAL_PREFIX, gold=gold)
print(report)
# largest orbit 20 -> 1, distinct answers 1 -> 20 of 20 entities,
# accuracy 0.000 -> 1.000  REPAIRED

repaired — true only for a genuine collapsed → fully separated transition.
worsened — true when the prefix merged entities that were previously separate.
accuracy_delta — reported only when gold is supplied; the verdict never consults it.

worsened is not a defensive check. An incoherent prefix of the same length drove the largest orbit from 4 to 20 in the measurements this module is built on, adding 0.700 to the certified floor. A prefix can make things much worse, and a repair function that cannot say so is a repair function that will eventually lie.

Why this is not prompt engineering in the pejorative sense. The prefix contains none of the answers, is identical across entities, and is unrelated in subject. It cannot be leaking an answer, because the same 128 tokens in shuffled order drive accuracy to zero on capital. What it supplies is distributional, not informational. NEUTRAL_PREFIX is 128 tokens on mechanical calculators, ocean currents, language and photosynthesis — the least interesting paragraph in the repository, and on capital it is worth 0.550 → 1.000.

The governor: selection by competition

A single hand-picked prefix is a bet on which of several you happened to write. Over the eight neutral prefixes of §10, individual accuracy on capital ranged from 0.000 to 0.800. The governor does not need to be lucky, only to be able to score — and Theorem 1 gives it a score that needs no ground truth.

Always enters the empty prefix, under the name none, so declining to intervene is a first-class outcome rather than a failure to find one.
Declines unless a candidate strictly beats doing nothing, breaking ties toward declining and then toward the earlier candidate.
Reports the margin over doing nothing as improvement, which is therefore never negative.
Raises on a non-injective relation rather than optimising a score it knows is inverted.
Returns every candidate's score, because a governor that shows only its winner is a governor that cannot be audited.

from caustic import select_prefix

verdict = select_prefix(spec, top1, {"prose": NEUTRAL_PREFIX, "terse": "Answer briefly. "})
print(verdict)            # selected 'prose': floor 0.250 -> 0.000 (+0.250), largest orbit 1
print(verdict.intervened) # False when nothing beat the empty prefix
print(verdict.scores)     # every candidate's floor, including 'none'

12 · Comparing geometry across model widths

Every geometric quantity here lives in R^D, so a coupling ratio measured at D = 896 is not directly comparable to one from a model of width 4096. That is the sharpest limitation in the results, and the bridge is the partial answer to it.

A seeded Johnson–Lindenstrauss projection into a fixed k. The JL lemma states that a random linear map into R^k preserves pairwise squared distances to within 1 ± eps with high probability, for k = O(log n / eps²) independent of the source dimension. Norms survive, and a ratio of norms survives with the distortions partially cancelling. The seed is the load-bearing detail: two runs sharing CANONICAL_SEED share a projection matrix exactly, so numbers from different models land in one frame rather than merely in one dimension.

A true coupling ratio of 1.33, recovered after projection from three source widths. Measured on synthetic gradients built with the observed structure.
target k	D = 896	D = 2048	D = 4096	spread	per-item error
64	1.3445	1.3376	1.3444	0.0052	9.3%
256	1.3316	1.3287	1.3288	0.0022	5.4%

The limitation, stated as prominently as the result. Per-item relative error is 9.3% at k = 64 and 5.4% at k = 256. The projection makes population statistics comparable across widths. It does not make individual measurements comparable. A claim about a distribution survives the projection; a claim about a single entity does not, and a 5.4% per-item error is larger than most of the effects this repository would want to attribute to a single entity.

jl_distortion_bound returns inf rather than a small number whenever k is too small for any useful guarantee, because a small number would be read as a promise.

13 · Inside a wrong answer

Measured on items the model answers wrongly, with grouped cross-validation so that no template appears in both folds.

Nothing was destroyed. The right answer lost a competition.
quantity, on wrong items	value	chance
entity still linearly recoverable from h_22	0.9624	0.0312
correct answer token recoverable from h_22	1.0000	0.0400
median rank of the correct answer	3 of 151,936	—
correct answer within the top 10	88 / 100	—
correct answer within the top 1000	100 / 100	—

Recoverability on wrong items (0.9624) exceeds that on correct ones (0.8981). The context survives; the answer survives, near the top. This is the state Theorem 2 addresses: the loss happens where distinct entities are mapped onto one answer, not in the representation feeding that map. It is also why Theorem 5 matters operationally — there is no local degeneracy at the moment of failure to go looking for, because nothing locally degenerate has happened. And it is the precondition for §8: noise can only lift a signal that is present, and this table is where the signal is shown to be present.

Median rank of the correct answer, each layer read through the final norm and unembedding.
layer	all	correct	wrong
20	30552	29594	31626
22	7	3	12
24	2	1	3

The answer materialises across two layers, late and abruptly, and does not degrade afterwards.

Caveat. The lens applies the final norm to intermediate states that were not trained to be read through it, so ranks before layer 24 are indicative. Layer 24 is exact — it reproduces the model's own output.

14 · Dynamics and cost

Measured on distilgpt2, D = 768, 6 blocks — not on the model used in §1–§13. No figure in this section may be combined with one from §1–§13. The dynamics and cost measurements predate the switch to a model that knows facts and were not re-run. Reconciling the two is outstanding work.

Theorem 4 needs a negative exponent sum. This is where that hypothesis is measured rather than assumed.

Characteristic exponents of the token-position Jacobian product, block 3, 46 steps, against a shuffled-token control preserving the token multiset.
	lambda_1	sum	expanding	last-step drift
grounded	+0.1653	−226.74	139 / 768	0.0012
shuffled	+0.1852	−170.37	151 / 768	0.0003

Positive leading exponent with a strongly negative sum: dissipative dynamics on a low-dimensional attractor. The drift column is what makes the values quotable — each is small against the value it drifts on.

Kaplan–Yorke dimension per block, validated against the textbook Lorenz spectrum (0.906, 0, −14.572) giving 2.0622, reproduced to 1e-3.
block	D_KY	D / D_KY	expanding
1	29.57	26.0	9 / 768
3	298.08	2.6	139 / 768
5	674.67	1.1	354 / 768

Block 1 places 768 dimensions of transport on a 29.57-dimensional attractor. D_KY varies across depth with cv 0.6801, so it is not a width-invariant constant, and the block-0 value saturates the formula rather than measuring a dimension. Both facts are stated because the number is otherwise easy to over-read: contraction is the hypothesis Theorem 4 needs, and D_KY describes how strong it is at a given depth, not a second invariant of the network.

Cost, measured against one block forward pass as the unit.
operation	time	vs forward
block forward	0.588 ms	1×
full 768 × 768 exact Jacobian	53.694 ms	91.3×
top-8 Krylov, 20 iterations	1428.938 ms	2429.5×

The exact Jacobian is 26.6× cheaper than the Krylov estimator of its own top eight singular values, which it matches to 1.685e-04. The estimator costs 26.6× more than the quantity it was brought in to approximate, and it is not a broken estimator: batched reverse-mode AD vectorises across all outputs, while k-column power iteration runs k × iters sequential passes. The crossover width is not measured and is not assumed.

None of this is on the detector's path. The shipped detector costs five forward passes and no Jacobian at all, which Theorem 5 predicts in advance: the Jacobian could not have supplied the missing signal at any price.

15 · Running it

git clone https://github.com/teerthsharma/caustic.git
cd caustic
pip install -e ".[dev]"          # numpy, torch, pytest
python -m pytest -q              # 163 passed, no model download, no GPU
pip install -e ".[experiments]"  # transformers, for a live model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

from caustic import NEUTRAL_PREFIX, RelationSpec, orbit_partition, repair_by_context
from caustic import select_prefix

MODEL = "Qwen/Qwen2.5-0.5B"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, dtype=torch.float32).eval()


def top1(prompt: str) -> int:
    """Prompt -> top-1 next-token id. Must be deterministic: a sampled answer
    makes the partition noise."""
    ids = tok(prompt, return_tensors="pt").input_ids
    with torch.no_grad():
        return int(model(ids).logits[0, -1].argmax())


facts = {"France": "Paris", "Japan": "Tokyo", "Peru": "Lima",
         "Kenya": "Nairobi", "Norway": "Oslo"}

spec = RelationSpec(
    templates=("The capital of {e} is", "{e}'s capital is"),
    entities=tuple(facts),
    injective=True,          # distinct countries have distinct capitals
)

# 1. Detect. No ground truth is consulted anywhere in this block.
report = orbit_partition(spec, top1)
print(report)                      # n entities -> m distinct answers, largest orbit s
print(report.certified_errors)     # Theorem 1: n - m, proved
print(report.certified_error_rate) # the same bound as a rate, which is the objective
print(report.collapsed)            # the boolean to act on

# 2. Repair with a fixed prefix, and measure the effect. `gold` is optional and
#    is used only to print an accuracy column beside the verdict.
gold = {e: tok(" " + a, add_special_tokens=False).input_ids[0]
        for e, a in facts.items()}

print(repair_by_context(spec, top1, prefix=NEUTRAL_PREFIX, gold=gold))
# largest orbit s -> 1, distinct answers m -> 5 of 5 entities,
# accuracy a -> b  REPAIRED

# 3. Or do not pick the prefix yourself. The governor runs candidates in
#    competition scored by the certified floor, always enters the empty prefix,
#    and declines when nothing beats doing nothing.
verdict = select_prefix(spec, top1, {
    "prose": NEUTRAL_PREFIX,
    "terse": "Answer with a single word. ",
})
print(verdict)              # selected 'prose': floor x -> y (+d), largest orbit 1
print(verdict.intervened)   # False if the empty prefix won
print(verdict.improvement)  # certified error removed relative to doing nothing
print(verdict.scores)       # every candidate, including 'none'

orbit_partition uses the first template. symmetry_scores(spec, top1) uses all of them and needs at least two, since invariance is undefined on one. Set injective=False for a many-to-one relation: certified_errors and collapsed then correctly report nothing, and select_prefix raises.

Each experiment prints its own table and its own control:

python -m caustic.experiments.coherence_vs_length       # §1
python -m caustic.experiments.orbit_invariant           # §2
python -m caustic.experiments.symmetry_break            # §3
python -m caustic.experiments.stochastic_resonance      # §8
python -m caustic.experiments.ensemble_and_neutrality   # §9, §10
python -m caustic.experiments.answer_presence           # §13
python -m caustic.experiments.attractor_dimension       # §14
python -m caustic.experiments.probe_cost                # §14

16 · Validation and environment

163 tests, every one against a closed-form or independently computed answer. Some of them are negative controls — inputs whose correct answer is "nothing here" — because the failure mode of a spectral pipeline is not an exception, it is a plausible number from noise.

A representative slice of the suite.
assertion	tolerance
Jacobian of a position-wise linear block equals its weight matrix	1e-10
Krylov estimate matches exact svdvals	1e-6
log-volume equals torch.linalg.slogdet	1e-8
Kaplan–Yorke reproduces the Lorenz value 2.0622	1e-3
Theorem 1 bound never exceeds true error count	2000 random instances
Theorem 2 bound beaten by no constant decoder	exhaustive
Theorem 3 path integral matches linear and quadratic closed forms	1e-6
Theorem 4 exponent sum equals log\|det Aⁿ\|	1e-9
Theorem 5 both preimages share every spectral invariant	1e-9
flat spectrum returns exponent zero (negative control)	1e-9
partition is bitwise identical across repeated calls	exact
select_prefix declines when no candidate beats the empty prefix	exact
JL projection is bitwise identical under a shared seed	exact

A passing suite is therefore a statement about the mathematics, not about the last time the code changed. There is no CI badge in this project, because there is no CI.

Measurement environment

Hardware   NVIDIA GeForce RTX 4060 Laptop, 8 GiB · Windows 11
Software   Python 3.11.9 · PyTorch 2.5.1+cu121 · transformers 5.3.0
           float32 · seed 0

§1–§13     Qwen/Qwen2.5-0.5B   D = 896   24 blocks   vocabulary 151,936
§14        distilgpt2          D = 768    6 blocks

The lambda_1 = +0.1653 that appears in the causal chain of §7 is a distilgpt2 measurement used to explain a Qwen2.5-0.5B observation. It is offered as a mechanism, not as a joint measurement. §12 is the beginning of the fix, and its per-item error of 5.4% at k = 256 is the beginning of why it is not yet the whole fix.

17 · Attribution

Four constructions in this repository were adapted from the author's other work.

Each construction, and the file that carries it here.
source	construction	used in
Epsilon	a seeded Johnson–Lindenstrauss map into a shared frame, so two agents can compare geometry without exchanging full representations	`caustic/bridge.py`
epsilon-cli	the stochastic-resonance framing: a sub-threshold signal, an inverted-U response, and the requirement that the peak beat the zero-noise baseline	`experiments/stochastic_resonance.py`
EPSILON-PHASE	adaptive noise scheduling — sweeping an injected-noise level rather than fixing it, and selecting per regime	the `sigma` sweep of §8
laamba-silence	a governor that runs several candidates in competition and lets a comparator pick the winner, rather than committing to one in advance	`caustic/governor.py`

What is new here is what each construction is pointed at: a scorer that requires no ground truth. The JL frame carries a certified quantity rather than an agent state; the resonance sweep is selected by Theorem 1 rather than by held-out accuracy; the competition is scored by the orbit partition rather than by a reward model.

18 · Limits

One model per claim, two models in the repository. §1–§13 are Qwen2.5-0.5B at D = 896; §14 is distilgpt2 at D = 768. They must not be combined. Whether coherence-gated retrieval is a general property of language models, or a behaviour of a 0.5B model outside its training regime, is not established here.
Two injective relations, of 12 and 20 entities. One distractor passage per condition, one seed. Every effect in §8, §9 and §10 rests on those two relations, and they disagree with each other more than once.
The resonance optimum is relation-dependent. language peaks at sigma = 0.80 with a gain of +0.333; capital declines monotonically under identical noise, 0.550 to 0.200. There is no recommended sigma, only a procedure for finding one.
Mean-logit ensembling is unsafe under a positive exponent. It scored 0.100 on capital, below a single pass, because one confident outlier dominates a mean of magnitudes. Average ranks, not magnitudes.
The system-prompt result is a measurement on one relation. Six prompts on capital. That cautious halves accuracy is a fact about these 20 entities and this model; that no prompt is neutral is the claim that generalises, and it generalises because neutrality is defined as ARI = 1.0 and nothing reached it.
The JL bridge carries populations, not individuals. Per-item relative error is 9.3% at k = 64 and 5.4% at k = 256.
Top-1 token comparison. Answers are compared by their first token, so a correct answer phrased differently counts as disagreement, and the partition, the accuracy column and every certified floor inherit that convention.
Injectivity is a required precondition, not a convenience. On a many-to-one relation the equivariance signal inverts to 0.273, the certificate is meaningless, collapsed is correctly always false, and select_prefix raises. The requirement was diagnosed by measurement rather than predicted in advance.
The bounds are bounds on a certificate. Theorems 1 and 2 are exact and they constrain a partition; they say nothing about how often such partitions arise in deployment. The certificate is one-sided: it can prove a model wrong, never right.
No pointwise Jacobian statistic can detect pooling. Theorem 5 is a no-go, not a gap to be closed by a better spectral summary. Zero of 768 singular values fell below 1e-6 × sigma_max, so the observed collapse happened with an everywhere-nonsingular Jacobian.
The logit-lens ranks before layer 24 are indicative, since the final norm is applied to states not trained to be read through it. Layer 24 is exact.
D_KY is not a width-invariant constant (cv 0.6801 across depth), and the block-0 value saturates the formula rather than measuring a dimension.
The two relations disagree on the shuffled condition (0.000 against 1.000), so the boundary between coherent text and merely lexically diverse text is not resolved by these measurements.