Token Fragility: Contextual Copying Failures Beyond Isolated Token Probes

[AemonAlgiz]

Hello,

I am an independent researcher, and I am looking for feedback on this research paper. The findings are interesting and highly reproducible.

Abstract

Successful repetition of an isolated token does not establish that the token will be preserved in surrounding text. I study FragileTokens, vocabulary entries that pass an isolated copying probe but exhibit copying errors in context. Our protocol inserts fixed token IDs into shared banks of random common-word sequences, measures the probability of the expected token at its copy position, and examines selected decoding failures. Across eight open-weight models from the Qwen and OLMo families, a probability-based screen flags 0.4% to 10.9% of isolation-passing tokens in at least 10% of tested contexts. These values measure screening prevalence, not the prevalence of verified decoding errors. Per-token screening scores correlate at 0.82 to 0.95 between context halves, and stored examples demonstrate deletion, substitution, truncation, and translation, including errors made with concentrated output distributions. I also evaluate whether proximity to known or context-screened glitch tokens identifies this conditional population. The better of input- and output-embedding proximity achieves an AUROC of 0.744 to 0.881 on seven models, although static features improve on surface-feature baselines on only three. Selected candidates are preserved less often than stable controls in several instruction and simulated tool-use tasks. A separate natural-entity experiment finds no incremental predictive benefit beyond existing controls. The contribution is a controlled characterization of contextual susceptibility among tokens that pass an isolation probe, together with evidence about its reproducibility, geometric associations, and task-specific consequences. The results motivate contextual verification while leaving prospective discovery performance and prediction of individual triggering contexts unresolved.

1 Introduction

A language model can reproduce a token when asked to copy that token alone and replace it when asked to copy a longer sequence containing the same token. This discrepancy matters whenever literal identity is part of an instruction: quoting a title, preserving an identifier, returning a list item, or placing a name in a tool query. A fluent response can satisfy the surrounding task while changing the item the task concerns.

The OLMo-2‑7B token according provides a concrete example. It is token ID 4184, and decoding and re-encoding its string returns the same ID. Under an isolated repetition probe, the correct continuation has log-probability −0.001. In the sequence deliver rel behind according sent floor wrong, however, the model emits sent at the position assigned to according and then continues from that following word. The probability of this incorrect emission rounds to 1.000 and its entropy to 0.00 bits. Across 48 sampled contexts, the recorded continuations delete according in 44 and reproduce it with probability approximately 0.999 in the remaining four. The contrast concerns the same model, token ID, and copying instruction.

Glitch-token research already studies failures to reproduce or manipulate particular vocabulary entries. It includes embedding-based detection, activation-based classification, prompt variation, and multi-task assessment (Land and Bartolo, 2024; Li et al., 2024; Zhang et al., 2024; Wu et al., 2026; Tang et al., 2026). The question addressed here is more specific: among tokens that pass a specified isolation probe, which remain susceptible when the same token is copied as part of surrounding text, and how consistently can we measure that susceptibility?

I use FragileTokens for this conditional behavioral class. Membership depends on the model, the copying protocol, and the context distribution. The term does not assert that vulnerability is independent of context, nor that every isolated-probe failure and contextual failure shares one mechanism. Our primary experiments use a probability screen to identify candidates for this class. I distinguish that screen from verified decoding errors throughout the paper.

The study makes three contributions. First, it measures the passing isolated-token population using shared context banks and fixed token IDs, allowing comparison of token susceptibility across contexts, independent runs, and model configurations. Second, it tests whether geometric signals associated with established glitch tokens also distinguish candidates within this population, including comparisons with surface features and random reference sets. Lastly, it examines whether selected candidates are preserved in subsequent instructions and simulated tool interactions, while reporting an explicit negative result for natural entity-name prediction.

The resulting claim empirically extends glitch-token assessment. I do not claim to introduce embedding-based glitch detection or to establish that prior research ignores all context. Instead, I show why an isolation result should be treated as one measurement condition rather than a general certificate of literal reliability.

2 Related Work and the Contribution Boundary

Early observations documented unusual responses to particular tokens under varied prompts and connected some of those tokens to unusual embedding regions (Rumbelow and Watkins, 2023). Subsequent work developed systematic detectors and evaluation procedures. These studies provide both the methodological foundation and the relevant comparison for FragileTokens.

Land and Bartolo’s (2024) Magikarp framework uses embedding-derived indicators to identify under-trained-token candidates and behavioral probes to verify them. Its analysis includes alternative prompts and examples involving copied code lists. Li et al. (2024) define glitch behavior broadly and evaluate repetition, spelling, and length; GlitchHunter uses an embedding graph and iterative clustering to locate problematic regions. Consequently, neither geometric candidate generation nor the use of surrounding prompt text is new in this study.

GlitchProber classifies tokens using attention and MLP features and studies mitigation by changing intermediate activations (Zhang et al., 2024). GlitchMiner uses entropy-guided discrete optimization with local embedding neighborhoods and verifies candidates under three repetition templates (Wu et al., 2026). Our experiments instead hold model weights fixed and evaluate contextual susceptibility after conditioning on success in an isolation probe. Low-entropy incorrect emissions are relevant to that evaluation, but their existence alone does not demonstrate that an entropy-guided detector will miss the corresponding token under every prompt.

Tang et al. (2026) provide the closest behavioral comparison. GlitchQuiz evaluates eight tasks, including sentence and paragraph construction, repetition, and multi-turn recall. Their Figure 2 shows a token passing simple tests and failing another task, and Section 5.3 acknowledges possible failures beyond the full assessment. Our contribution is repeated measurement of a fixed copying operation over shared context banks within an isolation-passing population, followed by reproducibility and prediction analyses. An isolation pass in our experiment does not establish a pass on GlitchQuiz, and I also do not claim priority for task-dependent token failure.

Table 1 summarizes the comparison. The present experiments directly compare available Magikarp labels for three model configurations. They do not constitute a head-to-head rerun of all five systems under a common contextual oracle; I make no comparative recall or efficiency claims for those systems.

Table 1. Relationship to the closest detection and assessment methods. Descriptions concern the published procedures, not a claim that their authors regard context as irrelevant.

Rarity is a further competing explanation. Yu et al. (2022) study how gradients affecting rare-token embeddings can contribute to representation degeneration. This motivates comparisons with rarity-related surface features, but it does not establish the training history or causal mechanism of the tokens studied here. Our token-ID feature is a vocabulary-order proxy, not a measurement of pretraining frequency.

3 Experimental Design

3.1 Models and sampling

The main copying experiments cover eight open-weight models from two families: one OLMo-2 model and seven Qwen models spanning three generations. Table 2 gives the tested configurations. A separate representation analysis uses Pythia-1.4B and is reported as an auxiliary experiment in Section 7.2. Thus, claims from the eight-model copying study do not automatically extend to Pythia or other model families.

For each model, we stratify the vocabulary sample by token-ID range. Sample sizes range from 4,200 to 8,400 IDs for the Qwen models. OLMo-2‑7B uses an initial sample of 2,098 IDs and a replication sample of 6,273. Each sampled token is inserted into every context in that run. The context banks contain random sequences of common words that individually pass the copying check; sequence lengths are 8, 16, 32, and 64, with a randomly chosen interior insertion slot. These are controlled common-word sequences, not samples of grammatical natural prose.

Table 2. Configurations in the main copying study. Vocabulary values are embedding-row counts. “Tokens × contexts” describes the sampled copying matrix. Flash-Next and 235B use FP8 checkpoints. Appendix A includes full identifiers and framing details.

The OLMo Team (2025) and Qwen Team (2025a, 2025b, 2026a, 2026b) describe the model families and releases. Differences among configurations include tokenizer, architecture, training, framing, and numerical precision. The model ladder is therefore a set of replications across configurations, not a controlled experiment on parameter count.

3.2 Copying protocol and token identity

The copying prompt contains two demonstrations followed by a sequence containing the target token. The isolation probe uses the same template, with the target alone as the text. Appendix A provides the exact template. I assemble every test prompt from token IDs, so the target ID remains fixed even as its surroundings change.

This choice is necessary to separate model behavior from re-tokenization. Adding a space to a decoded token string can change its segmentation before the model sees it. Direct ID insertion avoids that confound but also permits sequences that a tokenizer would not normally produce from an ordinary string. I therefore distinguish exact-ID copying from string preservation and restrict claims about normal text use accordingly. The canonical according example shows that the phenomenon is not confined to a noncanonical token string.

Under teacher forcing, I supply the expected copied prefix before evaluating the next-token distribution at the target’s output position. Let q₀(t) be the model’s probability of target ID t under the isolation probe, and q(t,c) its probability at the corresponding position in context c. A low value measures poor support for the required token under a correct prefix. It does not by itself show what an unconstrained continuation will do after an earlier mistake.

For the Qwen3.8 configurations, the copying instruction and demonstrations occupy the user turn of the model’s chat template, and Copy: open the assistant turn with thinking disabled. Other copying runs use a raw few-shot sequence. The framing is fixed within each model. This adjustment matters: under the raw cue, the Qwen3.8 models often start on a new line and change leading-space tokenization. Of 2,000 common-word checks, 3 copy under the raw framing, compared with 1,921 on 27B and 1,925 on Flash-Next under the chat framing.

3.3 Behavioral fragility and the probability screen

Let ŷ(t,c) denote the greedy next token at the target’s copy position under the teacher-forced prefix. The empirical rate of target-position copying errors over a context bank C of size m is

A behavioral FragileToken passes the specified isolated copying test and exhibits contextual errors. A quantitative class additionally requires a declared threshold on this error rate. Actual sequence-level copying failure is a separate outcome because it can involve errors before or after the target position.

The aggregate experiments use a related probability screen. The primary isolation gate admits q₀(t) > 0.5, which guarantees that t is the unique greedy output in isolation. It is a sufficient condition for an isolated greedy pass, not an exhaustive characterization of such passes. A stricter gate, used in the OLMo sensitivity analysis, admits ln q₀(t) > −0.1, or q₀(t) > approximately 0.905.

The contextual screen flags a cell if ln q(t,c) < −0.5, corresponding to q(t,c) < approximately 0.607. Its per-token susceptibility score is

A screen-positive candidate passes the isolation gate and has S(t;C) ≥ 0.10. A stable control passes the gate and has S(t;C) = 0. Tokens with scores between these thresholds form an intermediate group. With 24 contexts, the candidate criterion requires at least 3 flagged cells; with 48 contexts, it requires at least 5.

The probability screen and behavioral error rate must not be conflated. A token with probability 0.55, both alone and in context, passes the primary isolation gate and is flagged contextually, despite no probability decrease and a correct greedy output. Consequently, the reported screening prevalence is neither a direct estimate nor a guaranteed bound on the prevalence of behavioral FragileTokens. The stricter isolation gate removes this particular overlap, but contextual probability below 0.607 still does not establish a wrong argmax. I use decoded specimens to demonstrate actual errors and retain the aggregate probability results as screening measurements.

3.4 Reference classes and stored specimens

OLMo-2‑7B has an independent reference set of 442 Magikarp-verified glitch tokens. None passes the primary isolation gate in these experiments, and their mean contextual screening score in the initial bank is 0.993. For models without this label set, reference construction combines poor isolation performance with a contextual screening score of at least 0.90. Appendix B reports the reference-set sizes used in the geometric analyses. This contextual check is necessary because poor isolation performance can reflect the prompt format rather than persistent copying difficulty.

For inspected candidates and reference tokens, the specimen procedure retains the three worst contextual cells and one control cell per token. Records include the context IDs, target slot, expected-token probability, top-five output IDs and probabilities, entropy, and a greedy continuation when generation is available. Provenance fields identify the model and checkpoint revision, precision, architecture metadata, and software versions. These records support case analysis; selecting the worst cells makes them unsuitable for estimating a representative distribution of confidence over all contextual errors.

The descriptive failure categories are deletion, substitution, truncation, and translation, with script conversion recorded where appropriate. Deletion requires evidence that the model advances to the following text; an isolated top-token observation alone cannot establish how a full continuation proceeds. We use entropy below 2 bits as a concentration descriptor for a stored output distribution. It is not a calibrated measure of task correctness or of whether the model will verbally express uncertainty.

4 Contextual Susceptibility Among Isolation Passes

4.1 Screening prevalence

Table 3 reports the primary-gate results. Candidate prevalence ranges from 15 of 4,047 gated tokens on Flash-Next (0.4%) to 684 of 6,277 on Qwen3‑235B (10.9%). The range is conditional on the sampled vocabulary, selected isolation gate, context distribution, and threshold. It is not a prevalence estimate for natural text occurrences or for all tokens that a model can greedily repeat in isolation.

Table 3. Contextual screening under the primary isolation gate. Candidate counts use q₀ > 0.5 and S ≥ 0.10. “Zero score” is the percentage of gated tokens with S = 0. The correlation compares scores from even- and odd-indexed context halves. The final column is the context main-effect share of the reported variance decomposition.

Under the strict isolation gate, OLMo-2‑7B has 14 candidates among 873 tokens in the first bank (1.6%) and 29 among 1,888 in the replication bank (1.5%). These candidates have substantially higher isolation probability than the contextual threshold. They provide a sensitivity analysis against overlap of the two cutoffs, although verified argmax-error prevalence still requires direct measurement.

Comparisons restricted to shared token IDs preserve some cross-configuration differences. Among 1,592 IDs passing both gates for Qwen3‑1.7B and Qwen3‑235B, the candidate rates are 2.0% and 6.2%. Among 1,569 shared passes for Qwen3‑32B and Qwen3‑235B, they are 0.8% and 4.3%. These comparisons reduce differences in sampled IDs but do not isolate the effects of scale from architecture, precision, or training.

4.2 Reproducibility and context effects

Across the eight models, per-token scores correlate at 0.82 to 0.95 between context halves. A stronger replication on Qwen3‑32B doubles the token sample and draws a new context bank. Scores correlate at 0.984 over the 4,200 shared IDs, and 12 of the original 15 candidates meet the threshold again. Re-scoring stored Qwen3‑32B and Qwen2.5‑72B matrices on different hardware gives cell-level agreement of 0.999 and 0.996 and per-token score correlations of 0.9999 and 0.9996, respectively.

The initial OLMo matrix assigns 95.2% of the reported variance to the token main effect, 0.1% to the context main effect, and 4.7% to the residual. Restricting to the strict-gate population changes these shares to 71.7%, approximately 0.0%, and 28.2%. The large reduction in the token share after gating is consequential: much of the variation over the full sample separates broadly problematic tokens from reliable ones. Substantial residual variation remains within the gated population.

These observations support a reproducible token-level susceptibility under the sampled distribution. They do not establish that the triggering context is irrelevant. A small context main effect means that a particular carrier is not consistently difficult for every token; it can coexist with strong token-by-context variation. Consistent with this distinction, OLMo prediction of held-out cells reaches an AUROC of 0.94 using a token’s score in other contexts, compared with 0.56 using the carrier’s difficulty for other tokens. The tested token-context similarity features provide no additional predictive benefit.

Context length also matters for some populations. On the strict-gate OLMo sample, the flagged-cell rates for lengths 8, 16, 32, and 64 are 0.005, 0.008, 0.008, and 0.008. Among Qwen2.5‑72B candidates, the corresponding comparison reported for the shortest and longest carriers is 33% versus 52%. The study therefore identifies a stable ranking of susceptibility more clearly than it identifies the conditions that produce any individual error.

4.3 Comparison with released Magikarp labels

Table 4 compares the contextual candidate sets with available Magikarp records. None of the candidates in these three comparisons is a Magikarp-verified glitch token. Most are recorded as acceptable under that procedure; a small number are classified as undecodable and must not be counted as positively certified healthy tokens. Some candidates had entered Magikarp’s geometric candidate set but were rejected by its behavioral verification.

Table 4. Contextual candidates and Magikarp records. Counts describe the samples used in this comparison. “Rejected” refers to rejection of glitch status by the prior verification step. The Qwen2.5‑7B comparison uses 294 candidates, whereas the screening summary in Table 3 reports 296; these analysis populations are reported separately.

This shows that the contextual screen identifies behavior outside the available verified-glitch sets. It is not evidence that all contextual candidates are actual copying errors, nor that other detectors would fail to identify them under their own assessment tasks. A common-oracle comparison is required to quantify that broader claim.

4.4 Demonstrated errors and output concentration

The motivating according case establishes an actual copy error despite near-certain success in isolation. Other stored cases show related behaviors across languages and model configurations. Table 5 gives representative observations. The screening score describes the token across its context bank; the final column describes a selected observed emission. These are different levels of evidence.

Table 5. Selected contextual copying specimens. Probabilities are rounded as recorded. A score S summarizes threshold flags and is not necessarily the fraction of verified decoding errors. The table is illustrative and does not estimate failure-mode frequencies.

Across the seven Qwen configurations, the fraction of selected stored failing specimens with entropy below 2 bits ranges from 14% to 74% (Appendix B). Some errors therefore occur under highly concentrated distributions, while others occur under diffuse distributions. These records support the existence of confident errors but do not support a universal claim that contextual errors are usually confident or never accompanied by uncertainty.

[AemonAlgiz]

6 Preservation in Generation and Tasks

6.1 Free generation and identity metrics

Free-generation probes ask whether the model retains a supplied token while describing or responding to it. The OLMo experiment requests exact identification and an explanation, followed by the assistant prefix Let me think about this carefully. Other configurations use the token as the user turn under the model’s chat template. Outputs are generated for up to 256 tokens. These framing differences are part of the protocol and preclude treating the resulting counts as a controlled scale comparison.

Two reproduction metrics are needed. The exact-ID metric asks whether the seed ID occurs anywhere in the generated IDs. The string metric asks whether the decoded seed string occurs anywhere in the output, allowing a different segmentation. Neither alone establishes correct identification or exact reproduction of an entire requested sequence. For Qwen2.5‑72B, for example, none of the 32 worst-isolation seeds reappear as their original IDs, while 12 reappear as strings. A seed such as _ghost can be written using multiple output tokens.

Table C1 reports candidate-versus-control reproduction for the configurations with available generation. The candidate string-preservation rate is lower than the control rate in four of five comparisons and equal on Qwen3‑32B. Exact-ID rates are less consistent: Qwen2.5‑7B ties, and Qwen3‑32B reproduces 8 of 14 candidates versus 7 of 14 controls. A candidate can also be reproduced while being misidentified. These outcomes support task-dependent vulnerability rather than a universal inability to emit the candidate token.

The 32 worst-isolation seeds used in separate generation probes are not interchangeable with context-screened glitch references. On Flash-Next, a teacher-forced check of these seeds assigns the expected token probability at least 0.5 in 71% of 768 cells, whereas the analogous rate on Qwen3‑235B is 0.1%. This disagreement is another reason to verify the behavior of a reference set in the relevant context.

I did not run free generation, task probes, or specimen continuations on the two FP8 mixtures because the evaluation stack took about seven seconds per generated token, and I ran this with my own money; I would prefer not to go bankrupt. Their next-token observations must not be read as generated reasoning traces. We also did not run candidate-free generation on Qwen2.5‑72B.

6.2 Instructions requiring lexical preservation

Five instruction templates require the model to preserve the target string: name a Python function, quote a document title, return a list item, correct spelling, and summarize while preserving proper nouns. Each class contributes ten selected tokens per model and five tasks, giving 50 episodes per class. Tokens are inserted by ID into the model’s chat template. Appendix C contains the full templates and per-task results.

The primary metric asks whether the answer contains the given token string. It is deliberately narrower than complete task correctness. It does not verify whether generated code is valid, whether a function name is a legal Python identifier, or whether a summary is faithful in other respects. The spelling task also invites normalization, so its preservation rate must be interpreted in light of its wording.

Table 7. Token-string preservation in simple instructions. Each row contains 50 episodes per class. C denotes selected contextual candidates, S stable controls, and G the model’s glitch-reference class. The reported interval for C − S is a bootstrap interval over episodes; repeated use of each token across tasks limits independence.

Candidates are preserved less frequently than controls in all five configurations. The episode-bootstrap intervals exclude zero for OLMo-2‑7B, Qwen3.8‑27B, and both Qwen3‑32B modes, but not for Qwen2.5‑7B. Coarse task completion is reported at 0.96 to 1.00 while lexical preservation varies, illustrating why successful-looking task structure is insufficient for this measurement.

The transcripts make the discrepancy concrete. Asked for a function named according, OLMo-2‑7B writes sum_list. In the spelling template, it changes according to correct. Asked for the second item when the target is NSLocalizedString, it returns LocalizableString. Qwen3‑32B changes realtà to reality in the spelling task. These examples preserve much of the requested task form while altering the designated content.

6.3 Simulated tool interactions

The tool experiment uses five assignments involving a target term: investigate a configuration key, prepare a background note, recommend whether to adopt a technology, explain a term, or check a release. Models can call search, open_page, and run_python for up to six rounds. Ten tokens per class again produce 50 episodes per configuration.

The environment is constructed around the supplied token. A search containing its string returns relevant documentation, release notes, and an issue page. Other searches return no relevant hits and generic material. This design isolates whether the target reaches a tool call, but it does not estimate retrieval failure in a natural search engine. Qwen models use their native tool templates; OLMo follows a textual protocol, which further limits cross-model comparisons.

Table 8. Preservation during simulated tool use. Values are the fractions of 50 episodes in which the target string appears. Intervals are the reported episode-bootstrap intervals for C − S. Plain and thinking configurations are shown separately.

Qwen3‑32B in plain mode preserves stable targets in 96% of first queries and final answers. Candidates appear in 74% of first queries and 68% of final answers. The glitch-reference targets appear in neither measure in any of the 50 episodes. Some responses proceed from generic retrieved material after the term disappears or changes in the first query.

Qwen3.8‑27B has a similar first-query contrast but a different completion pattern. In plain mode, glitch episodes finish within the round limit in 14% of cases, versus 66% for stable controls and 58% for candidates. Identical tool calls recur in 28%, 10%, and 14% of cases, respectively. Thinking mode changes these completion rates and also changes the stable baseline. These results show that losing a target can lead to either a completed generic answer or repeated tool use, depending on the configuration.

The experiment does not establish widespread fabrication of specific retrieved facts. The recorded version-number fabrication rates are only 0.00 to 0.02 per class and configuration, with isolated nonzero cells in both glitch and control-related groups (Table C4). The clearer measured effect is loss of the literal target in a query or answer. The selected token sets, repeated templates, and constructed environment limit generalization to autonomous systems operating on natural tasks.

6.4 A negative result on natural entity names

A separate experiment evaluates 1,255 real code identifiers and proper nouns in a natural task. Adding the fragility of constituent tokens does not improve prediction of an incorrect entity name after controlling for entity rarity, constituent-token isolation scores, and the model probability of the complete entity. The reported change in AUROC is −0.001, with a 95% interval of [−0.008, +0.006].

This result limits the practical interpretation of the synthetic task findings. The current evidence does not establish that the contextual score explains natural entity-name errors beyond these existing predictors. It also does not show that rarity causes either outcome, or that it has excluded every possible relationship between token fragility and hallucination. The appropriate conclusion is that this experiment shows no incremental predictive power.

7 Auxiliary Analyses

7.1 Pair interactions and repetition

A factorial copying experiment tests whether pairs of tokens produce damage beyond the contributions of their members. Carriers contain two slots at separations 2, 4, 8, 16, 32, 64, and 128, with two independent filler sequences at each separation. Each ordered pair is evaluated in four conditions: both targets, either target with a control, and two controls. The interaction contrast subtracts the two single-target scores from the joint score and adds the control score.

The screen uses an additive fit within each carrier, standardized residuals, within-carrier permutations, and Benjamini-Hochberg correction at q = 0.05 (Benjamini and Hochberg, 1995). Its declared count criterion compares the number of rejected pairs with the 95th percentile from twenty simulated additive-null datasets. No tested pool exceeds that criterion: the rejected-pair counts are 1 of 4,032 for each OLMo pool, 13 of 1,190 for Qwen3‑32B, and 1 of 2,256 for Qwen3‑235B (Appendix D).

This finding is narrower than the absence-of-interaction case. The maximum-statistic tests are significant for the OLMo pools and Qwen3‑32B, while Qwen3‑235B has p = 0.059. The two criteria assess different aspects of the interaction distribution and must both be retained. Mean repeated-token contrasts are positive in the reported tests, consistent with facilitation in these carriers. Neither result establishes universal additivity, the absence of rare harmful combinations, or a causal account based on induction.

7.2 Recoverability of token identity

The Pythia-1.4B experiment places 36 verified glitch tokens and 1,088 controls at the ends of eight contexts. At each layer, the model matches a token’s contextual state to its state in another context within a pool of 1,124 candidates, after removing the component shared within each context. Biderman et al. (2023) describe Pythia.

Identity-matching accuracy is 1.000 for glitch tokens from the embedding layer through layer 12 and 0.988 to 1.000 for controls. At the final layer, it is 0.880 and 0.874, respectively. These results show that substantial token-identity information remains recoverable under the chosen comparison procedure. A logit-lens example also favors a continuation of a glitch token, idepress followed by ant, rather than reproduction of the input token itself.

Decodability does not causally localize copying failure. The experiment does not show that the model uses the recoverable information in the copying computation, that identity is perfectly preserved, or that the output layer alone causes the error. It also concerns independently labeled glitch tokens on Pythia rather than the full candidate population in the eight-model study. Causal interventions would be needed to distinguish inadequate readout, context-dependent competition, and other explanations.

8 Discussion and Limitations

The principal finding is a repeatable distinction between isolated copying success and contextual susceptibility. Shared context banks make it possible to estimate a per-token tendency without selecting an adversarial context for each token. Actual decoded examples establish that at least some of these discrepancies are literal copying errors, and the task experiments show that selected candidates can lose their identity beyond the copying template. Embedding associations connect this population to prior glitch-token research, while their variable incremental value prevents treating geometry as a complete explanation or a validated universal detector.

Several limits determine the scope of that conclusion. First, the aggregate candidate class is defined by a probability threshold whose relationship to true decoding error is incomplete. Strict-gate results and inspected continuations strengthen the case for the phenomenon, but they do not replace a population-wide argmax-error analysis. A revised behavioral prevalence estimate must retain both the isolation outcome and contextual output identity for each cell.

Second, the common-word carriers are artificial, and the model-dependent prompt framing affects what is measured. Direct ID insertion controls the experimental identity of the target but can create inputs unlike normal tokenizer output. String-preservation tasks partially address this concern, yet the study’s strongest natural-entity test is negative after existing controls. There is no basis here to extrapolate candidate prevalence to everyday text or to treat a score as a context-independent property of a token.

Third, eight copying configurations span only Qwen and OLMo, with differing precision and architecture. The auxiliary Pythia probe addresses another question. Generation coverage is incomplete, and some task classes contain only ten selected tokens. Episode-level bootstrap intervals do not fully account for dependence among five tasks using the same token. The reported intervals should therefore be read as descriptive evidence, with token-clustered or hierarchical uncertainty estimates needed for stronger population inference.

Fourth, the geometric analyses evaluate associations with screen labels. Reference construction partly uses contextual behavior, and selection of the stronger embedding space is retrospective. Random stable references do not provide a rarity-matched or density-matched null. Static feature improvements occur on three models, while several token-ID baselines are already strong. The UMAP panels provide qualitative evidence about the plotted population; they do not measure cluster coverage in the original space. The paper reports no fixed-budget prospective discovery experiment or common-oracle comparison with existing detectors.

Finally, the study does not identify which ordinary context will trigger a particular token’s error. Split-half prediction estimates average susceptibility under related context banks. Small context main effects do not eliminate token-context interactions, and the factorial experiment retains significant maximum-statistic results. Neither recoverable identity nor geometric proximity establishes a training mechanism. The evidence supports a measurement target and a set of associations; mechanistic explanation and context-specific prediction remain open.

The most direct methodological extension is to freeze the isolation test, candidate score, reference construction, and context distribution before evaluation on disjoint data. Reporting target-position argmax error, full-sequence copying accuracy, and exact-string preservation alongside probability scores would connect screening sensitivity to observable failures. For discovery claims, precision and recall at a fixed verification budget would provide a more practical test than retrospective AUROC alone.

9 Reproducibility and Reporting Scope

Reproducibility requires the model and tokenizer revisions, sampled token IDs, complete context IDs and insertion slots, prompt framing, thresholds, precision, and scoring code. The procedure does not modify model weights. Appendix A provides public model identifiers and textual prompts; the run-specific matrices and environment metadata described in Section 3.4 are also necessary. Figure 2 additionally requires the plotted IDs and class masks, embedding preprocessing, UMAP parameters, and random seeds. Generation coverage is limited to the configurations explicitly listed in the corresponding tables.

Analysis populations differ. Table 3 reports 296, 418, and 684 candidates for Qwen2.5‑7B, Qwen2.5‑72B, and Qwen3‑235B, while Table B1 reports 294, 417, and 682. Each analysis retains its associated counts. Half-bank prediction counts differ again because their labels use different contexts. Reconciling the remaining sample transitions exactly requires the underlying token manifests and exclusion masks.

The recorded protocol includes amendments to the isolation gate, sample size, printable-seed filtering, and string-level reproduction scoring. Vocabulary sampling follows embedding-row counts, so padding reduces token-ID overlap between Qwen2.5 and Qwen3 samples. Matched-ID scaling within Qwen2.5 remains untested. These sampling and framing decisions form part of the experimental specification and constrain cross-model interpretation.

10 Conclusion

FragileTokens studies contextual copying vulnerability among tokens that pass an isolation probe. Across eight configurations, screening scores are reproducible and decoded specimens confirm errors. Geometric proximity often ranks candidates above controls, with gains that vary across surface features. Selected candidates also show reduced lexical preservation in several tasks, while the natural-entity test finds no added predictive value. These findings support contextual verification and motivate direct evaluation of behavioral prevalence, prospective discovery, and individual triggering contexts.

References

Benjamini, Yoav, and Yosef Hochberg. 1995. Controlling the False Discovery Rate: A Practical and Powerful Approach to Multiple Testing. Journal of the Royal Statistical Society, Series B, 57(1):289–300. Publisher DOI.

Biderman, Stella, et al. 2023. Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. Proceedings of the 40th International Conference on Machine Learning. Paper.

Land, Sander, and Max Bartolo. 2024. Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Models. Proceedings of EMNLP 2024, pages 11631–11646. ACL Anthology. Code and token records: Magikarp repository.

Li, Yuxi, Yi Liu, Gelei Deng, Ying Zhang, Wenjia Song, Ling Shi, Kailong Wang, Yuekang Li, Yang Liu, and Haoyu Wang. 2024. Glitch Tokens in Large Language Models: Categorization Taxonomy and Effective Detection. Proceedings of the ACM on Software Engineering, 1(FSE). Publisher DOI. Full preprint.

McInnes, Leland, John Healy, and James Melville. 2018. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction. arXiv preprint arXiv:1802.03426. Paper.

OLMo Team. 2025. 2 OLMo 2 Furious. arXiv preprint arXiv:2501.00656. Paper.

Qwen Team. 2025a. Qwen2.5 Technical Report. arXiv preprint arXiv:2412.15115. Paper.

Qwen Team. 2025b. Qwen3 Technical Report. arXiv preprint arXiv:2505.09388. Paper.

Qwen Team. 2026a. Qwen3.8‑27B. Official model card. Model card.

Qwen Team. 2026b. Qwen3.8-Flash-Next-FP8. Official model card. Model card.

Rumbelow, Jessica, and Matthew Watkins. 2023. SolidGoldMagikarp (Plus, Prompt Generation). LessWrong, February 5. Original report.

Tang, Kunsheng, Peigui Qi, Yide Song, Wenbo Zhou, Zhicong Huang, Qing Guo, Tianwei Zhang, Weiming Zhang, Nenghai Yu, and Jie Zhang. 2026. One Bad Token Spoils the Barrel: Assessment, Detection, and Remediation of Glitch Tokens in Large Language Models. 35th USENIX Security Symposium. Conference page. Full paper.

UMAP developers. n.d. Using UMAP for Clustering. UMAP documentation. Accessed September 11, 2026. Documentation.

Wu, Zihui, Haichang Gao, Ping Wang, Shudong Zhang, Zhaoxiang Liu, and Shiguo Lian. 2026. GlitchMiner: Mining Glitch Tokens in Large Language Models via Gradient-based Discrete Optimization. Proceedings of the AAAI Conference on Artificial Intelligence, 40(40):33998–34005. Publisher DOI.

Yu, Sangwon, Jongyoon Song, Heeseung Kim, Seongmin Lee, Woo-Jong Ryu, and Sungroh Yoon. 2022. Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddings. Proceedings of ACL 2022, Volume 1: Long Papers, pages 29–45. ACL Anthology.

Zhang, Zhibo, Wuxia Bai, Yuxi Li, Mark Huasong Meng, Kailong Wang, Ling Shi, Li Li, Jun Wang, and Haoyu Wang. 2024. GlitchProber: Advancing Effective Detection and Mitigation of Glitch Tokens in Large Language Models. Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. Publisher DOI.

[John6666]

For now, I just did a small check:

---

I think the distinction you are making between passing an isolation probe and remaining reliable in context is useful. I tried a small independent sanity check on Qwen3-1.7B, mainly to understand the relationship between your probability screen and actual target-position copying errors.

This was not a replication of your reported prevalence: I used an independent prompt/context bank and a much smaller experiment. But the main phenomenon was easy to reproduce qualitatively — tokens that copied correctly in isolation could still lose the target token under contextual copying.

The part that became clearest to me is the role of the probability threshold.

Your contextual screen is

ln q(t,c) < -0.5

or approximately

q(t,c) < 0.607

If the expected token is not the greedy argmax at that position, its probability must be at most 0.5: if some other token has a strictly higher probability, both cannot exceed 0.5.

So at the individual cell level,

target-position greedy error
        =>
contextual screen-positive

under this particular threshold.

The reverse is not true: a token can have, say, probability 0.55, be screen-positive, and still be the correct greedy output. That is exactly the distinction you already make between the probability screen and behavioral FragileTokens.

In my small check, over 6,144 token × context cells:

• 229 cells were screen-positive;
• 182 had an actual target-position argmax error;
• all 182 argmax errors were inside the screen-positive set;
• 47 screen-positive cells still had the correct greedy target.

So I think the population-wide argmax analysis you mention in the limitations would be particularly informative — but perhaps slightly more specifically as a calibration / behavioral-prevalence layer on top of the existing screen, rather than mainly as a test of whether the screen catches greedy errors.

If the necessary top-1/logit data are already stored, this might even be a cheap additional analysis rather than a new experiment.

A useful first table might simply separate:

                         target argmax correct    target argmax wrong
screen-negative                  ...                    ...
screen-positive                  ...                    ...

and then, at the token level, compare the existing susceptibility score S(t;C) with the actual target-position error rate.

There is a second distinction at the token level. In my small run I had:

screen candidates (S >= 0.10):              23
tokens with argmax-error rate >= 0.10:       20
tokens with at least one argmax error:        35

Those numbers should not be compared to your candidate prevalence, because my contexts and framing were different. But the difference itself illustrates a measurement choice that may be worth making explicit: a token that produces one or two real contextual errors can still fall below the S >= 0.10 candidate threshold.

So there are really several useful populations:

isolation pass
    |
    +-- probability-screen positive
    |
    +-- at least one verified target-position error
    |
    +-- behavioral error rate >= declared threshold

None is inherently the “correct” definition; they answer slightly different questions.

A cheap complementary view: change from isolation

One additional analysis I would be curious about is the within-token change

delta_log_q = log q(context) - log q(isolation)

alongside the absolute contextual threshold.

You already point out an important edge case: a token with probability 0.55 in isolation and 0.55 in context passes the primary isolation gate and is context-screen-positive, even though the context did not make it any worse.

So I would not replace the current screen with a delta measure — the current screen asks a valid question about contextual reliability — but the two quantities separate different things:

absolute contextual q
    -> "Is the expected token weak enough here to trigger the screen?"

change from isolation
    -> "Did this context actually degrade this token relative to its own baseline?"

In my small run, the median contextual change was close to zero while the mean change was negative. Qualitatively, it looked less like “context makes every token slightly worse” and more like “most token/context pairs remain fairly stable, while a smaller subset drops much more sharply.”

I would treat that only as a small-run observation, not a mechanism claim, but it made me think that showing the distribution of delta_log_q for candidates versus controls could be useful.

It might also make the motivating according example even easier to place: that example is not merely below an absolute probability threshold; it undergoes an enormous within-token change from almost certain isolation success to an almost certain wrong continuation.

What predicts the individual triggering contexts?

This still seems like the most interesting unresolved part to me.

Your split-half results suggest that some tokens have a reproducible average susceptibility, but that does not tell us why a particular context triggers a failure while another context preserves the same token.

There is some adjacent copying literature that may provide useful carrier-side features to test, without implying that it already explains FragileTokens.

For example, Born a Transformer — Always a Transformer? studies copying/retrieval behavior and induction-like structure in pretrained Transformers.

More directly, the recent Frontier Language Models Struggle to Copy argues that exact copying can fail even well inside the context window and presents evidence for a shortcut based partly on matching local context rather than simply identifying an exact positional source. Its reference implementation also includes copy-task generation and evaluation code.

That initially made me wonder whether local repetition / ambiguous predecessor structure might predict your triggering contexts.

However, I tried a small matched manipulation and got the opposite of the simple version of that hypothesis: adding an earlier copy of the predecessor with a competing successor did not make the target less reliable on average; the effect actually leaned toward facilitation.

That seems compatible with your own factorial experiment, where the mean repeated-token contrast was not simply harmful either.

So I would not infer:

repetition / local ambiguity -> FragileToken failure

from the neighboring copying literature.

A more discriminating follow-up, if you ever wanted to isolate that axis, could use three matched conditions:

A. unique predecessor

B. repeated predecessor
   + same / reinforcing successor

C. repeated predecessor
   + competing successor

That would separate two effects which are otherwise confounded:

benefit from seeing the local pattern before
vs.
cost from having multiple possible continuations

If B improves copying while C degrades it, successor competition is informative.

If both B and C improve copying, repetition/context familiarity is probably dominating.

If neither explains the fragile/stable difference, that itself is useful evidence that generic copy ambiguity is not sufficient to explain the token-level susceptibility.

I would rank this below the population-wide argmax and delta_log_q analyses, simply because those can probably reuse your existing matrices and answer a more direct question first.

How this seems to connect to existing glitch-token work

I think the narrow framing in the paper is important here.

The broad observation that one simple repetition test cannot completely characterize a token is already connected to existing glitch-token work.

For example, Tang et al.'s GlitchQuiz / One Bad Token Spoils the Barrel evaluates tokens over multiple behavioral tasks rather than treating one repetition prompt as a complete certificate. Their released code makes that multi-task structure quite explicit.

And Fishing for Magikarp focuses on another related but narrower source of failure: vocabulary entries that are under-trained because tokenizer construction and model-training frequency do not line up.

So, to me, the particularly clean contribution of the FragileToken setup is not just:

"a token can pass one simple test and fail another test"

but more specifically:

condition on tokens that pass a stated isolation-copying probe
        ->
hold the copying operation fixed
        ->
place the same token into a shared context distribution
        ->
estimate repeatable token-level contextual susceptibility

That is a useful experimental separation because it changes one axis — surrounding context — while keeping the target identity and basic copying task fixed.

It also seems wise that the paper does not claim that all of these candidates share the same mechanism, or that previous glitch-token detectors necessarily miss them. A common behavioral oracle would be needed for that stronger comparison.

Direct token IDs and canonical tokenization

I also checked the possible “this is just a non-canonical direct-ID sequence” explanation, because direct token insertion is both a strength and a possible confound here.

The strength is obvious: it lets you hold the token ID fixed instead of allowing ordinary tokenization to silently change the object being tested.

The limitation, as you note, is that arbitrary ID sequences can sometimes describe a surface string differently from the tokenizer’s canonical encoding.

There is some relevant evidence in Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations and its reproduction repository: non-canonical tokenization is not automatically catastrophic, especially for instruction-tuned models, but robustness varies with how far the tokenization moves away from the canonical form.

In my small probe:

single target tokens canonical:       100%
constructed test sequences canonical: ~99.8%

and the few non-canonical cells were not where the observed copying failures concentrated.

So at least in that independent check, the contextual errors could not be explained simply as a consequence of feeding non-canonical tokenizations.

I still think recording a boolean such as

decode(ids) -> re-encode -> identical IDs?

is a useful cheap covariate in a released run artifact, because it lets future readers split that branch cleanly instead of arguing about it abstractly.

A compact decision tree for follow-up analyses

If I were choosing the next analysis purely by information gain / cost, I would probably use something like this:

Do the saved contextual runs retain target-position top-1 or logits?
|
+-- YES
|    |
|    +-> compute screen-positive vs actual argmax-error counts
|    +-> compute per-token argmax-error rate
|    +-> compare S(t;C) against that behavioral rate
|
+-- NO
     |
     +-> rerun one representative configuration first
         rather than immediately repeating all eight

Then:

Are q0(t) and q(t,c) both retained?
|
+-- YES
|    |
|    +-> add delta_log_q = log q(t,c) - log q0(t)
|    +-> inspect its distribution within candidates and controls
|
+-- NO
     |
     +-> the current absolute screen can remain the primary analysis;
         this does not require redesigning the study

And only after that:

Are the actual carrier IDs and insertion positions retained?
|
+-- YES
|    |
|    +-> compare successful and failing contexts within the same token
|    +-> test carrier-side predictors
|    +-> if needed, build controlled matched contexts
|
+-- NO
     |
     +-> a small reproducibility artifact containing those fields
         may have more future value than another large model sweep

For the carrier analysis, I would keep the model simple initially. Position, local n-gram recurrence, number of competing observed successors, and similar directly measurable properties are probably enough for a first pass.

I would only move to mechanistic explanations — induction heads, copy-suppression circuits, activation geometry, etc. — after seeing a behavioral feature that actually predicts the individual failures. There are several plausible mechanisms, but right now I do not think the available evidence selects one.

A few reporting distinctions that seem worth preserving

One thing I liked in the paper is that it already keeps several levels of evidence separate. I would preserve that very explicitly, because they are easy for later readers to collapse together:

screening prevalence
    !=
verified target-position greedy-error prevalence
    !=
full-sequence copying-failure prevalence
    !=
failure prevalence in ordinary natural text

The natural-entity negative result is especially useful here. As I read it, the current result is:

adding the token-fragility feature
did not add predictive value
for that natural-entity experiment
beyond the existing controls

rather than either of the much stronger conclusions:

"FragileTokens explain natural entity errors"

or

"FragileTokens can never matter in natural tasks"

Keeping that negative result prominent makes the practical claim much easier to interpret.

A similar caution applies across model configurations. Differences in candidate rates are interesting descriptive observations, but with different architectures/training/framing/precision conditions they are not by themselves a scaling law.

For the downstream task experiment, there is also one small statistical sensitivity check that may be cheap if the episode-level data are already available: because multiple observations come from the same token, a token-clustered bootstrap (or another token-level resampling analysis) could complement an episode-level bootstrap. I would consider that a secondary robustness check, not a prerequisite.

Overall, my small check did not give me a reason to doubt the basic isolation-versus-context distinction.

If anything, it made me think the most informative next layer is fairly concrete:

probability screen
        ->
actual target-position error rate
        ->
individual triggering-context predictors

I would probably do those in that order.

The first step may already be nearly free if the target-position logits/top-1 outputs are stored, and it would give the screening prevalence a much more direct behavioral interpretation without changing the existing experimental design.

The triggering-context question looks harder, but also potentially more interesting. My small attempt suggests that a simple “repetition/ambiguity causes the failure” explanation is probably too coarse, so I would keep that branch open rather than committing to one copy-failure mechanism yet.

[AemonAlgiz]

Hello John,

Apologies for the late response. This is phenomenal feedback, and I have been incorporating it into the paper. I will be updating it shortly!

Thanks,
Jeffrey