Hmm… Maybe the symptoms get easier to read if we split them apart:
My short version is: the basic formulation looks reasonable to me. For what you described, I would also think of this as continued pretraining / domain-adaptive causal language modeling, not instruction tuning. Raw novel text with full next-token loss is a normal formulation, and I would not convert the corpus into fake user/assistant pairs just because many fine-tuning examples happen to use chat data.
The part I would change first is not necessarily the training objective, rank, or even the number of steps. I would first separate a few things that can currently produce very similar-looking generation failures.
Short answers to your seven questions
| Question |
My current take |
| 1. Raw CLM on Qwen3.5-4B-Base? |
Yes. Raw token sequences with causal-LM loss are appropriate for this goal. A chat template is not required for prose continuation. Current TRL SFTTrainer documentation explicitly supports ordinary plain-text language-modeling datasets as well as conversational ones; the trainer class being named SFTTrainer does not make the objective instruction tuning. |
| 2. CPT rather than SFT? LoRA or full FT? |
I would call the objective continued pretraining / domain adaptation. LoRA is a valid way to attempt it; full-parameter CPT is not a prerequisite for getting an effect. But LoRA and full CPT are not equivalent in capacity, so full FT remains a later control if a well-validated LoRA run plateaus. The Qwen3.5-4B-Base card explicitly positions the Base checkpoint for fine-tuning/research rather than direct interaction. |
| 3. Concatenating novels? |
Reasonable as one training condition, but ⁂ is only a token separator; it does not reset attention/Gated-DeltaNet state inside the same sequence. Whether cross-book context is useful or noise is an empirical choice here. I would compare your current continuous stream against a no-cross-book condition rather than assume either is universally correct. |
| 4. LoRA targets? |
Your target set looks quite broad and structurally sensible, not obviously too narrow. In a small check against the current 4B text model, those suffixes covered essentially all transformer-block nn.Linear projections. I would not make “missing ordinary linear projections” my first hypothesis. |
5. r=32, alpha=64? |
Plausible. I would not expect 32 -> 64 by itself to diagnose this, especially since you already saw little consistent improvement from changing rank. There is even a public Qwen3.5 raw-CPT example, FuseLLM-9B, using the same r=32, alpha=64, dropout=0.05 pattern and essentially the same Qwen3.5 projection families — very different corpus/scale, so precedent rather than a recipe. |
| 6. 256 optimizer steps? |
“256” alone is not very informative. With 2048 × grad_accum 4 × 256, you present about 2.1M token positions before accounting for padding/details — already around two passes over a ~1M-token corpus. I would reason in tokens seen / corpus passes / checkpoint curves, not optimizer-step count alone. |
| 7. Recommended raw-text route? |
My default would be: raw CLM → explicit document-boundary choice → book-level held-out evaluation → identical Base-vs-adapter raw-prefix tests → then precision/optimization ablations. I would only start sweeping rank/LR/epochs after those cheaper branches are separated. |
So, before changing the training recipe much, my first pass would be roughly:
1. Verify what actually ran
- Transformers / PEFT / bitsandbytes versions
- actual model class
- actual dtype
- active adapter + targeted modules
2. Evaluate Base and adapter on exactly the same RAW prefixes
- no chat template
- same decoding settings
- preferably greedy/deterministic first
3. Split the success criteria
- prose/style adaptation
- local continuation
- corpus-specific facts
- >2048-token narrative consistency
- general-domain narrowing
4. Compare one document-boundary control
current continuous stream
vs
no cross-book blocks
5. If possible, compare one precision control
NF4 QLoRA
vs
16-bit LoRA
6. Only then spend runs on
LR / rank / number of corpus passes
That order is mainly because each early check eliminates several explanations at once.
A useful symptom map might be:
| Observation |
What I would suspect next |
| Held-out novel loss barely changes |
adapter/runtime, optimization, precision, or insufficient adaptation |
| Held-out novel loss improves, but generation does not |
generation serialization/sampling/evaluation path |
| Local continuation improves, but long narrative consistency does not |
training-window / long-context objective mismatch |
| Novel metrics improve, while unrelated prompts collapse toward similar themes |
over-specialization/narrowing, not necessarily under-training |
| 16-bit LoRA works substantially better than NF4 under otherwise matched conditions |
quantization becomes a serious suspect |
| Continuous-stream and document-isolated runs differ strongly |
boundary semantics are materially affecting the learned distribution |
1. Why I think the raw causal-LM formulation is basically right
For the objective you described, I do not see a reason to invent a chat format.
Conceptually:
labels = input_ids.clone()
labels[attention_mask == 0] = -100
is the normal full-sequence causal-LM objective.
Your masking choice is also sensible in principle: masking using the attention mask / actual padding positions avoids accidentally masking a real EOS just because PAD and EOS happen to share an ID.
Current TRL documentation is useful here because it makes a distinction that is easy to miss: SFTTrainer accepts both conversational data and standard plain text. For a pre-tokenized LM dataset, supplied labels are used directly; otherwise the language-modeling path can construct labels from the input IDs.
So I would separate:
trainer implementation:
SFTTrainer / Trainer / custom loop
from
learning objective:
full next-token loss on raw text
The second one is what determines whether this is behaving like raw-text continued pretraining.
I would therefore keep the raw formulation unless another experiment gives a concrete reason to change it.
The one place where chat formatting matters is evaluation. Since the model was adapted on raw prose, I would establish a raw-completion baseline before evaluating it through any chat wrapper:
prefix = "..."
x = tokenizer(
prefix,
add_special_tokens=False,
return_tensors="pt",
).to(model.device)
out = model.generate(
**x,
do_sample=False,
max_new_tokens=200,
)
Then compare Base and adapter using exactly the same token IDs.
That avoids accidentally turning:
novel prefix -> continuation
into a different task such as:
system/control markup
user message
assistant-generation marker
novel prefix
before you even know whether the CPT itself worked.
2. I would verify the runtime before tuning more hyperparameters
Qwen3.5 is new enough that I would record the actual software path, not just the intended configuration.
At minimum:
import transformers, peft, bitsandbytes
print("transformers:", transformers.__version__)
print("peft:", peft.__version__)
print("bitsandbytes:", bitsandbytes.__version__)
print(type(model))
print("model dtype:", getattr(model, "dtype", None))
print("embedding dtype:", model.get_input_embeddings().weight.dtype)
And after applying/loading the adapter, recent PEFT versions have very useful inspection helpers:
print(model.get_model_status())
and, if needed:
for x in model.get_layer_status()[:20]:
print(x)
The PEFT troubleshooting docs document these specifically for checking active/available/merged adapters and inconsistent states.
This is particularly cheap here because the symptom “the Base prior seems to dominate” is also what you would see if the adapter were not active in the generation path you think it is.
There have also been some real Qwen3.5-specific Transformers changes. For example, dtype could be silently ignored when Qwen3.5 composite checkpoints were loaded through AutoModelForCausalLM in the affected v5.9/v5.10-era path; that was subsequently fixed. That does not mean you hit that bug, but it is a good example of why I would print the resulting dtype rather than infer it from the arguments.
Likewise, if your GPU does not support BF16 and your configuration falls through to FP16, I would keep an eye on grad_norm. Current Unsloth code contains a Qwen3.5-specific float32 fallback because its Gated DeltaNet path has shown FP16 backward instability in some configurations. Again, that is tooling evidence rather than proof about your plain Transformers/PEFT run; it just makes the actual dtype a worthwhile branch to record.
3. Document boundaries: separator token vs actual isolation
I think this is one of the most useful design separations in your post.
Your current stream is effectively:
...Novel A...
⁂
...Novel B...
If those tokens are inside one 2048-token training sequence, the model is trained to predict the start of Novel B while conditioned on the end of Novel A.
A visible separator does not mean:
reset model state here
It only gives the model a lexical cue saying “something ended here”.
Even inserting the model’s EOS token is still conceptually different from a hard state/attention reset: EOS marks a boundary in the token stream, while separate examples / correctly boundary-aware packing control information flow.
So I see three reasonable experiments rather than one “correct” packing scheme:
A. Current continuous stream
book A -> separator -> book B
B. Hard no-cross-book blocks
split/chunk each book independently
never create a training block spanning two books
C. Efficient packing of independent examples
multiple examples share computation
but the model receives explicit sequence boundaries
For your small corpus I would probably test A vs B first, because that changes only one concept and does not require a complicated packing stack.
Qwen3.5 makes option C slightly more important to implement carefully. It is a hybrid model with Gated DeltaNet / causal-convolution layers as well as full attention. Current Transformers padding-free training documentation explicitly warns that position_ids alone are insufficient to isolate packed samples for GDN/causal-convolution models; boundary metadata such as seq_idx needs to reach those layers.
There was also a concrete historical issue for Qwen3.5 packing on Transformers 5.2.0–5.8.1, where GDN state could leak across independent packed samples while the loss still looked normal. The relevant upstream path was fixed in 5.9.0+.
I would not diagnose your present setup as that bug, though. You described manually constructing one continuous token stream, so cross-book conditioning is currently intentional from the model’s point of view.
The useful question is instead:
Does allowing cross-book context help the distribution I want, or add noise?
That is a very clean A/B test.
4. Your LoRA target list looks broad; I would not start by shrinking or enlarging it blindly
Qwen3.5 is unusual enough that the module list is worth checking, but the names you showed are not obviously suspicious.
For the current Qwen3.5-4B text model, the transformer blocks contain approximately:
8 full-attention layers:
q_proj
k_proj
v_proj
o_proj
24 Gated-DeltaNet layers:
in_proj_qkv
in_proj_z
in_proj_b
in_proj_a
out_proj
32 MLPs:
gate_proj
up_proj
down_proj
I ran a small structural check against the current public 4B Base checkpoint using the text-only AutoModelForCausalLM path. Your suffix list matched 248 transformer-block torch.nn.Linear modules out of 249 linear modules in the text model; the remaining one was lm_head.
That is already very close to the usual QLoRA idea of adapting the transformer’s linear layers. PEFT’s documented architecture-independent spelling for that is:
LoraConfig(
target_modules="all-linear",
...
)
See the PEFT LoRA documentation.
So all-linear would still be a useful sanity-control because it removes custom detection logic, but I no longer think it is likely to reveal a large forgotten set of ordinary Qwen3.5 text projections if your detector is doing what the excerpt suggests.
There is also public precedent on both sides:
- many ordinary LoRA examples target attention/MLP projections;
- FuseLLM-9B, which does raw-code continued pretraining on Qwen3.5, includes the Gated-DeltaNet projections as well.
That makes your broad target list look quite defensible.
One caveat: “all linear projections” is not the same thing as full-parameter CPT.
Qwen3.5’s GDN layers also contain state-dynamics/convolution/norm parameters that ordinary linear LoRA does not modify. I would not jump to adapting those exotic parameters — I have not seen evidence that this is necessary for novel CPT — but it is useful to remember when comparing PEFT with full continued pretraining.
5. Rank, learning rate, and training duration are three different questions
I would avoid treating these as one “more training” knob.
Rank
r=32, alpha=64 is not obviously tiny here.
With the broad target set above, r=32 already gives a substantial adapter. In my small structural check it corresponded to roughly 65M trainable LoRA parameters.
That does not prove capacity is sufficient, but it makes:
maybe rank 32 simply cannot learn prose
a fairly weak first diagnosis.
It also fits your own observation that increasing/decreasing rank did not cleanly remove the problem.
There is broader evidence that LoRA and full fine-tuning are not equivalent: LoRA Learns Less and Forgets Less found lower learning capacity for conventional low-rank LoRA than full FT in much larger math/code continued-pretraining experiments, while LoRA preserved more of the original model. Those conditions are vastly larger than your ~1M-token novel corpus, so I would use that paper as a caution about equivalence, not as evidence that your rank is wrong.
Learning rate
7e-5 also does not look obviously absurd to me for a broad LoRA run, but I would not try to infer the correct value from one final generation.
If optimization still looks suspicious after the cheaper controls, a small bracket such as:
lower
current
higher
with held-out loss measured at the same token budget tells you much more than another rank-only sweep.
Training amount
With your example:
2048 tokens / microbatch
× 1 sample
× 4 gradient accumulation
× 256 optimizer updates
≈ 2.10M token positions
So if the train corpus is roughly 1M tokens, you are not in an obvious “the model only saw a tiny fraction of the corpus” regime.
I would checkpoint by corpus exposure, for example conceptually around:
0.5 pass
1 pass
2 passes
4 passes
...
and plot:
train NLL
held-out-novel NLL
a few fixed raw-prefix generations
a few generic canary prompts
That last line matters because one of your symptoms is:
unrelated prompts sometimes converge toward similar topics
That can mean too little useful learning, but it can also mean the opposite: the adapter may already be pulling the model too hard toward a narrow small-corpus basin.
If novel NLL improves while generic behavior becomes progressively narrower across checkpoints, I would interpret that very differently from “the adapter failed to learn the corpus”.
An earlier checkpoint could then be better even though its training loss is higher.
6. I would split 'learn the novel' into four measurable goals
I think this is the biggest conceptual separation.
Your list combines:
1. prose/style distribution
2. local next-token continuation
3. corpus facts: characters / settings / relationships
4. long-range narrative coherence
They are related, but success on one does not guarantee success on the others.
A. Style / domain distribution
A very cheap metric is held-out novel NLL/perplexity:
Base model
vs
adapter checkpoint
on text from a book/story that was not used for training.
If the adapter clearly lowers held-out novel loss, it has learned something about the novel distribution even if free generation still looks messy.
B. Local continuation
Give both models the same raw prefix, with deterministic decoding first.
This tests the task closest to the training objective.
C. Corpus-specific knowledge
If the goal includes exact facts such as character relationships or obscure setting details, I would not use this as the only measure of CPT success.
Raw pretraining is not especially data-efficient for a fact that appears once or twice in a small corpus. Synthetic Continued Pretraining starts from exactly this problem: knowledge acquisition from a small collection of documents is difficult when each fact has very few distinct textual presentations.
That does not argue against your CPT idea. It just suggests separating:
"did the model adapt to this prose distribution?"
from:
"can it reliably retrieve every fact in these books from parameters alone?"
If the latter becomes important, augmentation, retrieval, or a second task-specific stage can be added without abandoning the prose-CPT stage.
D. Long-range narrative consistency
This is where max_seq_length=2048 becomes important.
Even though Qwen3.5-4B-Base has a much larger native context window, if training consists of independent 2048-token forwards, a prediction in block N does not directly condition on the hidden state from block N-1.
So:
the complete book was included in the training corpus
is not the same statement as:
the model was trained on dependencies spanning the complete book
All of the text contributes to parameter updates, so you can still learn style, vocabulary, recurring motifs, characters, etc. But a specific dependency crossing a 2048 boundary is not presented as one causal context.
Work on continued training for long-context models likewise finds that the sequence lengths used during continued training matter for long-context behavior.
So if local prose continuation improves but long-distance character/world consistency remains weak, I would not immediately interpret that as LoRA failure. It may simply be a different training objective.
This is useful because it lets you keep the current inexpensive 2048 CPT for style/domain adaptation while treating long-range narrative memory as a separate design branch if you actually need it.
7. I would change the validation split depending on what you want it to mean
You mentioned holding out about 10% of the packed blocks.
That is perfectly usable for monitoring optimization, but there is one possible interpretation issue.
If the procedure is:
all novels
-> concatenate
-> make 2048 blocks
-> randomly split blocks into train/validation
then validation blocks can come from the same books — even immediately neighboring regions — as training blocks.
That tells you something like:
how well am I fitting unseen chunks from this same corpus distribution?
It does not cleanly tell you:
did the adapter generalize its learned prose/domain behavior to a held-out story?
For that second question, I would split first:
books/stories
|
+-- train books -> tokenize/pack
|
+-- held-out book(s) -> tokenize separately
For a very small corpus, you do not necessarily have to throw away a large fraction of the data permanently. You could keep:
- your current block-level validation for convenient training monitoring; and
- a small book-level anchor set for Base-vs-checkpoint comparisons.
The important part is just to label the two metrics differently.
8. QLoRA: I would test it, not blame it
General PEFT guidance absolutely supports NF4 QLoRA; for example the PEFT quantization guide recommends NF4 and broad linear targeting for standard QLoRA-style training.
So I would not say:
QLoRA is inappropriate for this task.
However, Qwen3.5 is new enough that I would run one precision control if hardware allows it.
The current Unsloth Qwen3.5 fine-tuning guide flags larger-than-normal Qwen3.5 quantization differences and recommends higher-precision LoRA where feasible. I would treat that as practical implementation guidance, not proof of a root cause.
I also did a small sanity probe on the public Qwen3.5-4B-Base checkpoint using several synthetic Korean-prose prefixes:
FP16 Base
vs
standard bitsandbytes NF4 Base
The NF4 result was measurably different, but not catastrophically different: the next-token distributions remained quite similar. A one-pass r=32 / alpha=64 NF4 QLoRA backward check also produced finite gradients.
That tells me two things:
"NF4 is obviously broken"
-> not supported by this tiny check
"NF4 makes no difference at all"
-> also too strong
So the informative experiment is still a matched:
same data
same seed
same sequence construction
same LoRA config
same token budget
NF4 QLoRA
vs
16-bit LoRA
If both behave similarly, you can stop spending time on the quantization branch.
If they diverge sharply, then the Qwen3.5 precision path becomes much more interesting.
9. A compact decision tree
If I wanted to debug this without turning it into a large benchmark project, I would use something like this:
START
|
|-- Does the adapter change held-out novel NLL?
| |
| |-- NO
| | |
| | +-- verify adapter active / actual dtype / targets
| | +-- inspect train-vs-val curve
| | +-- compare 16-bit vs NF4 if possible
| | +-- only then bracket LR / token budget
| |
| `-- YES
| |
| |-- Does raw-prefix local continuation improve?
| | |
| | |-- NO -> inspect generation serialization /
| | | sampling / checkpoint loading
| | |
| | `-- YES
| | |
| | |-- Only long narrative consistency weak?
| | | -> separate long-context branch;
| | | 2048 CPT did not train those
| | | dependencies directly
| | |
| | |-- Corpus facts weak?
| | | -> measure separately from style;
| | | rare-fact acquisition is a
| | | different problem
| | |
| | `-- Generic prompts narrow/collapse?
| | -> inspect earlier checkpoints /
| | domain over-specialization
| |
| `-- Does A/B document-boundary training change things?
| |
| |-- YES -> boundary semantics matter for this corpus
| `-- NO -> deprioritize packing design
|
`-- Only after the above:
rank / LR / more passes / exotic target experiments
This is why I would not start with a big r=16/32/64/128 × LR × epochs grid.
Several much cheaper checks can tell you which axis is worth spending GPU time on.
If you later post the full script/logs, the parts that would discriminate the most branches are probably just:
- exact package versions + model revision
- model load / quantization code
- PEFT construction and adapter-load code
- preprocessing / train-validation split / block construction
- one decoded block containing a book boundary
- actual model/adapter status + targeted module list
- generation function + exact serialized input + decoding config
- train/eval loss by checkpoint
But I would not wait for all of that before proceeding. The raw-CPT formulation itself looks reasonable enough that I would keep it, establish the simple Base-vs-adapter measurements above, and use those results to decide which branch deserves the next run.
The main thing I would avoid is using one free-generation impression to answer all of these at once:
Did it learn the style?
Did it memorize the corpus?
Did it learn long-distance narrative state?
Is the adapter active?
Is the model over-specialized?
Is NF4 hurting it?
Those can all produce “this doesn’t feel like the novels yet”, but they point to very different fixes.