I created a dataset called CC-FilteredCorpus where I’m trying to filter Common Crawl and extract the content as clean Markdown.
The goal is to preserve useful document structure—headings, lists, links, code blocks, tables, etc.—instead of flattening everything into plain text.
I’m hoping this can be useful for LLM pretraining and other NLP research where preserving the structure of web documents matters.
The dataset is at helloadhavan/CC-FilteredCorpus
For now, I did a quick check in Colab:
I think the basic direction here is worth exploring.
Preserving some of the original Web document structure instead of immediately flattening everything to plain text is not just a cosmetic choice: recent work on web-corpus construction suggests that the HTML extraction step can change which pages survive downstream filtering, and can matter especially for structured content such as tables and code. At the same time, I would be careful not to jump from that to “Markdown is inherently better than plain text” — that stronger claim still needs a controlled comparison.
I ran a couple of lightweight checks against the current CC-FilteredCorpus snapshot, mainly to see what kinds of failure modes are actually visible rather than guessing from the pipeline description.
The short version is:
| Check | What I observed | How I would use it |
|---|---|---|
| Exact source-record duplication | 153,470 rows, 147,927 unique WARC record_ids → 5,543 redundant rows (~3.61%) |
Add a final global uniqueness check after shard assembly |
| English filtering | In a deterministic 10k sample, two independent LID models both flagged ~8.87% as non-English | Treat this as a review-candidate rate, then re-check the LID stage/model/threshold |
| Raw HTML ↔ output spot-check | 10/10 selected pages were recovered from Common Crawl with matching payload digests; several showed structure loss and/or boilerplate retention | Evaluate main-content extraction separately from Markdown serialization |
There was also a positive result: in the first 5k sample, Markdown-like structure was present in a substantial fraction of the data — headings in ~61%, lists in ~34%, table-like syntax in ~11%, and fenced code in ~1.3%. So I would not describe the Markdown conversion as simply failing across the dataset. It looks more like there are particular boundaries worth tightening.
If I were iterating on this, my default order would probably be:
record_id uniqueness check after all shards are assembled.On the snapshot I checked:
record_id: 147,927record_id: 5,543record_id clusters: 5,543That is a cleaner signal than ordinary “same URL” or “same text” deduplication.
WARC-Record-ID is defined by the WARC 1.1 specification as a mandatory identifier that is globally unique for its period of intended use. So seeing the same WARC record ID twice in the derived dataset is good evidence that the same source record has entered the final assembled dataset twice, rather than merely two similar pages being classified as duplicates.
I would not infer the exact cause from the outside. It could be something around shard overlap, resume/checkpoint boundaries, regenerated shards, final concatenation/publishing, or simply the scope at which exact deduplication is applied.
But it seems cheap to guard against at the end of the pipeline, independently of whatever near-dedup method you use.
Conceptually, something as simple as:
SELECT
COUNT(*) AS rows,
COUNT(DISTINCT record_id) AS unique_record_ids,
COUNT(*) - COUNT(DISTINCT record_id) AS extra_rows
FROM final_dataset;
and then requiring extra_rows == 0 before publishing would catch this class of issue.
I would treat this as a final assembly invariant, not as a reason to redesign the MinHash/near-dedup logic.
The duplicate pattern was unusually clean:
record_id clustersThe duplicated rows were also not spread uniformly across all shards; a large fraction was concentrated around a small number of later shards. That makes a shard/resume/assembly boundary worth checking first, although I would still avoid calling that the root cause without seeing the generation code.
One slightly odd detail: some identical-record pairs had small differences in edu_score or perplexity:
edu_score driftI would not read much into that by itself. It may mean the repeated records passed through scoring separately, or there may be another benign implementation detail. The important part is simply that identical source records reached the final dataset twice.
I also ran two independent lightweight language-ID checks on a deterministic 10,000-row sample:
lid.176.ftz: 8.97% non-English candidateslangid.py: 9.58% non-English candidatesThe largest consensus groups included Portuguese, German, Spanish, Polish, Italian, Czech, and French, and manual inspection of the high-confidence examples showed plenty of pages that are plainly written in those languages.
So if the target is still an English-only / English-focused corpus, I think the LID stage is worth revisiting.
But I would not call 8.87% the dataset’s true language-error rate.
Real Web language identification is substantially messier than clean benchmark LID: short pages, boilerplate, multilingual pages, code, named entities, and mixed-script content all complicate classification. The recent CommonLID benchmark was created specifically around noisy real Common Crawl text and shows that systems evaluated on cleaner datasets can look much better there than they do on actual Web data.
So I would interpret the 8.87% as:
a fairly large, high-information candidate set that is worth checking,
not:
8.87% of the dataset is definitely mislabeled.
A few practical things that would make this easier to reason about are documenting:
If LID currently runs on raw/pre-extraction content, one useful branch would be to compare it with classification on the actual text that will enter training. That can reduce cases where menus, language selectors, footer text, etc. influence a decision about the main document.
For reference, the fastText language-identification models are useful lightweight baselines, but I would still use a small human sample for the final decision rather than treating any LID model as ground truth.
For the 10k audit, the largest exact fastText + langid agreements among non-English predictions were approximately:
| Language | Rows |
|---|---|
| Portuguese | 197 |
| German | 172 |
| Spanish | 143 |
| Polish | 108 |
| Italian | 66 |
| Czech | 31 |
| French | 25 |
| Dutch | 14 |
| Swedish | 12 |
| Chinese | 10 |
There were also smaller groups for Hungarian, Danish, Norwegian, Romanian, Japanese, Arabic, Vietnamese, Korean, Russian, etc.
Again, I would use these counts for triage, not as an evaluation benchmark. A real error-rate estimate would need labeled sampling.
This was probably the most interesting part to me.
A useful conceptual split seems to be:
raw HTML
|
v
main-content decision
|
+-- content that belongs to the document
| |
| +-- was the content retained?
| |
| +-- were useful structures retained?
| (headings, lists, tables, code, links, formulas...)
|
+-- navigation / footer / sidebar / cookie UI / boilerplate
|
+-- was it removed?
In other words, there are at least three different metrics hiding inside “clean Markdown”:
Main-content recall
Did the extractor accidentally remove useful document content?
Main-content structure fidelity
For content that should stay, did headings/lists/tables/code/links survive in a useful representation?
Boilerplate leakage
Did menus, cookie notices, related-post widgets, service-area lists, login UI, etc. survive into the training text?
Those should not be collapsed into one score.
For example, raw HTML may contain 200 links, but if 190 are navigation links, dropping them is a success, not a structure-preservation failure. Conversely, flattening the actual article’s section headings or code blocks is a different failure.
A very close precedent is SWEb. Their pipeline deliberately treats HTML → Markdown conversion and main-content extraction as separate stages. They note that converting HTML to Markdown does not by itself remove menus, advertisements, and other unwanted page content, so they subsequently use a dedicated extractor. Their Markdown extractor is also published on Hugging Face.
That separation seems useful here even if your implementation is completely different from SWEb.
I selected 10 high-information examples and recovered the corresponding original records from the Common Crawl index. All 10:
payload_digest.So those comparisons were against the archived source payload rather than whatever happens to be live at the URL today.
Among those selected case studies:
These are not corpus-wide failure rates. The cases were selected because they looked informative, so they are intentionally not a representative random sample.
But they do demonstrate that both failure modes exist:
The Common Crawl CDXJ index makes this kind of check fairly reproducible because it exposes the WARC filename, byte offset, length, and digest for a capture.
If retaining links is one of the explicit goals, I would give link preservation its own sanity check.
In the first 5k deterministic sample, only about 0.24% of rows contained ordinary Markdown-link syntax of the form:
[text](url)
That number alone does not prove a bug — you might intentionally be preserving anchor text while removing targets, or your serialization may represent links another way.
But combined with the 10 digest-matched WARC examples, where all raw pages contained <a> elements and all ten final outputs had zero Markdown links, I think it is worth confirming the intended contract:
Do you want:
A. visible anchor text only
or
B. actual link targets preserved in Markdown?
If the intended output is A, then this is probably expected.
If the intended output is B, this looks like a useful dedicated regression test.
I do not want the spot-check above to give the impression that the whole conversion is flattening everything.
In a separate deterministic 5k sample, simple syntax checks found:
These are only syntax-presence measurements — they do not tell us whether the semantics are correct — but they are a useful positive control.
So my current interpretation would be:
the structured representation is doing useful work, but some page/extraction paths appear to flatten structure or retain boilerplate.
That is a much more encouraging problem than “the Markdown conversion does not work”.
It also lines up with the broader extractor literature. For example, Beyond a Single Extractor finds that changing the HTML extractor can substantially change which Web pages survive a fixed downstream filtering pipeline, and that the extractor choice matters more on structured content such as tables and code than on ordinary language-understanding benchmarks.
The important caveat from that paper is also useful here: structure preservation does not imply that one particular Markdown serialization is universally optimal. The thing to optimize is useful retained information, not Markdown punctuation for its own sake.
The dataset is already useful to inspect because it preserves URLs, WARC metadata, scores, etc.
The next thing I would find most useful is a compact “recipe” section in the Dataset Card.
Something like:
Extraction
- HTML -> Markdown tool:
- exact version / commit:
- important options:
- main-content extractor:
- main-content options:
Language
- LID model:
- threshold:
- applied before/after extraction:
- document/chunk/line level:
- short/mixed-language policy:
Deduplication
- exact-dedup key:
- near-dedup algorithm:
- n-gram / similarity parameters:
- dedup scope:
per shard / per crawl / final corpus
- which member of a duplicate cluster is retained:
Quality signals
- perplexity model/tokenizer:
- document/chunk aggregation:
- filtering threshold, or metadata only:
- FineWeb-Edu model/revision:
- threshold, or metadata only:
Stage counts
raw records
-> parse success
-> extraction success
-> LID pass
-> quality pass
-> exact-dedup survivors
-> near-dedup survivors
-> final published rows
This is less about documentation for its own sake and more about making each observed failure assignable to a stage.
For comparison, FineWeb’s Dataset Card documents its major processing stages and links to a working implementation, and the public DataTrove FineWeb pipeline makes the extractor, language filtering, quality filters, and MinHash configuration inspectable.
One particularly useful pattern there is saving removed documents via exclusion_writer.
I like that pattern a lot for iteration: even keeping a small sample of rejected/borderline pages from each stage makes it much easier to detect the opposite failure mode — good documents being removed.
For example:
audit_samples/
language_rejected/
quality_rejected/
dedup_removed/
borderline/
This does not have to be part of the public release or contain everything. Even a small deterministic sample can make threshold changes much easier to audit.
I would not start with a huge benchmark.
A small manually reviewed set is probably more informative at this stage, but I would avoid using only random article/blog pages.
Something like:
articles/blogs
forums
documentation/code
product/e-commerce pages
listings/directories
and then, for each page, score only:
main content retained?
main content hallucinated/added? # should always be no
heading hierarchy useful?
lists retained?
tables retained?
code retained?
links retained as intended?
boilerplate remaining?
The reason for page-type stratification is that ordinary article extraction is comparatively easy. Recent work such as WCXB reports much wider extractor differences on structured/non-article page types than on ordinary articles.
So even 10–20 pages per type can reveal more than a larger sample dominated by blog posts.
If the eventual claim becomes something stronger than “here is a useful cleaned Common Crawl dataset” — for example:
preserving Web structure this way improves pretraining
— then I would use a matched-input ablation rather than comparing unrelated corpora.
For example:
same raw HTML pages
same language filtering
same quality filtering
same deduplication
same token budget
same training setup
|
+-- extractor / serialization A
|
+-- extractor / serialization B
That isolates extraction/serialization much more cleanly.
AICC / MinerU-HTML is relevant here because it evaluates structured HTML extraction and also performs controlled pretraining comparisons.
Beyond a Single Extractor is useful for a slightly different reason: it shows that even under a fixed downstream pipeline, extractor choice can change both data coverage and structured-content performance.
I would consider this a later step, though. The current mechanical checks seem much cheaper and higher-value first.
I also looked at whether the existing perplexity / edu_score signals seemed to be systematically suppressing Markdown structure.
I did not find strong evidence for that in the small audit.
In particular, higher perplexity was more clearly associated with the secondary non-English candidates than with headings/lists/tables/code. The structural features did not show the pattern I initially expected if perplexity were simply rejecting Markdown-heavy documents.
Similarly, edu_score did not show a large obvious negative relationship with Markdown structure in this sample.
So I would not recommend changing those scoring stages based on this audit alone.
There is still a conceptual caveat: FineWeb-Edu-style scores are educational-quality signals, not universal “cleanliness” scores. But that is a general interpretation issue, not a concrete bug I found here.
A few references that seem especially close or useful:
SWEb: A Large Web Dataset for the Scandinavian Languages
Probably the closest conceptual reference I found.
The pipeline goes back to WARC HTML rather than accepting flattened WET text, converts to Markdown, and then performs a separate main-content extraction stage. Data, models, and code are published.
Their Markdown main-content extractor is also useful as a concrete example of treating “Markdown serialization” and “which Markdown lines are actually main content?” as separate problems.
AICC: Parse HTML Finer, Make Models Better
Focuses strongly on preserving structured elements such as tables, code, and formulas and evaluates the impact downstream.
Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Useful because it shows that extractor choice can change:
It is also a good warning against treating one representation as universally best.
Useful context for the language-filtering result. It evaluates LID systems on noisy real Web data rather than only clean translated/news-style benchmarks.
FineWeb processing script in DataTrove
Useful primarily as examples of making the full curation recipe inspectable and retaining rejected samples per filtering stage.
For the current stage of the project, I do not think you need a large GPU experiment just to make the dataset more defensible.
The highest-information / lowest-cost route looks more like:
1. final global record_id uniqueness
2. LID recipe + small labeled sanity sample
3. separate:
main-content recall
structure fidelity
boilerplate leakage
4. keep small rejected/borderline samples
5. document the recipe
Then, if the goal later becomes demonstrating that this representation improves model training, move to matched extractor/serialization ablations.
So overall: I think the structure-preserving direction is interesting, and the quick audit did show that it is already retaining a lot of Markdown structure. The main things I would tighten first are final cross-shard exact-record deduplication, the English LID stage, and the boundary between main-content extraction and structure preservation.
Those all look fixable/testable without changing the core idea.
Yes, I understand. At first, I went through many iterations of the code, frequently stopping, modifying, and rewriting parts of the pipeline. One of the main problems was that I didn’t have a reliable way to resume processing from an exact position within a WARC file.
Because of that, some WARC records were processed again when a job was restarted, which is what caused the duplicate rows you found. The ~5.5k duplicate records make sense in that context.
I agree that this should be handled independently at the final assembly stage as well. I’ll add a global record_id uniqueness check so that even if something goes wrong during processing or resuming, duplicate source records cannot make it into the final published dataset.
The English/non-English filtering was another weakness in my earlier pipeline. I didn’t have a reliable language-identification stage; instead, I was relying on filters that were intended primarily for English content. As a result, some non-English websites could still pass through the pipeline. Your LID results make it clear that this needs to be handled much more explicitly, by using the language metadata from the WARC files for language identification
Your other points about separating main-content extraction from structure preservation are also very useful. I’ll go through those next and document the pipeline stages and settings more clearly.
Thanks for taking the time to actually run these checks — this is exactly the kind of audit that helps me identify the weak points in the pipeline and make the dataset more reliable.