For now, I tried narrowing this down a little further:
Following the suggestion above to correlate the slow fetches with worker/cache/storage behavior, I ran two small DataLoader-only controls against the same historical regression.
The short version is:
the recurring stall cadence looks genuinely coupled to the DataLoader worker pool, but not to ordinary worker restarts and not primarily to FIFO return ordering.
The expensive work still appears to be on the historical dataset-access side. The worker configuration seems to determine how that cost becomes visible over time.
The first control was simply to keep the historical broken stack fixed and vary num_workers.
| Condition |
What happened |
broken, num_workers=0 |
essentially every fetch was slow |
broken, num_workers=1 |
large stalls essentially every step |
broken, num_workers=2 |
large stalls recurred about every 2 steps |
broken, num_workers=4 |
large stalls recurred every 4 steps |
fixed, num_workers=4 |
no >5 s stalls |
I ran the broken sweep in both directions (4→2→1→0 and 0→1→2→4) to make order/cache effects a little less likely to fool me.
For the runs with enough >5 s stalls:
workers=1 → median stall gap ≈ 1workers=2 → median stall gap = 2workers=4 → median stall gap = 4- in the 2-worker and 4-worker runs, the observed large-stall gaps were exact multiples of the worker count
So the four-step pattern from the earlier T4 reproduction no longer looks like an arbitrary periodicity.
The second control was useful because it falsified one of my own first guesses.
I had suspected that ordered DataLoader return might be producing a simple head-of-line effect: one slow required batch blocks the consumer while later batches become ready behind it, producing one long wait followed by several nearly instantaneous fetches.
PyTorch does normally return multiprocessing DataLoader batches in order, and the pinned PyTorch 2.7.1 used here already has both prefetch_factor and in_order=True by default.
So I held everything else fixed at:
num_workers = 4
prefetch_factor = 2
same historical broken LeRobot revision
same datasets==4.1.1 environment
same deterministic sampler stream
and compared:
in_order=True
vs.
in_order=False
I also recorded the original sampler-batch rank of every returned batch, so I could verify that in_order=False was actually allowing later-ready batches to overtake earlier ones.
It did.
Across the two unordered runs I observed 14 overtaking deliveries, while the ordered runs had none.
But the large-stall cadence remained:
ordered:
stall steps = 0, 4, 8, 12
unordered:
stall steps = 0, 4, 8, 12
Both conditions still had:
4 stalls > 5 s out of 16 deliveries
median large-stall gap = 4
That seems like a useful negative result.
Removing FIFO enforcement changed the delivery order, but it did not remove the four-worker cadence.
So I would now weaken my earlier head-of-line interpretation substantially.
There was still some effect on the shape of the tail:
| Broken condition |
mean fetch |
p95 |
>5 s stalls |
stall gap |
| ordered, pass 1 |
~4.71 s |
~19.28 s |
4 |
4 |
| unordered, pass 1 |
~4.31 s |
~16.67 s |
4 |
4 |
| unordered, pass 2 |
~4.38 s |
~16.43 s |
4 |
4 |
| ordered, pass 2 |
~4.62 s |
~17.75 s |
4 |
4 |
So ordering can redistribute some of the waiting and change which batch exposes it, but it does not appear sufficient to explain the recurring large stalls.
The interpretation I currently find most consistent with the observations is therefore closer to:
expensive historical dataset access
↓
substantial CPU-side producer work
↓
multi-worker DataLoader / prefetch pool
↓
worker-count-coupled availability pattern
↓
consumer sees ~1 / ~2 / ~4 step stall cadence
depending on worker count
rather than:
FIFO ordering
↓
four-step stalls
In other words, I think it is useful to separate two questions:
- Why is producing these batches expensive?
- Why does that expense appear to the training loop in this particular temporal pattern?
For this historical case, the first question was already answered upstream: LeRobot PR #2408 traced the training slowdown to _query_hf_dataset and changed the dataset access path. I am not trying to rediscover or reassign that root cause.
The new part here is mostly about the second question.
A few other observations helped narrow it further:
- I saw no worker PID-set changes during any of the measured broken runs, so ordinary worker restart/reset does not look like a good explanation for recurring within-run stalls.
- Slow intervals contained much more aggregate worker CPU time than fast intervals.
- The median worker physical
read_bytes during the slow intervals was 0, and the median major-page-fault count was also 0.
- The known-fixed revision with four workers again had no >5 s stalls.
So, from the counters I collected, the evidence points more strongly toward CPU-side dataset processing than toward recurring physical storage reads.
I would not stretch that farther than the measurements allow: read_bytes == 0 does not prove that every cache/memory effect is irrelevant, I did not measure CPU hardware-cache misses, and I did not independently isolate the OS scheduler.
One other nice sanity check was that removing ACT compute did not destroy the original heavy-tail shape.
The previous full ACT T4 reproduction looked approximately like:
broken pair 1:
mean input wait ≈ 4.95 s
median ≈ a few ms
p95 ≈ 19.75 s
broken pair 2:
mean input wait ≈ 4.48 s
median ≈ a few ms
p95 ≈ 18.90 s
The short DataLoader-only four-worker control produced roughly:
pass 1:
mean ≈ 4.74 s
median ≈ 1.8 ms
p95 ≈ 19.58 s
pass 2:
mean ≈ 4.41 s
median ≈ 4.0 ms
p95 ≈ 17.23 s
I would not call those numbers a strict numerical reproduction—the short probe deliberately uses artificial consumer pacing rather than real ACT compute—but qualitatively the same unusual shape survives:
most delivered batches are immediately available, while a small regularly spaced subset dominates the wall time.
For reference, the earlier full T4 reproduction is still here:
Executed Colab T4 reproduction notebook
There is also an important host-specific caveat: this Colab T4 instance exposed only 2 logical CPUs.
So I would not infer from this that, for example, num_workers=2 is generally preferable to 4. Four CPU-heavy workers contending on a two-CPU host can obviously affect absolute stall duration.
What seems much more reusable to me is the structural change under the controlled variation:
worker count changes
↓
stall cadence changes with it
rather than the absolute number of seconds.
This actually reinforces my impression of where a tool like TraceML is useful.
The compact diagnosis answered the first important question correctly:
Where did the extra training time go?
INPUT-BOUND
Then the shape of the input timing suggested which cheap experiment to try next:
INPUT-BOUND
↓
uniformly slow, or heavy-tailed?
↓
heavy-tailed
↓
random, or temporally structured?
↓
structured
↓
vary one cheap parameter
↓
cadence follows num_workers
I do not think TraceML needs to become a full DataLoader/PyTorch profiler to make this useful.
If you are deciding what belongs in a compact comparison, I would personally find a very small amount of distribution/temporal evidence valuable—enough to distinguish something like:
uniformly slow input
from:
mostly-fast input with recurring large stalls
because those two cases suggest rather different next experiments even if their mean input wait is identical.
The current TraceML README already points an INPUT-BOUND diagnosis toward workers, transforms, tokenization, collation, or storage. This case makes me think the useful extra layer may be less about automatically naming the final root cause, and more about preserving enough timing shape to select a cheap discriminator.
Exact control setup and the full worker-count result
The historical pins remained the same as in my earlier reproduction:
- TraceML reference revision:
107432d17c45f9c28e9704db5e1d3eba3cb616c3
- LeRobot broken revision:
f6b16f6d97155e3ce34ab2a1ec145e9413588197
- LeRobot fixed revision:
67a6c8dfef90351830ad02afc5bf1bd299f9a521
- dataset:
imstevenpmwork/aloha_sim_transfer_cube_human_image
- dataset revision:
13e0d3bff90f02ec417761b58199eb1d33a72efb
datasets==4.1.1- batch size: 8
- action horizon: 100
- seed: 1000
prefetch_factor=2 when multiprocessing was enabled
The broken worker-count sweep was run twice:
pass 1: 4 → 2 → 1 → 0
pass 2: 0 → 1 → 2 → 4
The probe recorded blocking next(dataloader) time, delivered worker ID/PID, worker PID-set changes, worker CPU time, physical read counters, page faults and system I/O counters.
Summary:
| revision |
pass |
workers |
median |
p95 |
>5 s |
median >5 s gap |
| broken |
1 |
0 |
6248 ms |
6845 ms |
32 |
1 |
| broken |
1 |
1 |
6027 ms |
6923 ms |
30 |
1 |
| broken |
1 |
2 |
4416 ms |
8292 ms |
15 |
2 |
| broken |
1 |
4 |
1.8 ms |
19581 ms |
8 |
4 |
| broken |
2 |
0 |
6219 ms |
6722 ms |
32 |
1 |
| broken |
2 |
1 |
5884 ms |
6418 ms |
32 |
1 |
| broken |
2 |
2 |
3785 ms |
8924 ms |
16 |
2 |
| broken |
2 |
4 |
4.0 ms |
17228 ms |
8 |
4 |
| fixed |
1 |
4 |
0.39 ms |
1.55 ms |
0 |
— |
A schematic view of the result is approximately:
workers=1:
S S S S S S S S ...
workers=2:
S . S . S . S . ...
workers=4:
S . . . S . . . S . . . ...
S = >5 s fetch
That simple scaling relationship is why I now think the worker-pool coupling is more interesting than the original observation that the four-worker run happened to stall every fourth delivery.
The `in_order` control and why I changed my interpretation
For the second control I fixed:
num_workers=4
prefetch_factor=2
and compared in_order=True against in_order=False.
The sampler permutation was fixed and every returned batch carried its original sampler-batch rank.
That matters because it lets us distinguish:
I requested unordered behavior
from:
batches actually overtook one another.
They did overtake one another.
The result was:
| condition |
mean |
median |
p95 |
>5 s |
stall gap |
overtakes |
| broken pass 1, ordered |
4708 ms |
0.37 ms |
19282 ms |
4 |
4 |
0 |
| broken pass 1, unordered |
4312 ms |
423 ms |
16667 ms |
4 |
4 |
9 |
| broken pass 2, unordered |
4377 ms |
656 ms |
16432 ms |
4 |
4 |
5 |
| broken pass 2, ordered |
4616 ms |
1.56 ms |
17750 ms |
4 |
4 |
0 |
| fixed, ordered |
53 ms |
0.37 ms |
211 ms |
0 |
— |
0 |
The important invariant was:
ordered:
0, 4, 8, 12
unordered:
0, 4, 8, 12
So my earlier thought that FIFO/head-of-line blocking might be the primary explanation for the four-step cadence did not survive the control.
PyTorch 2.7.1 defines in_order=False as disabling FIFO enforcement for multiprocessing DataLoaders. The same source also documents prefetch_factor=2 as two prefetched batches per worker by default when num_workers > 0.
Because actual overtaking happened here, but the four-step cadence survived, I think the safer conclusion is:
FIFO ordering can alter the presentation of the wait, but the recurring cadence is more fundamental to the worker/prefetch producer side than to ordered return itself.
I also would not interpret the earlier observation that the slow returned batches appeared to belong to “worker 0” as evidence that worker 0 itself was defective.
Once FIFO was disabled, the worker identities associated with the slow deliveries changed, while the four-step cadence remained.
So the more useful invariant was worker-width cadence, not a particular worker identity.
in_order=False is only a diagnostic control here. PyTorch explicitly warns that disabling ordered delivery can reduce reproducibility and, with imbalanced data, can skew the data order seen by training. I would not turn this experiment into a recommendation to train that way.
Historical LeRobot context and why I would keep this version-pinned
I think it is important not to turn this historical reproduction into a general claim that one Hugging Face Dataset access direction is always better.
The original LeRobot PR #2408 was merged in November 2025 and fixed the specific historical regression used for this TraceML case study.
But the story around _query_hf_dataset continued evolving.
For example:
- #2549 later reported a >20× slowdown from the column-first path on another dataset/setup.
- #2895 reported that multi-index queries used with
delta_timestamps could be about 40× slower with direct indexing than with select().
- #3523 showed that the relative performance changed again across
datasets versions; with datasets>=4.4.0, a hoisted row-first access pattern was around 3.5–3.7× faster than the measured column-first path in that microbenchmark.
So I think the reusable lesson is not:
column-first is faster
or:
row-first is faster
but something more conditional:
access pattern
× transform semantics
× datasets version
× requested columns/indices
can materially change the input cost.
There is also a newer, distinct example in #4392: a numeric delta query on an image-dtype dataset can end up formatting/decoding image columns that the query does not need, saturating DataLoader workers and making training CPU-bound.
That is not the same mechanism as this historical datasets==4.1.1 reproduction, but it is an interesting future validation case for the same higher-level diagnosis:
same symptom:
INPUT-BOUND
different implementation cause
At the time I checked, PR #4314 is still open. It proposes answering delta queries through cached single-column select_columns views, specifically to avoid feeding unrelated image columns through the transform. The reported benchmark there is very large, but since it is still an open PR I would treat it as a proposed current direction rather than a merged/current baseline.
That version dependence is why I would keep the claims in this thread explicitly attached to the pinned historical LeRobot / datasets==4.1.1 case.
What I would test next — only if someone wants the lower-level mechanism
For the practical TraceML question, I think I would stop here.
We already got useful discrimination from two small controls:
worker count
→ cadence follows worker width
FIFO disabled
→ batches really overtake
→ cadence remains
If someone specifically wanted to continue into the internals of the worker/prefetch behavior, the next cheap variable I would try is probably:
prefetch_factor=1
vs.
prefetch_factor=2
while keeping num_workers=4.
That could help separate effects of worker width from queue/prefetch depth.
Beyond that, I would probably stop using timing-only inference and move to a more explicit profiler/queue-level observation if the exact internal mechanism really mattered.
But I do not think that extra depth is necessary for the original performance-diagnosis question.
So my updated reading of the whole case is roughly:
TraceML:
correctly points to INPUT-BOUND
↓
per-step data:
shows a strongly heavy-tailed / periodic input pattern
↓
cheap worker-count control:
shows the cadence follows worker count
↓
PID + I/O counters:
weaken worker-restart and recurring-physical-I/O explanations
↓
in_order control:
produces real batch overtaking,
but the four-step cadence survives
↓
current interpretation:
historical CPU-heavy dataset access is the root cost;
worker-pool behavior shapes the temporal cadence;
FIFO ordering changes the presentation somewhat,
but is not the main explanation
For me that is a useful outcome for the original TraceML question.
The compact diagnosis did not need to know the final low-level mechanism in advance. It only needed to point to the correct subsystem, while preserving enough evidence that the next cheap experiment was obvious.
That seems like a good boundary for a lightweight diagnostic tool.
