Hmm… some precedents from the Minecraft ecosystem may be relevant here:
Yes — I think this kind of data can be useful for research. Paired gameplay video plus native keyboard/mouse actions is the basic ingredient used by projects such as Video PreTraining (VPT), and more recent multi-game efforts such as D2E and Learning to Play show that the idea is not limited to Minecraft.
For a 10–30 minute first pilot, though, I would probably not use “does this improve a Skyrim agent?” as the acceptance criterion. That amount of data is more defensibly useful as a data-contract and ingestion pilot: can another team load the final package, understand the action semantics, inspect the synchronization, and decide whether a larger collection would be worth doing?
A practical default route might be:
1. A short calibration recording
2. One bounded task from a fixed save, repeated 2–3 times
3. The exact final deliverable package
4. A small loader/viewer and validation report
5. Acceptance at three levels: artifact -> task protocol -> consumer pipeline
The repetition count here would only be a smoke test for the collection protocol, not a statistically meaningful model evaluation.
Two small pilot cards
I think it may help to offer one or two pre-defined cards instead of asking the first interested researcher to design the entire protocol from scratch.
Card A: compatibility/calibration
Roughly 1–2 minutes:
- remain idle for a visible interval
- slowly sweep the camera left, then right
- short press/release of W, A, S, and D
- one longer movement-key hold
- open and close inventory or pause several times
- perform one visually obvious interaction
- finish with several seconds of no input
This creates known examples of short taps, held keys, relative mouse motion, UI transitions, interaction, and no-input periods. It gives both a human reviewer and a validator something concrete to check.
Card B: fixed-save bounded task
For example, a loot-and-equip task, perhaps 3–4 minutes repeated two or three times:
Start from the same named save.
1. Approach a known item.
2. Pick it up.
3. Open inventory.
4. Equip or use the item.
5. Close inventory.
6. Hold a clearly visible result for several seconds.
Restrictions for this card:
- no console
- no fast travel
- no unrelated menus
- no unrelated combat unless combat is part of the requested task
A short dialogue task could work equally well. I would probably avoid making combat the first validation task, because mouse sensitivity, enemy behavior, player skill, animation timing, damage state, and sampling frequency all become entangled at once. A visible interaction/UI task is easier to diagnose.
The important part is not that loot-and-equip is necessarily the most valuable Skyrim research task. It is simply a reasonably clean test of movement, camera control, interaction, GUI operation, repeatability, and visible completion.
What would “accepted” mean?
I would separate acceptance into three layers:
| Layer |
A reasonable first-pass result |
| Artifact / contract |
The agreed reader or documented adapter can open the final delivered files; timestamps and references are coherent; field meanings and invalid intervals are documented. |
| Task protocol |
The correct save and task-card version were used; success, legitimate task failure, timeout, operator deviation, and capture failure are distinguishable. |
| Consumer pipeline |
The receiving team can produce its next intended artifact — for example a synchronized viewer, converted dataset, model input sequence, or evaluation record — without hidden manual repair. |
That makes a pilot informative even if the downstream model performs badly. Poor model output may mean the model, task, amount of data, or domain transfer is difficult; it does not automatically mean the capture package failed.
Why the Minecraft precedents seem relevant — and where the analogy stops
The useful lesson from Minecraft is not that Skyrim should expose the same internal state. It is the collection-contract structure.
The VPT repository includes concrete contractor prompts rather than only “play Minecraft.” Examples include finding a cave within a time limit without digging straight down, making a waterfall and repositioning for a scenic final view, and building a village-style house while avoiding damage to the village. The prompts define:
- a starting condition;
- a bounded objective;
- allowed and prohibited shortcuts;
- a time limit;
- an explicit exit rule;
- and often a visible final presentation.
The VPT collection process also changed its prompts and recorder versions when different behavior distributions were needed — early-game play, block placement, house building, tree chopping, specific crafting goals, and so on. Recorder changes included details such as GUI mouse scaling, wheel input, hotbar handling, step timestamps, null inputs, and timer behavior. This suggests that the task prompt and collector version are part of the data provenance, not incidental notes.
The MineRL BASALT benchmark goes one step further: it separates demonstration data, a small end-to-end pipeline test, rollout generation, and human evaluation. Its small test pipeline is valuable because it verifies that partial data can travel through preprocessing, training, rollout, and review — not because the tiny test model proves the task has been solved.
A Skyrim pilot could borrow this structure:
task_card_version
collector_version
schema_version
game_build
save_identifier
relevant mods/settings
operator instruction text
success / failure / timeout / capture-failure rules
What should not be imported from Minecraft is its internal-state richness. Minecraft/MineRL setups can expose or control things such as ticks, coordinates, yaw/pitch, inventory, statistics, structured actions, checkpoints, and environment-side task state. An external Windows capture of Skyrim may not have authoritative access to any of those.
So the translation would be:
- a fixed save approximates a controlled start, but is not necessarily an exact simulator reset;
- a visible end state approximates task evidence, but is not authoritative semantic ground truth;
- external frame/input timing can be made auditable;
- unknown internal game state should remain explicitly unknown.
Recent multi-game work provides some corroboration beyond Minecraft. Learning to Play records human gameplay with keyboard/mouse actions at 20 Hz across multiple 3-D games, initially reviews new annotator/game combinations manually, and then combines random review with simple automated anomaly checks. That does not establish a universal frequency or protocol, but it supports the broader idea that collection quality depends on onboarding, explicit coverage, automated checks, and review — not only on whether the recorder launches successfully.
A possible minimal package, timing contract, and low-cost QA
The current 100 ms representation looks convenient for inspection and for consumers that want a simple fixed-rate view. I would keep it if it is useful. The main thing is to define its semantics precisely.
For example:
- Does
keysDown mean the state at the start of the interval, the end, or the union over the interval?
- If a key is pressed and released inside one 100 ms bin, is that event retained?
- Is mouse
dx/dy the sum of all Raw Input deltas in the interval, the latest event, an average, or something else?
- Are mouse buttons represented as states, transitions, or both?
- Is
nearestFrame selected from actual captured frame timestamps/PTS, or derived using an explicit constant-frame-rate rule?
- What clock origin and units are used for input events, frame timing, and telemetry?
- What exactly do the FPS and frame-time fields measure, and over what update window?
If event-level input timestamps are already retained internally, exposing a lossless event export in addition to the 100 ms view would preserve more downstream choices. Consumers could then deterministically create 50, 100, or other millisecond views without recollecting the session. If raw events are not retained, the 100 ms representation may still be useful for slower interaction tasks; it would simply have a clearer boundary.
An illustrative package could be:
source_video.mkv or source_video.mp4
raw_input_events.jsonl or .mcap # only if event-level data is retained
samples_100ms.jsonl # simple fixed-rate view
session_metadata.json
task_card.json
validation_report.json
preview_with_input_overlay.mp4
README / data dictionary
minimal_loader.py or documented adapter
The filenames and format are not important; the separation of responsibilities is.
A first validator could remain very small and deterministic:
- timestamps are monotonic where required
- video duration and event range agree under the documented timing rule
- referenced media files and paths exist
- the calibration sequence contains the expected action types/counts
- active-play tasks contain some keyboard/mouse activity
- no key/button appears implausibly stuck for the entire recording
- focus or active-window changes are reported
- capture/encoder failure intervals are marked
- there is no unexplained recording tail after the task ended
- the exact final package opens through the agreed reader/adapter
I would run this twice:
- immediately after capture;
- after sanitization, transcoding, renaming, path rewriting, and packaging.
That second pass matters because publication processing can damage an otherwise valid recording. The D2E repository and dataset provide a useful adjacent example: D2E packages synchronized video/audio/input events, readers, inference, and evaluation tooling, and its release history also documents fixes made after dataset reprocessing. The general lesson is that validating the capture is not the same as validating the files that the researcher eventually receives.
For actual Windows timing, Windows Graphics Capture exposes a system-relative frame time, while Raw Input provides event-level keyboard/mouse information. I would not assume a particular internal implementation from the public sample, but those APIs provide one possible route to documenting how the clocks and frame references are derived.
The synchronized overlay is very useful for review, especially around short taps, menu transitions, and task endpoints. I would treat it as a review surface rather than independent synchronization ground truth if it is generated from the same trajectory file.
How I would branch the pilot by downstream use
Is the first goal only compatibility / ingestion?
|
+-- Yes:
| calibration card
| + known transitions
| + exact final package
| + reader/adapter
| + validator and preview
|
+-- No, the target use is known:
|
+-- IDM / action labeling
| event-level action evidence if available
| explicit frame/action timing contract
| temporal windows or offsets
| repeated visually recognizable transitions
|
+-- behavior cloning / task-conditioned control
| versioned task text
| fixed start state
| episode boundaries
| success / failure / timeout labels
| repeated executions of the same task
|
+-- action-conditioned video or world model
| source-video timing and transcode provenance
| action windows
| loading/menu/death segmentation
| scene and behavior diversity
|
+-- GUI / inventory / dialogue agent
active window and focus
cursor/window coordinate conventions
window geometry and display scaling
explicit UI phase boundaries
Different consumers may legitimately choose event streams, 10 Hz, 20 Hz, 30 Hz, or 60 Hz representations, and they may use different temporal offsets or context windows. I do not think the pilot needs to declare one universally correct rate. It needs to make the timing and aggregation contract explicit enough that a consumer can decide whether it is appropriate.
Interoperability could also be incremental. A full migration to another ecosystem is not necessary. A documented adapter may be enough. D2E/OWA provides one example of event/state/video-reference tooling, while MineStudio illustrates how conversion, segmentation, visualization, and dataset access can be treated as part of the data lifecycle rather than as afterthoughts.
For text labels, I would prefer operator intent recorded at episode start, or broad task-phase intervals, over frame-exact retrospective instructions unless the alignment procedure really supports that precision. The multi-game study above notes that low-frame-rate video processing by VLM services can make highly time-specific retrospective descriptions drift away from the actual action moment.
So, in compact form, I think the offer becomes easier for a research team to accept or reject if it is presented as two concrete options:
Card 1: compatibility/calibration pilot
Card 2: fixed-save bounded-task pilot
For each card, specify the exact starting condition, repetition count, operator instruction, prohibited shortcuts, delivered files, and the three acceptance results. That would preserve the flexibility of your specification-driven approach while reducing the amount of protocol design that the first interested team has to do.
The precedents show that this type of paired video/action data has real research uses. They do not by themselves prove demand for this exact Skyrim sample, and a 10–30 minute pilot would not establish useful training scale. But it could produce something quite valuable: a clear, inspectable answer to whether your capture contract can plug into a real research workflow and whether a larger collection should be attempted.