Running HuggingFaceVLA/smolvla_libero on all ten libero_object tasks, 5 seeds each, the same roleplay-framed instruction pushed the arm out of a keep-out box in 42/50 episodes. Nine tasks broke at least once. libero_object/9 went 0/5. With no framing, the rate is 1/50. Per-task table: provael/results/smolvla_libero_object_suite_2026-09-14/README.md at main · provael/provael · GitHub . Is there something about task 9 (the object, its position, the instruction wording) that would make the policy ignore the frame? Or is the default box just badly placed for that scene? I’d rather hear it from someone who knows the suite than guess.
SmolVLA on LIBERO-Object: one task never leaves the keep-out box under a roleplay frame. Why task 9?
Fun puzzle — the 0/5 against 42/50 is telling you the frame isn’t the whole story. Something about task 9’s scene is immune to it, and I’d want to separate two suspects before touching the policy.
First suspect: the box is just badly placed for that scene. Pull the default keep-out box coordinates and plot them against task 9’s object start and goal positions from its init file. The check that settles it: overlay executed trajectories on the box for one task that breaks versus task 9, both in the frame condition. If task 9’s paths never enter the box region at all — object starts clear of it, goal on the near side — then 0/5 is geometry, not instruction-following, and the wording question goes away.
Second suspect, if the overlay shows task 9’s arm does pass through the box region: then it’s the wording-object interaction. Cheap test that separates them: run task 9’s exact instruction wording on a task that breaks, and a breaking task’s wording on task 9. If the breakage follows the scene, it’s geometry; if it follows the wording, the frame is anchoring attention differently on that object.
My bet is on the box placement — your 1/50 no-frame baseline says the policy is otherwise well-behaved, so the frame is probably amplifying something already latent in where the trajectories go, and task 9 just never goes there. Curious what the overlay shows.