Now showing

KINO

A Keyframe Interface for VLM Planning and Whole-Body Control in Humanoid Loco-Manipulation

ETH Zürich, Switzerland

Abstract

Humanoid loco-manipulation requires robots to interpret task instructions and scene semantics while executing coordinated whole-body motions. We propose a hierarchical framework that uses motion keyframes as an intermediate representation between Vision-Language Model (VLM) planning and Reinforcement Learning (RL) control.

Each keyframe specifies a target whole-body robot pose and, when applicable, an object pose. Given a language instruction, scene observations, and execution feedback, the VLM selects successive task-relevant keyframes from a predefined library. The selected keyframes are retargeted to the current scene to account for object poses and dimensions. A keyframe-conditioned whole-body policy then generates joint-level actions to reach these goals.

We introduce a saliency-based keyframe sampling strategy for low-level policy training that improves end-to-end task success rate from 44% to 92% when using sparse VLM keyframes. We evaluate our framework on object pickup, transport, and placement tasks in simulation and on a Unitree G1 humanoid. The system performs both one- and two-handed manipulation and generalises to placement locations beyond the training reference data.

The interface

A task is a short sequence of keyframes

For “move the box to the target”, the VLM picks six poses from the keyframe library, one at a time, and waits for the robot to reach each before choosing the next. Select a keyframe to see it.


    

Method

Pipeline: instruction, scene observation and robot state go to the VLM, which selects a keyframe from the library; the keyframe retargeter adapts it to the scene; the low-level policy executes it.
Fig. 2. The VLM chooses a keyframe from the library based on the scene and robot context. The keyframe is retargeted and passed to the low-level policy.
Semantic reasoning

VLM selects

Qwen3.6-27B reads the instruction, a third-person image and a planner context (previous keyframe, success flags, robot, object and target poses, goal errors) and returns one keyframe as structured JSON. Under 0.2 s per query on one RTX 4090.

Scene alignment

Retargeter adapts

End-effector and root targets are mapped from the library object’s bounding box to the observed one, then damped least-squares IK recovers the full pose with feet on the ground. About 5 ms.

Policy execution

RL policy reaches

A keyframe-conditioned PPO policy outputs 29 joint targets for the Unitree G1 from proprioception, object pose, the previous action and the goal, with a 10-step observation history.

Saliency-based keyframe sampling

Uniformly sampled training goals are mostly walking frames, so the policy rarely practises the pick and place poses the VLM will ask for. KINO samples keyframes where the motion actually changes stage.

Peaks of the low-pass filtered body acceleration give candidates; candidates dominated by periodic leg motion are removed; the rest are aligned with peaks in the object’s vertical acceleration.

$$\Gamma(t)=\sum_{j}\big\lVert \ddot{\mathbf{p}}_j(t)\big\rVert_2, \qquad \mathcal{P}_{\Gamma}=\operatorname*{arg\,max}_{\tau\in\mathcal{N}}\ \mathrm{LPF}[\Gamma](\tau)$$
Filtered acceleration signal with kept and removed keyframes, and the resulting keyframe poses before and after leg-motion filtering.
Fig. 3. Leg-motion filtering removes intermediate walking frames and keeps the salient ones.
A generic picking keyframe before and after retargeting to a box with a different pose and size.
Fig. 4. A general picking pose (left) retargeted to a different box pose and size (right).

Keyframe retargeting

Each point keeps its normalised position relative to the object’s bounding box (centre \(\mathbf{c}\), rotation \(\mathbf{R}\), half-extents \(\mathbf{h}\)):

$$T_{i\to *}(\mathbf{p})=\mathbf{c}^{*}+\mathbf{R}^{*}\Big[\big(\mathbf{R}_i^{\top}(\mathbf{p}-\mathbf{c}_i)\big)\oslash\mathbf{h}_i\odot\mathbf{h}^{*}\Big]$$

For long walks, a curvature-limited arc or Dubins path (curvature \(\le 3\,\mathrm{m}^{-1}\)) is split into intermediate locomotion keyframes, each at most 45° of heading change and 2 m apart.

Results

44% → 92%
End-to-end success with saliency-based sampling, 100 trials
< 0.2 s
Average VLM time per keyframe
277
Reference clips: 52 locomotion, 105 box, 120 bucket

Ablation: keyframe sampling

Success rate per stage over 100 trials. Start distance 0.5–2.3 m, target 0.2–2.0 m from the box; success within 0.3 m.

Uniform (1.5 s interval)Saliency-based (ours)
Success-rate heat map over placement positions from −4 to 4 m, nearly all at 100%, extending well beyond the outlined training coverage; and a planned path with intermediate keyframes.
Fig. 7. Box placement success across a grid of targets. The outlined region is covered by training data; success stays high far outside it. Right: planned intermediate locomotion keyframes.

Goal-reaching error

StageSamplingRoot (m) ↓Joint RMSE (rad) ↓Object (m) ↓
ApproachUniform0.125 ± 0.0500.099 ± 0.007–
Saliency0.113 ± 0.0470.075 ± 0.004–
PickUniform0.149 ± 0.0430.150 ± 0.0200.170 ± 0.094
Saliency0.119 ± 0.0340.116 ± 0.0070.043 ± 0.035
PlaceUniform0.162 ± 0.0480.139 ± 0.0090.082 ± 0.026
Saliency0.067 ± 0.0180.121 ± 0.0040.227 ± 0.043

The one exception is object error when placing: the reference place keyframe holds the box too close to the robot, and the policy matches the root target while setting the box down further away.

Simulation: one- and two-handed tasks

Bucket and box pick-and-place in simulation through approach, pickup and placement, with green keyframe markers.
Fig. 6. The VLM picks one-handed keyframes for the bucket and two-handed keyframes for the box. Green markers show the target keyframes.

Hardware: Unitree G1

Two-handed · box
One-handed · bucket
The full pipeline on hardware. The VLM is prompted to pick up and transport the object, and guides the policy through each keyframe. In the box clip, the overlays show each VLM request, the selected keyframe and the retargeted goal. Object pose comes from motion capture.

Failure recovery

The bucket is moved while the robot grasps it, and the first grasp misses. From the new observation and the goal errors in its context, the VLM notices the missed keyframe and sends the pick again, and the retry succeeds.

Path planning

Planned intermediate keyframes reach new locations. Four directions are shown at once, each with a target 3.5 m away. Without them, a single far goal outside the training data can make execution fail.

BibTeX

@article{chen2026kino,
  title   = {KINO: A Keyframe Interface for VLM Planning and Whole-Body Control
             in Humanoid Loco-Manipulation},
  author  = {Chen, Sitong and Zargarbashi, Fatemeh and Cheng, Jin and
             An, Tianxu and Coros, Stelian},
  journal = {arXiv preprint arXiv:2609.18869},
  year    = {2026}
}