Where Robot Policy Orchestration Pays Off

There has been significant progress developing robot policies that are performant on specific tasks. But policy generalization to out-of-distribution tasks and environments has remained difficult for state-of-the-art methods to achieve. Various methods and policies have emerged to tackle the robot learning problem, each with their own strengths and weaknesses. π_0.5 is a widely used end-to-end VLA model that is able to perform complex manipulation motions and react quickly to changes in the scene, but it sometimes fails to ground its movement on the correct target object and can struggle on long-horizon task execution. Other systems like TiPToP take a more modular approach, where perception foundation models and VLMs ground a scene into representations that task and motion planning algorithms use to plan robot actions. TiPToP has strong generalization across diverse scenes and language commands, but it is not guaranteed to find a constraint-satisfying plan. Its current instantiation also executes plans open-loop, so it is not as reactive to scene changes. This raises the question, can orchestration of multiple policies be valuable to leverage the unique strengths of different methods? And where is policy orchestration most useful?

Several recent works have tried different strategies for composing robot skills, from writing code over low-level primitives (CaP-X) to compiling skills into simulation-refined graphs (GaP) to distilling self-directed play into reusable skill libraries (RATs). How best to compose a robot’s capabilities, and where composition pays off, are still open questions.

To explore this problem setting, I ran a series of experiments in RoboLab, a benchmark built on NVIDIA Isaac Lab offering a diverse set of tasks in photorealistic scenes. The tasks span a breadth of competencies: visual grounding of color, size, and object semantics; relational reasoning over spatial, temporal, and numerical constraints; and procedural reasoning about actions and multi-step sequencing. The benchmark comprises 120 tasks, and neither TiPToP nor π_0.5 came close to solving all of them. Interestingly, when comparing individual policy performance, there were 37 tasks where one policy succeeded and the other didn’t (Figure 1). This suggests an opportunity to orchestrate policies together because their success and failure modes are partly disjoint.

Slicing coverage by task type sharpens the picture of complementary strengths (Figure 2): π_0.5 has higher success on stacking and reorientation tasks, while TiPToP performs well in semantic grouping tasks like putting away dishes or snacks. Affordance and spatial-relation tasks remain largely unsolved by either policy.

Orchestrating policies together can yield success on tasks where both policies individually fail

Figure 3. The state-machine orchestrator: TiPToP plans first, π_0.5 recovers if the task is still unsolved, and the cycle repeats until the task is solved or the episode budget runs out.

Even the simplest form of orchestration can boost task performance. TiPToP and π_0.5 were orchestrated together via a simple state machine: plan with TiPToP first, recover with π_0.5 if the task remains unsolved, and repeat (Figure 3). This combination achieved 39.2% on the RoboLab benchmark, beating π_0.5 alone (27.5%) and TiPToP alone (18.3%) (Figure 4). Qualitative video rollout analysis showed that π_0.5 and TiPToP were able to build off each other’s task progress. Partial progress from one policy, such as a subset of objects already moved or the gripper left near the target after a failed attempt, put the scene in a state that the other policy could finish the task from.

Importantly, orchestrating policies does not equate to only selecting the right policy for the task. Running the two policies together under the orchestrator succeeded on 17 tasks that neither policy solved on its own. One such example was on the task “Put all the green fruit on the plate”. TiPToP successfully computes a plan to move the limes to the plate, but its trajectory causes a robot arm collision with the wooden spoonholder, and its grasp completely misses. π_0.5 is able to move one of the limes successfully to the plate, but then incorrectly focuses on the pumpkin as its next target object. With both policies combined in an orchestrator loop, TiPToP first attempts the task. After it fails, the gripper is left close to the target object, and the VLA takes over to successfully finish the pick-and-place action. With the gripper being initialized close to the target object at the start of the VLA segment, the VLA did not make the mistake of manipulating the wrong object. This suggests that a policy can contribute a useful segment of a task even when it cannot complete the task alone: TiPToP’s failed attempt still staged the gripper next to the correct object, and the VLA completed the pick-and-place from that favorable start state.

Agents orchestrate policies in creative ways

Giving an agent the flexibility to combine policies and tool calls in any order led to the emergence of creative task-solving behaviors. The state machine setup suffices for a proof-of-concept, but is restrictive in its sequencing of policies in a rigid order. The orchestration should instead be flexible enough to allow any ordering of policies, skills, primitives, or tool calls. And the orchestration of those components should be dynamic and learned.

So I replaced the state machine with an agent orchestration framework. At its core is a reasoning VLM (Gemini) running a tool-use loop. Every turn, the agent sees the task instruction, the current camera views of the scene, a memory of every tool call it has made so far and what each one returned, and how much of its budget remains (wall-clock time and tool calls). It replies with brief reasoning and a single tool call. The tool executes, its result is appended to the memory, and the loop repeats until task success or a budget runs out. The policies themselves, TiPToP and π_0.5, are exposed as tools, alongside a set of motion primitives and perception tools (Figure 5). Policies can be invoked on the full task or the agent is free to break the task down into subtasks. The agent can leverage a VLM or Robometer to measure task progress, and the VLM can be invoked with any free-form question about the scene.

Figure 5. The agent orchestration loop: each turn, the agent picks a single tool, and the result flows back into its context before it decides again.

Benchmark runs with the agent orchestration harness demonstrate that novel behaviors can emerge. In one example, the agent chose to split a pick-and-place across the two policies. TiPToP planned the full pick-and-place and executed the pick. Since the harness pauses execution at each subtask boundary of the plan, the agent regained control after the pick. Instead of letting TiPToP continue, it abandoned the remaining plan and invoked π_0.5 to perform the place. One failure mode of TiPToP was that planned trajectories had insufficient clearance over obstacles, ending in collisions with the walls of target bins. Leveraging the TAMP method for the precise pick and switching to the VLA for the place avoided this failure mode and yielded a successful place.

Agent loops learn through trial, reflection, and memory

The natural next question, then, is: can agent policy orchestration with automated self-reflection be a scalable way to learn tasks, improving over repeated attempts without a human rephrasing the task’s language instruction or tweaking the underlying policies at every iteration? The short answer: yes, but not uniformly across all tasks. The learning loop improved success rates, but only on tasks of intermediate difficulty, neither out of reach for the base policies nor trivially solved by them, and only with tens of trials per task. With too few trials, the agent cannot reliably identify the causal factors of success. Learning takes both a sufficient trial budget and harness mechanisms that separate real lessons from coincidental events.

I ran the same agent framework in a repeated-trial loop: one task, attempted up to 60 times, with the loop ending early once the agent succeeds 6 times in a row. The loop ran in two settings: one where every attempt starts from the task’s original scene initialization, and one where the initial object poses are randomized at every reset. After every episode, a reflection step reviews the episode’s full tool-call trace with scene camera frames interleaved, and extracts concise lessons stored into memory. These lessons span task-specific and task-agnostic insights. Periodically, memory is compacted to keep the context finite. Each new task attempt then conditions on the memory in its agent prompt, detailing which attempt this is and how the previous ones went. The intended dynamic is explore-then-exploit: early attempts try varied strategies, later attempts converge on successful ones. After the learning loop finishes on a task, its memory is frozen, and the agent is evaluated on 60 episode rollouts.

I found that orchestration only helped on tasks within a certain band of difficulty. If a task is too difficult, no amount of exploration can solve it: orchestration is bottlenecked by the base policies. Even if the task can be solved occasionally, it is hard to efficiently change behavior from rare successes as it requires the agent to identify what specifically led to the success over a long episode, which could be due to a number of factors: which tools were invoked in what order, how the instruction passed to each tool was phrased, or the arm and scene state a tool happened to start from. These factors are sometimes hard to identify and reproduce. The underlying policies also carry inherent stochasticity of their own: the VLA samples its actions, and TiPToP samples the candidate grasps, placements, and trajectories that its optimizer refines, so even an exact replay of the agent’s decisions is not guaranteed to reproduce a success. On the other side of the spectrum, some tasks are easy enough for single policies, and orchestration does not add any benefit. The sweet spot lies somewhere in the middle: tasks hard enough that orchestration has something to add, but where successes recur often enough for an agent to isolate which decisions helped. Empirically, the tasks where the learning loop produced gains had baseline success rates in the range of roughly 20% - 50% before learning.

On tasks in this difficulty band, the agent is able to leverage its memory to improve task performance. In the pose-randomized setting, success rate rose from 50.0% to 78.3% on PhoneOrRemoteInBin, from 30.0% to 40.0% on CookingClearPlate, and from 25.0% to 28.3% on FoodPacking2Cans (Figure 6). Gains in the original-initialization setting were comparable or larger (48.3% to 88.3%, 31.7% to 53.3%, and 23.3% to 33.3%, respectively). Because the pose-randomized memory has to work across 60 different object arrangements, the learned lessons cannot simply encode a single layout.

To see what the loop actually learns, let’s walk through one task in detail, “Put the phone or the remote in the grey bin”, in the original-initialization setting. Even with a fixed layout, episode outcomes vary substantially due to the stochasticity of simulator physics, VLA rollouts, TiPToP planning, and the agent’s own tool choices, which is enough variation for learning behavior to emerge. The learning loop ran for 25 episodes before terminating on 6 consecutive successes (Figure 7). Its initial episodes are a mix of successes and failures where the agent explores different tool calls for actions that make progress. On the first episode, the agent learns that TiPToP fails to find a plan to move the phone but can plan a pick-place trajectory for the remote. During the failure streak in episodes 7 - 12, it logs several behaviors to avoid in memory (the learned memory itself is shown in Figure 8), including that invoking the VLA to clear the obstructing keyboard burns the entire step budget and calling place_on when nothing is grasped raises an error. The breakthrough comes in episodes 13 - 14, when TiPToP halts at the subtask boundary after the pick, and the agent calls the place_on primitive while holding the remote. With this strategy, it is able to complete the task. From episode 19 on, the agent validates this strategy to confirm it works, and closes the loop with 6 straight successes.

The other learning tasks followed the same explore-then-exploit arc: sparse successes in early episodes while the agent tries out strategies, then more frequent success once a reliable recipe is found. The length and failure rate of the exploration phase varied by task.

The agent memory outlines tool call combinations with episode usage counts, an avoid-list, and hypothesis tags ([hypothesis, n=1]) that get promoted as evidence accumulates (Figure 8).

As the agent learns strategies that work, it becomes more efficient at solving the task, reducing the number of tool calls required per episode (Figure 9). On PhoneOrRemoteInBin, successful episodes take a median of ~500 simulator steps with memory versus ~1,600 without memory, and agent turns per success drop from 8.2 to 3.5. Without memory, the agent retries failed strategies, and more than half of the episodes end in a timeout after long stretches of planner failures and futile VLA rollouts. With memory, π_0.5 is invoked much less frequently, as the agent learns that on this task, running π_0.5 tends to exhaust the step budget without making much progress.

Failure modes are demonstrated in Figure 10. Without memory, the agent directs π_0.5 to pick up the phone, and it instead tries to grasp the remote. This could have still worked given that the task instruction here is to place either the phone or remote in the bin, but π_0.5 is not able to perform a successful grasp on the remote. So the agent switches to TiPToP, but the planned trajectory is too low, and the remote is dropped due to a collision. To overcome this, the agent learns to switch to the place primitive after the TiPToP pick, where the arm is raised high enough to clear obstacles during its place motion.

Given the right harness and a sufficient number of task trials, an agent can learn from its past failures to improve task execution. But it does need repeated rollouts, on the order of tens of attempts per task, to get any real signal. With too few rollouts, the agent is prone to latch onto spurious behavior that may be uncorrelated with the true reasons for success or failure. On PhoneOrRemoteInBin, for example, one early success encoded the lesson “if an auxiliary scene-clearing action fails motion planning, attempt the primary task goal directly; the planner may still find a valid path despite visual obstructions,” which then failed when retried across several later episodes. Consistent with this, a breadth-first experiment that ran the learning loop across all 120 benchmark tasks with only two attempts per task did not improve performance, likely because two trials are not enough to extract causal insight from an episode’s many decisions.

Two harness mechanisms proved important for separating signal from noise. First, lessons enter memory as hypotheses with evidence counts and are only promoted as support accumulates across episodes, which keeps a single lucky episode from hardening into doctrine. Second, the loop encourages ablation tests: after a run of successes, a few attempts deliberately drop a step that every success shared to test whether that step was actually necessary, and the verdict is recorded back to memory.

Learned memory transfers across tasks

Lessons learned on one task carry over to another. I ran the agent learning loop on PhoneOrRemoteInBin, then bootstrapped the agent with that memory on a different task, FoodPacking2Cans. The cross-task memory lifts success from 23.3% to 31.7%, within 1.7 points of the 33.3% that the task’s own learned memory achieves (Figure 11). Cross-task transfer is the ultimate goal: an agent that accumulates experience continually, each new task drawing on the lessons of the ones before it rather than starting fresh.

Episode traces show the transplanted lessons at work (Figure 12). The agent learns to pivot away from retrying π_0.5 multiple times on either the same or a similar instruction (33% of turns down to 22%), and switches to other tool calls more quickly. VLA step-budget timeouts drop from 254 to 166, and place_on-with-an-open-gripper errors fall from 79 to 59 (both are entries saved into memory on the avoid-list). As all episodes are timestep budget-bounded, more efficient strategies have a higher likelihood of success. The lessons also shape the instructions themselves: after an unknown-object planning error, the transplanted-memory agent’s next TiPToP call reuses an exact object name mined from the error message 87% of the time. In one episode, after TiPToP repeatedly failed to ground “canned_food_2”, the agent reasoned “The lesson notes suggest adding specific visual attributes (e.g. color descriptors) to TipTop instructions to help ground objects,” and retried with “the red and white soup can”.

Looking forward

So, given all this, is agent orchestration of multiple robot policies the way forward? My view is that it can bring value to parts of the robot learning pipeline, but I actually don’t believe it will be the dominant deployment paradigm. Trajectories from orchestrated policies can be suboptimal and misaligned with how an expert would perform a task. And I think any behaviors learned through orchestration may be distilled or finetuned back into the policies at scale. I believe that promising ways to leverage policy orchestration in robot learning include training data collection, incorporating it into RL exploration and learning algorithms, and continual learning.

Some trajectories produced from robot policy orchestration are smooth, avoid collisions, and complete the task efficiently. Others are less clean and make mistakes along the way, but end up succeeding on the task. With a mechanism to filter for high-quality trajectories, agent-orchestrated rollouts can be one source of data for supervised learning. Methods can also learn from the mixed quality data by conditioning on a quality label, or recovery behavior can be trained on suboptimal rollouts.

With the reasoning layer handled by strong VLMs, policy orchestration can also be a scalable source of feedback on where and how policies fail. Policy rollouts can be scaled up in simulation across a wide variety of scenes and task conditions. Efforts are underway to enable this feedback loop in the real world too (ENPIRE). An agent can mine this data in an automated manner, analyzing successful and failed trajectories to produce insights. This can inform which data should be prioritized for collection, to be used in the next training run for whichever base policies are being used (VLAs, WAMs, etc.). And it can serve as useful knowledge for where these policies will succeed in the real world at deployment-time.

Another setting where policy orchestration might be valuable is reinforcement learning or continual learning. Reinforcement learning relies on a policy being able to explore its environment and try novel actions that might improve performance. It can be challenging to steer a policy to explore in a meaningful and efficient way. Leveraging policy orchestration as the exploration steering signal can be one avenue to induce behavior that is sufficiently different from the base policy. Where exploration is otherwise driven by injected randomness, an orchestrator could instead source exploratory actions from its other policies, each offering a qualitatively different mode of behavior.

I think policy orchestration can be a rich source of signal for robot learning. The key will be figuring out how to best incorporate that signal into learning algorithms.