SCOUT: open-vocabulary detection on mobile robots

Mobile robots are increasingly asked to find objects described in plain language. Open-vocabulary detectors make that possible, but a model that scores well on static images can still be far too slow to guide a moving robot on small, power- and compute-limited hardware.

A framework that judges detectors on speed, not just accuracy

SCOUT turns detector selection into a deploy-or-reject decision across three gates: benchmark accuracy, on-device feasibility, and real closed-loop behavior. Its core idea is the perception gap, the distance a robot travels between detections, which makes a detector’s frame rate a physical consequence. It runs on a real AgileX LIMO with a Jetson Orin Nano.

When is a detector actually deployable, not just accurate on a benchmark?

A synthetic Unity benchmark before any real trial

This was a group research project at NYU Abu Dhabi’s eBRAIN Lab with Prof. Muhammad Shafique and Research Engineer Abdul Basit. My contribution was the Unity simulation side: a synthetic indoor environment that generates a 1,500-image dataset with domain randomization, attribute prompts, and impossible prompts, exported with COCO-format labels to stress-test how detectors handle robot-like scenes before risking hardware.

Unity synthetic dataset

The result is that deployability diverges from the accuracy leaderboard. Compressed transformer detectors kept their scores but ran under 1 Hz on the robot, opening half-meter perception gaps, while YOLO-World-S FP16 was the only configuration that met the timing budget and sustained continuous search across 40 real-robot trials.

Domain randomization

The work is under review at CoRL Conference 2026.