Mingke Lu

MIGU grounds multimodal human instructions by combining language with uncertain pointing gestures. The framework models geometric uncertainty in 3D and fuses it with semantic priors from a vision-language model to form a belief over intended objects and regions. This belief helps the robot decide whether to act or request clarification, then supplies goals for mobile manipulation and tabletop task-and-motion planning. Real-world evaluations demonstrate improved grounding over the evaluated baselines and show the benefit of explicitly modeling uncertainty across modalities.

Long-term monitoring of numerous dynamic targets can be tedious for a human operator and infeasible for a single robot, e.g., to monitor wild flocks, detect intruders, search and rescue. Fleets of autonomous robots can be effective by acting collaboratively and concurrently. However, the online coordination is challenging due to the unknown behaviors of the targets and the limited perception of each robot. Existing work often deploys all robots available without minimizing the fleet size, or neglects the constraints on their resources such as battery and memory. This work proposes an online coordination scheme called LOMORO for collaborative target monitoring, path routing and resource charging. It includes three core components: (I) the modeling of multi-robot task assignment problem under the constraints on resources and monitoring intervals; (II) the resource-aware task coordination algorithm iterates between the high-level assignment of dynamic targets and the low-level multi-objective routing via the Martin’s algorithm; (III) the online adaptation algorithm in case of unpredictable target behaviors and robot failures. It ensures the explicitly upper-bounded monitoring intervals for all targets and the lower-bounded resource levels for all robots, while minimizing the average number of active robots. The proposed methods are validated extensively via large-scale simulations against several baselines, under different road networks, robot velocities, charging rates and monitoring intervals.

We propose a Path-Tracking Hybrid A* planner coupled with a hierarchical Model Predictive Control (MPC) framework for path smoothing in agricultural vehicles. The goal is to minimize deviation from reference paths during cross-furrow operations, thereby optimizing operational efficiency, preventing crop and soil damage, while also enforcing curvature constraints and ensuring full-body collision avoidance. Our contributions are threefold: (1) We develop the Path-Tracking Hybrid A* algorithm to generate smooth trajectories that closely adhere to the reference trajectory, respect strict curvature constraints, and satisfy full-body collision avoidance. The adherence is achieved by designing novel cost and heuristic functions to minimize tracking errors under nonholonomic constraints. (2) We introduce an online replanning strategy as an extension that enables real time avoidance of unforeseen obstacles, while leveraging pruning techniques to enhance computational efficiency. (3) We design a hierarchical MPC framework that ensures tight path adherence and real-time satisfaction of vehicle constraints, including nonholonomic dynamics and full-body collision avoidance. By using linearized MPC to warm-start the nonlinear solver, the framework improves the convergence of nonlinear optimization with minimal loss in accuracy. Simulations on real-world farm datasets demonstrate superior performance compared to baseline methods in safety, path adherence, computation speed, and real time obstacle avoidance.