Two New ILLIXR Papers: Boba and SemanticXR Advance Next-Generation XR Systems

The ILLIXR group is excited to share two new papers that address complementary challenges in next-generation XR systems: efficiently simulating interactive physical digital twins and enabling low-power devices to understand and query their surroundings in real time. Boba will appear at ECCV 2026, while SemanticXR will appear in IEEE TVCG / ISMAR 2026.

Boba: Batched Simulation for Physics-Based Gaussian Digital Twins — ECCV 2026

Authors: Yihan Pang, Hanxiao Jiang, Sushant Kondguli, Sarita Adve, and Shenlong Wang

Boba is a collaboration involving the ILLIXR team, Prof. Shenlong Wang at UIUC, Columbia University, and Meta Reality Labs Research. It is a systems framework for making Gaussian-based physical digital twins practical for interactive XR, while also supporting high-throughput robotics workloads.

Boba's key idea is template–state separation and system co-design, which lets the same digital-twin stack support low-latency single-instance execution, distributed edge–server deployment, and large-scale batched execution. For XR, Boba can run entirely on an edge device or split simulation and skinning onto a server while keeping rendering and visualization on the XR-class device. We have also built an interactive Quest 3 XR prototype demonstrating physical-digital-twin manipulation in an immersive environment.

Key highlights:

• Delivers over a 10× end-to-end speedup over PhysTwin across both edge-class and workstation-class platforms
• Reaches 30.8 FPS on an XR-class Jetson Orin and 204.5 FPS on a single RTX 4090
• In distributed deployment, reduces end-to-end latency from 32.5 ms to 25.1 ms while lowering device-side incremental power by 22.2%
• Demonstrated in an interactive Meta Quest 3 XR prototype
• Reaches 3,310 FPS average aggregate throughput and supports up to 688 parallel instances on a single RTX 4090
• Accelerates MPC rollouts by 26× for rope manipulation and an estimated >2,000× for cloth manipulation under matched self-collision settings

Together, these capabilities make Boba a systems foundation for efficient, interactive physical digital twins in XR, while providing the scalability needed for demanding robotics and learning workloads.

SemanticXR: Low Power, Real-time Queryable Semantic Mapping with a Device-Cloud Architecture — IEEE TVCG / ISMAR 2026

Authors: Rahul Singh, Devdeep Ray, Connor Smith, and Sarita Adve

SemanticXR is a collaboration between the ILLIXR group at UIUC and NVIDIA. It introduces the first end-to-end device-cloud system for real-time, open-vocabulary semantic mapping and querying under the power, bandwidth, and memory constraints of mobile XR devices.

SemanticXR's key idea is to treat identifiable objects as first-class units of computation, communication, and memory management. This object-level organization enables efficient cloud-based semantic mapping while maintaining a sparse local map on the device for fast, network-robust queries. Semantic maps allow XR applications to understand not just the geometry of a physical environment, but also the identities and meanings of objects—for example, allowing a user to ask, “Where are my keys?” and have the system locate and highlight them.

Key highlights:

• Delivers 2.2× faster server-side semantic mapping at equivalent semantic quality
• Keeps upstream bandwidth below 2.5 Mbps
• Supports sub-100 ms queries for up to 10,000 objects, even during network drops
• Supports tens of thousands of objects within a 500 MB device memory footprint
• Adds only about 2% device power over idle during normal operation
• Demonstrated end-to-end on both Meta Quest 3 and Apple iPad Pro

Together, these capabilities make SemanticXR practical for persistent, intelligent scene understanding on power- and resource-constrained mobile XR devices.

Together, Boba and SemanticXR advance the ILLIXR vision of XR systems that can both understand the physical world and interact with realistic digital representations of it, while operating within the latency, power, bandwidth, and scalability constraints of mobile XR.