GSOC Project List

Google Summer of Code Project Detail

Generalized inter-process communication (IPC)

ILLIXR uses a method of IPC to explore its offloading work. Furthermore, OpenXR applications also communicate through an IPC implemented by Monado, another project that we use to interface with OpenXR. However, since OpenXR is still developing as a whole, there are also instances where we would like to be able to communicate with ILLIXR outside of the OpenXR specification. This requires another IPC implementation.

In order to avoid repeated implementations of IPC, it would be more efficient for ILLIXR to implement a generalized interface for IPC. Ideally, this means that ILLIXR has a single plugin (e.g. a service plugin) which can communicate with other plugins.

Expected outcome: The participant expects to build a plugin for generalized IPC. This should at least support our current IPC use-cases.

Prerequisites: C++, experience with IPC

Contact person: Qinjun Jiang, Sainath Ganesh, Jeffrey Liu

Duration Size: 175 hours

Difficulty: Medium

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

Foveated Rendering

Foveated rendering is a rendering technique that only renders at full-resolution in regions of interest. In the context of XR, the area of interest is generally around the user’s gaze. If done correctly, the periphery can be rendered at a lower resolution without noticeably impacting the user’s quality of experience (QoE). This allows XR devices to achieve more efficient rendering in terms of performance and power consumption.

Although developers are aware that foveated rendering can have its benefits, there are many factors that have yet to be considered. For example, there may be different tradeoffs involved depending on which rendering technique is used (e.g. rasterization or raytracing). The degree of acceptable foveation may also vary depending on scene complexity (e.g. geometric detail, textures, etc).

In order to analyze this more thoroughly and part of an end-to-end system, ILLIXR needs a foveated rendering pipeline. This should support some degrees of scene complexity, e.g. textures. There are a few possible methods of implementation:

Rendering the same scene at multiple different resolutions, and then compositing the frames together. Refer to the gldemo plugin as a simple toy example of how ILLIXR handles rendering scenes manually.
Using Vulkan extensions for variable rate shading (refer to NVIDIA’s VRS SDK demos).

For those interested in the project, please contact Rahul Singh or Jeffrey Liu to discuss the proposed method of implementation. Proposals without prior discussion will likely not be approved.

Expected outcome: The participant expects to build a foveated rendering pipeline for both rasterization and raycasting in ILLIXR. This should allow ILLIXR to experiment different foveation parameters in varying degrees of scene complexity.

Prerequisites: C++, familiarity with graphics APIs (OpenGL, Vulkan)

Contact person: Rahul Singh, Jeffrey Liu

Duration Size: 175 hours for implementation of the rendering pipeline, 350 hours for additional research explorations

Difficulty: Medium/Hard (depending on implementation details)

Offloading Rendering

Rendering is one of the largest consumers of compute and power on XR devices. In this project, we want to implement the rendering offloading infrastructure within ILLIXR, which will enable future research both algorithm-wise and networking-wise. The infrastructure will include but not limited to the transmission and reception plugins on the server and device and appropriate codec pipelines for the rendered frames.

Expected outcome: The participant expects to build an offload state-of-art rendering plugin in ILLIXR and explore potential optimizations that explore above stated benefits.

Prerequisites: C++ and (optional) some familiarity with rendering processes in some game engines (Godot, Unreal, or Unity)

Contact person: Qinjun Jiang, Rahul Singh, Steven Gao, Jeffrey Liu

Duration Size: 175 hours for implementation of the offload module, 350 hours for additional research explorations

Difficulty: Medium

Offloading Scene Reconstruction

Scene reconstruction is the process of creating a 3D model of a real-world environment. Scene reconstruction, especially real-time scene reconstruction, is a key technology for Extended Reality (XR). It provides the basis for creating realistic and interactive XR experiences by accurately reconstructing the real-world environment.

Unfortunately, current real-time scene reconstruction does not consider an environment as resource-constrained as an XR system, where the power budget for XR systems ranges from a few Watts to a few hundred milliwatts. Most scene reconstruction algorithms are developed and tested on desktop- or laptop-grade systems, which consume one to two orders of magnitude more power than untethered mobile XR devices. While some approaches can run on mobile devices, they still consume significant power.

This project explores the power consumption reduction in future XR devices by analyzing the potential of running scene reconstruction algorithms in the cloud. Offloading scene reconstruction to the cloud can bring several benefits to users.

Offloading scene reconstruction reduces the computational burden on local devices, allowing for more efficient use of limited resources and improved performance.
Offloading scene reconstruction to the cloud enables access to high-performance computing resources, such as GPUs/domain-specific accelerators, which are unavailable on most local devices.
Offloading scene reconstruction to the cloud allows for collaboration and data sharing between multiple users and devices.
Offloading scene reconstruction to the cloud provides a more reliable environment for processing sensitive data.

Expected outcome: The participant expects to build an offload state-of-art scene reconstruction plugin in ILLIXR and explore potential optimizations that explore above stated benefits.

Prerequisites: C++ and (optional) some familiarity with existing scene reconstruction algorithms (InfiniTAM, KinectFusion etc.)

Contact person: Yihan Pang, Rahul Singh, Steven Gao, Henry Che

Duration Size: 175 hours for implementation of the offload module, 350 hours for additional research explorations

Difficulty: Medium

Integrating More Datasets

The extension of the ILLIXR to encompass a diversity of benchmarks for offline experiments is imperative for a number of reasons:

Diverse benchmarks demonstrate the reliability and adaptability of the tested XR system, giving a complete understanding of its strengths and weaknesses.
Diverse benchmarks comprehensively evaluate the system's performance across various tasks, providing a more accurate picture of its capabilities and limitations.
Offline experiments, in particular, allow for controlled and repeatable testing, allowing for accurate and consistent comparisons between different versions of the design or with other designs.

At the moment, ILLIXR mainly uses the EUROC dataset for offline experiments. This project aims to extend ILLIXR's support to various new datasets. Participants first start exploring creating plugins for each new dataset separately. Then, after participants get a more comprehensive understanding of the ILLIXR and incorporated datasets, participants can research and design a new generic interface for ILLIXR that is compatible with existing datasets and is flexible to support future datasets.

Expected outcome: The participant expects to implement support for other datasets in ILLIXR.

Prerequisites: C++

Contact person: Yihan Pang, Henry Che

Duration Size: 175 hours for implementation of each individual datasets, 350 hours for designing the new generic interface.

Difficulty: Medium