Starter Tasks

Starter Tasks for New Students Interested in Joining ILLIXR

Introduction

Research with ILLIXR requires proficiency in C++ and Linux. We designed a starter task to help you set up ILLIXR on your own machine and experiment with it.

You are welcome to use LLMs to explain concepts, usage for libraries or CMake, or look at error messages for you, but feeding any part of the ILLIXR codebase into an LLM or using agentic AI to complete these tasks is not allowed. When you are matched with a project after completing these tasks, we will ask you to explain your thought process and solution, so establishing an understanding of the codebase without the help of LLMs is key.

Prerequisites

ILLIXR currently supports CentOS 9, Fedora 37/38, and Ubuntu 20/22. If you are not on one of these OS, we highly recommend that you install Ubuntu 24.04 on your machine. Alternatively, you can use a VM such as VirtualBox, but it lacks graphics acceleration.

If you are on macOS, we recommend that you set up Ubuntu with UTM. If you are on one of the later MacBooks with an arm64 CPU, you might want to use this image for Ubuntu 24.04 to avoid emulation. Do not turn on graphics acceleration due to issues in UTM's OpenGL -> Metal translation layer.

Task 0

To get a sense of the XR research landscape and what ILLIXR is about, please read the ILLIXR Paper if you have not already.

Task 1

Set up ILLIXR on your machine using the native_vk profile.

The goal of this task is to set up ILLIXR on your machine for development. ILLIXR has a modular design where features are compartmentalized into plugins that are dynamic libraries. We have profiles that each enables and configures a different set of plugins. The profile we want you to set up is called native_vk, which runs a lightweight test application using the Vulkan graphics pipeline.

The following documentation might be helpful. We recommend that you try to understand what each step does instead of just copy-and-pasting the commands:

Note:

After cloning, checkout commit 60982daa051dacb97f382bfc76fad27f08ab876f as we are currently releasing a new version and the master branch might break
To understand why we need to set the LD_LIBRARY_PATH environmental variable, refer to the man page here.
If you’re on Ubuntu, you additionally need the following packages
- cmake
- glslang-dev
- libboost-all-dev
- Compiler of your choice

Task 2 (Optional)

In ILLIXR, plugins pass data around using what we call Switchboard, an event-driven messaging interface using shared memory. Documentation of Switchboard can be found here. In the Debug View window, the Onboard Camera View that displays the stereo camera feed is located on the bottom right as shown in the screenshot below. The camera feed is supplied by another plugin to a Switchboard topic, from which the Debug Window reads.

Identify where the camera feed is published and apply a Gaussian filter to the stereo feed. Feel free to use any σ . You should not modify the dataset or the Debug View itself.

Task 3 (Extra Brownie Points)

As an XR runtime, ILLIXR needs to keep track of the user's head pose. In the native_vk profile that we are using, this is implemented with OpenVINS, a library for Visual Inertial Odometry (VIO). You might have noticed that OpenVINS also reads from the camera feed. Please research a bit into how VIO works on a high level using any resource you like.

Based on your understanding, will the Gaussian filter in Task 2 help, hurt, or have no effect on the accuracy of VIO? Modify the value of σ and compare the headset’s trajectory with and without the filter. How would you compare the trajectories? Does the accuracy get better or worse as σ increases? Can you offer an explanation?

Resources:

Here's a blog post that offers an intuitive understanding of how VIO works
SIFT is a keypoint extraction algorithm that VIO might use
The ORB-SLAM paper. You are welcome to use an LLM to explain CV concepts such as the fundamental / essential matrices

Submission

Submit your results via this Google Form.

Task 1: a screenshot of ILLIXR’s debug and eyebuffer window
Task 2 (Optional): your code, a screenshot of ILLIXR’s debug window after the modification, and a brief write-up
Task 3 (Optional): a brief write-up and any code you would like to submit

Getting Help

Please join the new-students channel in our Discord server if you have questions or need help with these tasks.

Accessibility

The task is designed to run on any machine that meets the College of Engineering Computer Recommendations. If you do not own a computer or your machine has broken down, please feel free to reach out for accommodation options.