For the third and final trimester of my MSc program, I had the options of an internship, a Summer project, or a research thesis. With the tech market downturn, I had failed to secure an internship, so I joined a group of my colleagues to work on a Summer project instead. We were presented a choice of topics, and had to craft pitches for them. Most revolved around LLM’s, the hot topic of the moment, but there was also one to work on an XR project sponsored by Microsoft. That was the most appealing to me, and I was able to share my enthusiasm with the team as well. We were the only team that was interested in that project, and so we were selected for that one with no difficulties.

We had a few ideas for XR applications, but embracing the multi-cultural and polyglotal aspects of our team, we focused in on an AR translation application. The concept, and what we eventually built and implemented, was a translation application for the Microsoft HoloLens2, which used the device camera to capture an image, upload it to a cloud image recognition service, spot key objects, translate those objects into a target language, and cast AR captions into the environment over those objects.

The initial concept was fairly simple to build, and our team had decent experience with the Unity Game Engine, so we were quickly able to implement aspects of the Mixed Reality Toolkit (MRTK), and integrate with Microsoft Azure’s backend cloud services for image recognition and translation. We also experimented with running our own ML services, but the cloud options definitely supported the application and the hardware better. The entire project and its results are outlined pretty well in the final report that we delivered.

We were faced with some key challenges in the project, mainly that we only had access to the device hardware for about an hour once a week at Microsoft, so the turnaround time on our testing was very slow, and not conducive to the rapid iteration that was needed. We also pushed the boundaries and limitations of what the HoloLens 2 hardware was capable of, which sent us back to the drawing board multiple times on some of our tests and experiments. That said, we had wonderful support from the class mentors, and our contacts at Microsoft who took time to meet with us each week, and even did some remote testing for us on occasion as well.

We had a chance to test our application with classmates towards the end of the project (though earlier testing would have been much preferable had it been possible). Then at the end of the course, we had the opportunity to present our project to Microsoft Engineers with a live demo of our application. It went well and we compiled our final report of the project, where we received much praise, and most importantly, a nice grade for our efforts. I was really happy with my team, the incredible work that they did, and the loads of support we received throughout the course to make our project work, and I’m looking forward to whatever project comes next. I definitely hope to do more XR stuff in the future.