Unreal Engine Apps & Tools

This app lets you control virtual production sets, which use wrap-around screens powered by Unreal Engine to provide backdrops and lighting for TV and film production. With Unreal Stage, operators can control Unreal Engine directly from the stage using an iPad.

I was the main programmer for this app, which I built from the ground up based on designs provided by Epic Games. This work included:

• Collaborating with designers at Epic Games to build an intuitive user experience for features that didn't yet have industry-standard tablet controls and regularly iterating based on user feedback.
• Developing a network protocol for smooth, responsive slider controls, even in poor network conditions or when multiple users adjust the same setting simultaneously. I took inspiration from my background in games and based my approach on rollback netcode.
• Developing a network discovery protocol allowing users to easily connect to compatible Unreal Engine instances on the local network.
• Creating a suite of widgets for rapidly building Flutter apps with a standardized Unreal Engine style, which would go on to be used in all other apps I built for Epic Games.
• Implementing a system in Unreal Engine for capturing and transmitting stage previews to the app, separate from the usual rendering stack.

This app lets you use your mobile device as a virtual camera, meaning you can remotely control and view a camera in Unreal Engine by moving a phone or tablet in physical space.

Unreal VCam already existed as an iOS app, but after the success of Unreal Stage, I was brought in to rebuild the app in Flutter with cross-platform support. This allowed us to both release a highly anticipated Android version of the app for the first time, as well as to give it a facelift and some new features in the process.

My work included:

• Rebuilding the Unreal VCam interface using the reusable components I built for Unreal Stage.
• Porting existing app logic (written in Swift) to Dart for cross-platform support.
• Implementing support for low-latency video streaming, AR integration, and gamepad support, each of which involved writing native extensions for both platforms.
• Porting over the network discovery protocol I built for Unreal Stage, making it easier for users to connect to Unreal Engine.
• Extending the app's existing networking protocol to support new features requested by designers at Epic Games, like triggering custom dialogs in the app from Blueprint.

This app lets you record facial performances on a mobile device and play them back live on realistic 3D models in Unreal Engine.

An iOS version of the app already existed, but I was brought in to create an Android port, as well as to implement a sweeping new set of features. The biggest new feature was the ability to see animation results in real time instead of waiting for them to be processed after recording, which required special attention to the app's performance.

Highlights of my work include:

• Integrating a machine learning model into the app for the real-time animation feature. I was not involved in creating the model itself, but worked with the research team to compile it for Android and write high-performance binding code that could maintain a steady frame rate of 30+ FPS.
• Navigating Android's rapidly changing machine learning ecosystem to support a variety of chipsets via the LiteRT framework. This involved working directly with developers at Google to experiment with cutting-edge features and optimize for real-time performance at a time when the existing tools were focused on higher-latency models.
• Adding USB camera support to the app, allowing for the use of specialized facial capture cameras. Android has no native support for USB cameras, so I had to write a custom C++ extension to communicate with lower-level libraries (libUVC and libUSB). This feature was unique to the Android port and ended up being extremely well received, so the effort was worth it!

On top of my app work, I also made a number of contributions to Unreal Editor itself, including:

• Extracting and expanding on the Color Grading Panel, making it usable for game projects for the first time and giving it a nice facelift in the process.
• Adding quality-of-life features to the Console Variables Editor.
• Adding minor features and addressing existing bugs to the Remote Control plugin.
• Fixing bugs and adding an automated testing suite to the nDisplay plugin.

If you have access to Unreal Engine, you can find a full list of my public contributions on the project's GitHub.