Securing Trustworthy XR Interactions through Human-Machine Networks in Healthcare

The primary research activities conducted during Fall were broken up into literature review of XR collaborative systems in a medical context, DepthKit Studio configuration and collaboration with  

University Libraries/ARIES, system exploration for alternative collaboration systems. 

Literature Review 

Literature review was conducted of research results and existing systems that fit our collaboration model  (local user and remote expert). Papers were first characterized based on the collaboration configuration,  the data transferred, and context.

We then synthesized new labels for the most common sensors/systems to examine trade-offs between   feasibility in deployment, user experience, and potential privacy risks. 

DepthKit Studio Testing 

The project team initiated a collaboration with ARIES and the University Libraries to start the configuration of tele-operation system  for VR/AR. This was inspired by prior work from Cornell Tech published at an IEEE VR workshop. 

However, during this process we encountered many roadblocks in terms of calibration, capture of the user  and the environment in one run, networking captured data to a nearby computer, and compatibility of the system with a Magic Leap 2 AR headset. 

System Exploration 

Beyond DepthKit Studio, we reviewed other tele-operation systems for VR/AR, including Magic Leap’s  Remote Assist and Workshop software, MakeSEA Catapult, and building a tele-operation networked  application from scratch using Unity and 360-degree videos. From this evaluation, we determined that  MakeSEA Catapult and Magic Leap Remote Assist are the most viable options. Additionally, we explored  the potential to integrate generative AI for the target task through Microsoft’s CoPilot AI, which offers to  provide automated annotations and responses for frontline workers using AR headsets. So far, we have  not been able to identify a useable system for this situation, but plan to explore this recent breakthrough in  AI in the context of remote healthcare when using spatial computing/mixed reality devices. 

Spring 2024 Plans 

Based on our results, we have determined a modified plan for Spring 2024. First, we will conduct up to  two design or focus group workshops on the context of emerging technologies, in line with the original  proposal. The only modification to these workshops would be related to a more interactive workshop  centered around and using generative AI mechanisms (language and image synthesis models). Second, we  narrowed our scope for the preliminary remote user study to make use of Magic Leap 2 Remote Assist, as  the plug in play solution allows us to execute a user study with three clear conditions, no remote assist  (baseline), remote assist using only the first-person view from the local client, and remote assist using  both the first-person view and the third-person view enabled by the environment sensing built into the AR  device. The user study can then evaluate the ability of remote experts (medical students) to guide novices  (general participants from the VT community) through the remote medical task to evaluate usability and  privacy concerns. Based on our literature review, this setting is representative of a practical and feasible  deployment to areas with limited access, as it only involves sending a mobile lightweight headset to the  client instead of a ten sensor DepthKit Studio set up that requires precise calibration. The user study is in  line with the originally proposed work, only remove the need to require the remote expert to use a VR  headset and instead fall back to the use of a traditional desktop setting for guidance, and includes the tools  and gestures originally proposed from a VR remote expert and the produced risks to viewing the local  client’s home environment during guidance.