The team has concluded the project, Wireless Psychophysiological Recording in the Augmented Reality Environment, with the presentation of our work at the ICAT Creativity + Innovation Day on May 6. This work exhibited the final creative development of the Immersive Storytelling Team (IST), informed by psychophysiological research from the Mind-Body Lab (MBL), and enhanced into a Virtual Reality (VR) environment by the Virtual Environments Team (VET) in computer science. Because the VET had to share limited resources with other computer science exhibits at the ICAT Day, they did not have access to either of their department’s two Microsoft HoloLens augmented reality (AR) goggles. As a result, VET made the strategic decision to change the final exhibited format from the originally-proposed AR to VR instead. The following is the final report for the project supported by ICAT’s Student SEAD grant for Spring 2019.

As stipulated in the grant, ICAT purchased the wireless physiological equipment (BioNomadix Non-Invasive Cardiac Output System [BN-NICO] plus leads and electrodes) and made it available to the project team on April 15. It was discovered that the initial Biopac shipment did not include one of the necessary sets of leads. The replacement leads were received on May 2.

While awaiting this equipment, the MBL developed, tested, and revised the research protocol. The protocol was piloted using pre-existing non-wireless impedance cardiography (ICG) equipment. As expected, pilot participants were severely restricted in their movement when attached to the non-wireless equipment. Additionally, movement artifact frequently obscured the non-wireless ICG signal whenever pilot participants moved their limbs, head, or even laughed.

When the ICAT-funded wireless equipment was finally tested in May, it out-performed the non-wireless equipment in several respects. First, participants experienced a significant increase in their range of available movement. They could sit down, stand up and even move about the room without hinderance from the recording equipment. Movement artifact was also reduced. Movement in the head and limbs did not produce the extent of movement artifact that the non-wireless equipment did. Certain excessive movements still led to movement artifact even with the new wireless unit, but artifact was reduced.

Wireless Psychophysiological Recording in the Augmented Reality EnvironmentWhile the research protocol was being piloted, the IST was developing the storyboard for the research stimulus. A first draft storyboard was presented on March 29, and revisions from the VET and MBL were incorporated and presented on April 11. IST also developed a proof-of-concept video to augment the storyboard. The storyboard and the accompanying video were entitled “Diagnostik” and were meant to be reminiscent of a darker era in the history of psychology when psycho-therapists used the Rorschach Inkblots ostensibly to uncover unseen elements of subject’s psyche. The Diagnostik video begins with seemingly innocent inkblot images, and then progresses through a series of images that take on an increasingly sinister tone. The primarily black-and-white imagery combines with a soundtrack that also starts sounding innocent but progresses to more sinister tones. Ultimately, the inkblots were scripted to take on the form of a threatening creature.

Also simultaneous to the psychophysiological pilot testing, the MBL added the Creative Technologies Team (CTT) from Wireless Psychophysiological Recording in the Augmented Reality Environmentthe School of Visual Arts to the collaboration in mid-March. Consisting of 3 undergraduate students under the mentorship of Professor Les Duffield, the CTT was charged with developing the digital assets needed to bring IST’s storyboard to life. CTT collaborated with IST throughout the month of April and delivered concept sketches on April 29. CTT has taken IST’s sketches and storyboard and begun to develop a “monster” that will take shape from the inkblots and appear to approach the research participant during the survival scenario.

CTT’s concept sketches and digital assets had not yet been rigged by early May. VET therefore made the strategic decision to purchase an animation asset to represent the “monster” under development for ICAT Day. VET also developed a VR environment to replace the natural AR environment that was originally scripted but inaccessible due the to the department’s Hololens shortage. VET then incorporated IST’s proof-of-concept video into the VR environment in a way that appeared as if the VR participant were viewing a projection of the video onto a wall in the VR environment. Finally, VET incorporated the animated monster into the scenario, making the monster appear to crawl through the projection into the VR space. The ultimate goal was to engage participant’s survival behavior through an increasing sense of threat.

For the final exhibition at ICAT Day, the MBL incorporated a display of live, streaming physiological data to supplement and help inform the VR experience. The original plans were to attach each participant individually to the physiological recording equipment, including the ICAT-funded BN-NICO. However, for each exhibit visitor the physiological equipment would require 5 minutes for electrode placement and then ideally 15 additional minutes for the data to settle and calibrate. In order to move participants more efficiently through our exhibit, we simply attached an MBL research assistant to the equipment and kept a live-stream of her physiological data running as a complement to the VR projection that participants were experiencing.

When the final product was exhibited at the ICAT Creativity + Innovation Day, visitors to the exhibit generally seemed to experience the intended sense of threat. Several were photographed in various stages of avoidance behavior. The MBL plans to leverage these poses to develop a behavioral coding system that categorizes behaviors that represent different stages of the survival progression.

While the exhibit at ICAT Creativity + Innovation Day was considered a success, most of the team plans to continue our collaboration into the summer, furthering our research and development. IST and CTT are continuing development of the survival progression including the story, sound design, and the animation. The grant’s Principal Investigator, Brittany Nackley will be seeking IRB approval for this survival research over the summer, with the goal of running participants this fall. The IST and CTT both commented that having a research goal has helped focus their creative development and given their work an additional sense of purpose. The MBL has observed how the creative collaboration has breathed new life and enthusiasm into a previously-stagnating line of psychophysiological research.

Related projects