When leaders at the Center for Academic Innovation announced an XR initiative a little more than a year ago, they knew there was a hunger from a growing segment of campus to advance teaching through the development of augmented and virtual reality technologies.

They couldn’t have anticipated, however, that a global pandemic would so disrupt education that faculty eager to make sure their students didn’t miss out on valuable learning experiences would turn to augmented, virtual and mixed reality solutions for hands-on, in-person teaching that could not occur—at least in its usual form—in a COVID-19 world.

Among the XR solutions are three projects using Microsoft’s HoloLens2 technology for medical rounds, nursing training and construction architecture.

“Through an unrelenting 2020 we’ve gained a greater appreciation of the hidden and complex systems that shape our lives,” said James DeVaney, associate vice provost and founding executive director of the Center for Academic Innovation. “As problem spaces become more complex and the stakes feel higher than ever before, our pre-pandemic predictions about the transformative power of XR technologies put us in an incredible position to create the rich and immersive learning experiences that will soon feel foundational in a post-pandemic university environment.”

As XR Initiative Director Jeremy Nelson takes stock in the first year, it’s a two-thumbs-up in terms of reaching the original goals set for year one: the first funded faculty projects are moving along nicely and a massive-open online, three-course specialization that spun off of a successful teach-out by the School of Information’s Michael Nebeling launched last week.

But Nelson also gets excited talking about the unexpected projects that came about because faculty and staff in the Medical School, School of Nursing and others tapped the center for how they might use the technology to continue to offer important hands-on experiences during emergency remote learning and after hybrid online and socially distanced learning return to classrooms this fall.

“The first year of the XR Initiative has been an exciting time to build upon all of the great work that was already happening here at Michigan,” Nelson said. “We have accomplished a lot in this first year related to bringing together a broader community of faculty, students and staff interested in XR technologies and laying out a strategy to integrate these technologies into curriculum across the university.

“I am excited about the team we have assembled, the projects we are working on, and how we can lean into the challenges that COVID has presented to offer new and exciting opportunities for students and faculty to engage in learning in new ways.”

Meeting the original promise

The initial charge of the XR Initiative was to seed new projects that would integrate XR technologies into courses within the university’s 19 schools and colleges, as well as U-M’s growing portfolio of online courses.

A December 2019 call for proposals to advance learning and teaching using AR and VR yielded 16 proposals, of which eight projects from seven schools and colleges were funded.

Although COVID-19 threw a few curve balls, faculty in architecture, nursing, engineering, physics, performing arts technology, dentistry, medicine and information have begun integrating XR technology into courses and programs throughout the academic year. Another new round of proposals has been received with similar enthusiasm and a new round of selected projects will be announced in early January.

Student Caitlyn Chua, architecture. Image courtesy: Academic Innovation

Before the initiative was announced, 12 schools and colleges had begun the work to put together a graduate certificate program featuring various XR courses across their curriculum. Administered by the School of Information, with help from the organized initiative, the XR Graduate Certificate Program launched this fall. Learners can earn 12 credit hours taking multidisciplinary courses in five schools and colleges.

As the first XR Innovator-in-Residence, Nebeling has helped bring many ideas to the initiative and now launches a MOOC series titled Extended Reality for Everybody on the Coursera learning platform. True to its name the specialization will offer something for novices to more advanced users of the technology.

Part of the original charge of the initiative was to create innovative public/private partnerships to develop new XR-related educational technology. Collaborations have begun or are in discussion with Microsoft, Hewlett Packard, Disney, Epic Games, Lenovo, PTC and Unity.

In fact, many of the projects featuring HoloLens 2 technology have been done in conjunction with Microsoft, whose representatives have assisted several faculty members with development of their virtual materials.

Pandemic-born projects

From the launch, Nelson has consulted with more than 100 faculty about 30 potential XR projects. Several groups emerged or collaborated to advance the technology in various areas, but the pace of that consultation accelerated after March 17.

Early in the COVID-19 pandemic, clinical rotations were halted for medical students, which meant they could not participate in patient rounds—those bedside consultations with seasoned medical staff to discuss and learn about a patient’s condition.

This led Dr. Mark Cohen, a professor of surgery and pharmacology who runs the Innovation and Entrepreneurship program at the Medical School, and others who are at the forefront of technology in medicine, to think about a plan for virtual rounds. One person can be at the bedside attached to a MR headset while others nearby or even at home can be connected. Through the technology, those remotely can “examine the patient,” ask questions and give input as if they were in the room.

Cohen reached out to Nelson who connected him with Microsoft. This led to a HoloLens2 rounds project, believed to be the first broad rounding project in the U.S. In fact, the only other known project of its kind is at Imperial College of London, which recently collaborated with U-M on an International Mixed Reality Grand Rounds event.

“The goal on the telerounding side is to increase connectivity and limit exposures in the short term but also make this part of the workflow process that will be helpful for rounding in general,” Cohen said.

In a related project, Nelson also heard from the School of Nursing, which has a very modern Clinical Learning Center for teaching students many of the procedures they will perform as professionals, like inserting a catheter and starting an IV.

Brian Kaminski practicing a catheterization procedure using the HoloLens2. Image credit: U-M School of Nursing

The inventive center was established in 2012 and invested in further in 2015 during a school facility renovation.Yet, COVID-19 remote learning and the need to abide by proper social distancing guidelines when students returned to campus labs presented challenges for a program known for its technological, hands-on approach to training.

Again, HoloLens 2 technology provided an answer, and leaders in the school found the lessons they adapted to teach through mixed reality headsets allowed students to perform the procedures step-by-step, at their own pace, which solved the immediate problem but provided additional learning benefits.

“Originally it was, ‘How can we minimize person-to person interaction?’ but it’s also sparking a lot of creativity for solutions that have been going on before the pandemic, and that’s been great to see,” said student Charles Lennes, who helped test the nursing app.

Keeping up requires creativity

If there were hitches at all in the work of the initiative during the first year, it was keeping up with the demand for equipment and project development, Nelson said, which necessitated creative thinking to come up with additional resources.

For equipment purchased by the center, not individual programs, Nelson developed a loan system. For the first few months, select classes have been allowed to check out Oculus Quest VR headsets from the Shapiro Library. The controlled check-out system at first is important to establish proper cleaning protocols using a new CleanBox he purchased to kill the pathogens associated with COVID-19. If all goes well, the loan system will open to others at the start of winter semester, with priority still given to faculty for specific courses.

The extra projects stretched the staff and necessitated expedited hiring of additional student designers and developers to create the apps faculty need to make their projects work.

Nelson jumped at the chance for a recent Epic MegaGrant from Epic Games, the people who make the popular online game FortNite. The company made $100 million of its proceeds from the popular game available for projects that among other aims, would assist with “enhancing open-source capabilities for the 3D graphics community.” U-M received a $120,000 grant that Nelson said will help with hiring up to 30 student fellows to contribute to XR projects by building storyboards, writing code and creating 3D models.

More information:

• XR Initiative

Related stories:

• U-M online course specialization makes extended reality accessible to everyone
• Lessons in architecture: 3D without leaving the classroom (or living room)
• HoloLens2 for nursing: ‘Hands-on’ learning during pandemic & beyond
• Patient rounds: Ages-old care & learning concept gets technology upgrade, international collaboration