Even if you’ve never been hospitalized you undoubtedly have seen media depictions of patient rounds—that decades-old tradition of a medical team moving about the hospital from bed to bed to discuss patient care.

The hands-on practice of collectively reviewing a patient’s symptoms and conferring over next steps for the care plan has changed over the years as professional organizations have found ways to improve it for the benefit of patients, students and health care teams.

Not surprisingly, close contact among patients, caregivers and those trying to learn to be health professionals got upended when the coronavirus pandemic hit, and health care institutions were forced to eliminate many procedures and hospitalizations, and limit unnecessary exposure to the very ill people being treated for COVID-19.

Rounds were not going to happen for a number of students around the patient’s bedside without a creative solution. A logical choice was to consider technology that would bring caregivers to the bedside virtually.

For Dr. Mark Cohen the concept was not a huge leap. He directs the medical school path of excellence in innovation and entrepreneurship, overseeing more than 100 medical student-developed innovation projects over the last five years.

“For me this is the next evolution in education and innovation, and can be applied not just for education but also for research and in clinical care delivery,” Cohen said. “This provides a novel way to teach at the bedside, lower exposure risk, and still bring the team to the bedside to enhance both the learner and the patient experience.”

Cohen knew mixed reality technology was a logical answer but a bigger question was what it would look like and how he could get it implemented quickly for clinical and educational use with Institutional Review Board and HIPAA approval, stringent IT requirements to protect patient privacy that typically take a while to work through, and with consideration for the legacy of rounds.

He talked with several of his colleagues on the medical campus and together they formed a new initiative for Medical and Surgical XR that included more than 10 clinical departments, medical school leaders and other faculty, residents and medical students who were interested in exploring use of the virtual and augmented reality technology in medicine.

He then reached out to the XR Initiative in the Center for Academic Innovation for help with how the technology might be employed in the rounds setting. CAI connected him with Microsoft, which had already been working with another university on a virtual rounds project using HoloLens2 technology.

In just a few months, the U-M team with help from CAI was testing out a virtual rounds project using HoloLens2 devices borrowed from the center, and joining with Imperial College of London on an historic joint international rounds event, during which the two institutions presented cases to one another and discussed the new means of communication and collaboration.

A run-through of the rounding event prompted this response from faculty member Clifford Craig, associate professor of orthopedic surgery:

“Briefly, watching this I had the same feeling as when I watched the first moon landing and walk—the future is now. The video quality was excellent, including the imaging studies. Ready for prime time. Appreciate all your work and effort on all this—happy to see it come to fruition.”

How it works

Through Microsoft’s HoloLens2 technology, one person conducts rounds wearing the Hololens2 headset connected via software to a Zoom or Microsoft TEAMS call, where the rest of the rounding team can be in multiple remote locations, even internationally.

This headset device, only recently on the market, allows the user to bring images, lab reports and documents into view using holographic windows that everyone in the rounding Zoom or TEAMS call can see as clearly, with the patient right in front of the headset wearer. The team is brought right to the bedside and can interact with the patient and see the headset wearer perform a physical exam or evaluate the patient.

Cohen put together several mobile XR carts consisting of a Hololens 2 headset, a laptop connected to the secure wireless network on the medical campus, as well as PPE and cleaning supplies. These carts can easily be moved around the hospital wards, ER, ICUs or even ORs to round on patients anywhere in the medical center, with a plan to expand this to U-M satellite facilities in the future.

The students and other participants can be in the back of the room, in another room nearby or at home, Cohen said. Just as in regular rounds, the physician and students can speak with the patient, ask questions and make observations.

“It eliminates the barrier of the medical student sitting in the back, who has to go through 5-6 people to ask a question,” said Taylor Kantor, a cardiothoracic surgery integrated resident who worked on the project with Cohen nearly from the beginning. “They have access right then and there because they are right in the fellow’s ear.”

Medical student Donovan Inniss, who helped Cohen with technical aspects of the app development, said the quality of both the audio and video was quite amazing, bringing those viewing remotely right into the room.

In one of their trial rounds, Inniss said the lung transplant patient was asked to blow bubbles to check his chest tube and the clarity of the lens allowed the bubbles to be seen clearly.

He and Kantor see endless possibilities in XR for learning.

“The virtual space is only just opening up to us,” said Inniss, a self-professed video game “nerd” whose fascination for that world fits well with his interest in using the 3D aspects of XR technologies in educational applications. “This pandemic has actually cemented the use of virtual tech.”

Kantor had worked with a HoloLens for another application at Mott Hospital and has interest in other projects for using it in education.

“In the next few years here at Michigan we’re going to show the variety of applications this virtual and augmented reality can have, he said.

Training tomorrow’s professionals

At the peak of the pandemic in spring, the American Association of Medical Colleges made the difficult decision to pull all US students out of clinical medicine rotations, said Dr. Rishindra “Rishi” Reddy, associate professor of thoracic surgery, who is in charge of training the medical students during their core surgery rotation—usually some 45-50 at a time. It wasn’t until June that our students were allowed back.

“First-year students still aren’t permitted in the clinics, and with the reduced clinical volume, we have been concerned for students to be able to find enough clinical experiences,” Reddy said. “Patients are staying away from the hospital right now—that’s been well documented—so there is a possibility that students may not have the same number of opportunities for clinical exposure that they would have had a year ago, so we’re looking at this as an opportunity to bridge the gap.

“There is recognition that if we don’t challenge our current generation of students, they won’t be as competent as tomorrow’s doctors. We have to include them now in the care of patients.”

This prompted Reddy to first come up with a rounding solution in May and June using a GoPro camera to maintain education by streaming one person’s experience, but he was glad to find out his colleague Cohen was looking at the higher quality, more versatile HoloLens2.

Also, under consideration is U-M’s commitment to interprofessional education. In recent years the university has been expanding its definition of care teams—acknowledging that coordinated patient care involves professionals from medicine, nursing, pharmacy, social work, dentistry, behavioral science and movement science—and teaching across disciplines accordingly. Although this kind of training had not fully reached rounding, it had been “on the radar,” Reddy said, to find a way to allow more of these students in on the experience.

Benefits for the patient

The mixed reality group assembled to advance XR at Michigan Medicine includes a number of faculty, residents and students who have worked with technology in their own areas or have big ideas for how to use it in medicine.

Michelle Benedict, a joint medical and MBA student, has a long track record in the tech industry including 10 years at Apple, and wants to bring more of this kind of technology to medicine. She admits that in many ways the profession has been slow to adopt.

“Of course, Mark Zuckerberg’s famous quote is, ‘Move fast and break things’ but when it’s human lives you need to be cautious,” Benedict said. “But there are opportunities for patient care to evolve, whether in the hospital or at home, or in how we educate people.”

Benedict said the grand rounds with Imperial College is an example of the “much richer experience for the learner” that is possible through technology, but also brings great benefit for the patient.

“When you (virtually) bring in other physicians and experts for cross collaboration with other institutions and countries or get weigh-in from a colleague who has more perspective—this represents a new model for education and cross collaboration,” she said.

Reddy said the collaboration in the international grand rounds offers a promise for care closer to home. Like many Michigan Medicine physicians, he travels across the state to perform surgeries in places like Grand Rapids but can’t stay for rounds. If the other hospital had the equipment he could participate virtually.

Kantor said that having fewer people around the bedside of transplant patients like those used in the trial runs is important. The risk of complication from infection is a concern so limited exposure is always a good idea, and the required medical team for these patients is large enough, he said. So, the transplant surgery team at Michigan Medicine is very interested in the project.

In fact, as other faculty learn about the HoloLens2 telerounding, overall the reaction has been extremely positive, with a number of faculty expressing interest in tailoring the application for their own clinical services.