Mixed Reality in Healthcare
Exploring use cases in medical education and clinical training
Next week, Apple is officially launching its new Vision Pro mixed reality (MR) headset. While it is not the first MR device to hit the market (e.g. Microsoft’s HoloLens has been out for years), for many people, Apple’s device might be their first foray into “extended-reality” (XR).
We will have to wait and see if the Vision Pro is Apple’s next great consumer hit. In the meantime, there are a few emerging use cases for MR in healthcare - specifically, various flavors of medical education and clinical training - that I think are worth paying attention to.
What is Mixed Reality?
MR is different than virtual reality (VR). The main difference is that with MR, devices project interactive visual elements into the real-world environment whereas VR provides full immersion into a fully virtual environment.
As an aside, I think most companies in this space will likely utilize the nomenclature of “XR” as it becomes a catch-all term. Also, X is a way cooler letter than M (see also: X-Games, Vin Diesel’s XXX, the short lived but electric XFL, X gon give it to ya).
The primary advantages of MR vs. VR are:
Compute - for both MR and VR, all of the compute required to generate the experience takes place within the device. While device compute will almost certainly increase with time, there is always going to be some limit to device compute power, which will present tradeoffs. With respect to rendering environments, MR has a strong advantage over VR in that with MR, you don’t need to allocate precious compute resources on rendering the background environment. With VR, you do need to spend compute resources on the background to create the fully virtual field of view. The impact of this is that objects rendered on a MR device can be way more detailed than they would be on a VR device with equal compute, which is important for some of the use cases explored below.
User Experience - a common experience for people putting on a VR headset for the first time is cybersickness - i.e. nausea and disorientation. MR typically induces less cybersickness for users. There are a couple of interesting reasons for this. The sheer act of overlaying content onto the real world reduces disorientation for the user because it is easier for the user to ground themselves in the physical world. MR typically incorporates a wider field of view, which also reduces user disorientation. Further, MR headsets typically utilize eye tracking and adjust the projected images accordingly, which reduces nausea-inducing visual conflict. Solving for less user nausea might unlock greater adoption, given how often cybersickness has been cited as a barrier to widespread usage of VR.
MR x Healthcare
A couple of years ago, I wrote that I thought that the most intuitive applications of “metaverse” technologies in healthcare are in the enablement of “immersive learning experiences”:
Metaverse use cases in healthcare would likely take advantage of the ability to:
Create deep immersion via richly rendered 3D spaces
Synchronously convene with both real peers and programmed entities in those spaces
Interact with co-inhabitants via programmable inputs/outputs
Simulate activities/events/situations that are rare or dangerous in real life
Transform one-to-one experiences to one-to-many experiences
The confluence of the capabilities above unlocks a core power of the metaverse which I would call immersive learning experiences. The prospective healthcare use cases … enhance our ability to acquire skills/knowledge from difficult to access sources (due to geographic or supply constraints) in a memorable, impactful way.
The areas where I am seeing this idea of MR-enabled immersive learning experiences play out most prominently so far are in:
Anatomical Education
Clinical Simulations
Surgical Planning and Operation
Behavioral Healthcare
I’ll cover the first three in greater detail below; I think the utilization of MR for behavioral healthcare is part of a broader discussion around mental health digital therapeutics and I have not studied this as closely. As always, feel free to let me know your thoughts or in a comment or an email!
Abra Cadavera 🪄
Medical education already relies on multiple forms of immersive learning experiences.
For example, in post-secondary and medical school education, students take anatomy courses. A common standard teaching instrument for anatomy education in medical schools is the cadaver lab.
Work in the cadaver lab involves dissecting and labeling body parts on a body donated to science. This has actually been a standard of education for centuries. And yet, cadaveric dissection has multiple drawbacks:
It’s hard to see specific organs and systems. Embalmed cadavers have compressed organ presentation and the grey-ish color of a cadaver obscures distinct body parts that would typically be more visually distinct in a living person. Further, it’s impossible to see certain systems, like the circulatory or nervous systems, in a cadaver. The presentation of any particular body may be so distinct from the anatomical standard that it can convolute the student’s understanding of how the body systems could look in another person.
It’s expensive. Cadavers can cost $2-3k per “unit” and require significant storage infrastructure. Particularly in urban settings, it can be very expensive to build and maintain adequate storage capacity for cadavers. Some medical schools report difficulty in sourcing cadavers at all due to a variety of supply chain issues.
It's time consuming. Cadaveric dissection takes a long time which limits the amount of content medical schools can pack into curricula. Further, students have limited time to spend examining cadavers (you can’t just bring the cadaver home with you lol).
Most of the proponents of cadaveric dissection that I have spoken to defend the practice primarily based on the idea that it is a rite of passage. The idea is that working with a body that somebody donated to science can lend a sense of gravity to the work students are doing at medical school. I think that’s a fair claim, but it suggests that the true value of traditional cadaveric dissection is limited. Thus, it’s no surprise that academic institutions are increasingly looking to adopt digital anatomy training tools.
For example, one digital tool to teach anatomy that has achieved decent traction in the last several years is the Anatomage table. Anatomage tables are basically ginormous iPads which enable students to view a full body scale digital rendering.
The advantages of the Anatomage table are primarily related to the digital nature of the form factor. These devices enable students to view life-size renderings based on a deep library of anatomical assets. This is a step up from traditional cadavers.
However, one of the major pitfalls of the 2D form factor is the inability to sufficiently depict depth. This is where 3D visualizations are a major level up and where mixed reality offers a leap forward in capabilities. Also, like a traditional cadaver, you can’t take this giant iPad home with you.
With MR platforms like HoloAnatomy by AlensiaXR1, students can literally poke into the body which enables a deeper understanding of how body systems work together. The technology is mobile and enables a far more convenient experience for students seeking to master anatomical content.
Early adopters of MR technology in anatomy education indicate that students absorb the content much faster, which enables institutions to enhance the curriculum by incorporating additional programming.
Just like the simulations
Another widespread component of medical school curricula are patient simulations with “standardized patients”. This is where medical schools bring in actors to portray patients with particular conditions (the famous Seinfeld scene comes to mind).
I’m rehashing this point from my last article because I think it bears repeating that one of the most exciting potential use cases of MR is in advanced clinical simulations. For example, the University of Illinois uses GigXR in their medical school education program to train students in “real-world” scenarios.
As I also discussed in my last post, these simulations are only going to get more advanced as advancements in LLM’s enable life-like conversations with simulated entities. For example, check out the capabilities from Convai (the group that partnered with NVIDIA in that demo I linked in the last post).
The possibilities blow my mind. Dropping these emerging AI capabilities into patient simulations could significantly increase the value prop of MR simulations, further enhancing their value as tools for clinician training.
MR in the OR
Another setting showing promise for mixed reality technology is surgical operating rooms.
Mixed reality enables a patient’s medical imaging to be visualized in 3-D to assist in surgical planning. Companies like Augmedics, OssoVR, ImmersiveTouch, Medivis and MediView offer platforms that could transform the standard of surgical care, improving pre-operation planning/training and remote collaboration to reduce the risk of procedural errors. In the future, these technologies might assist providers during the actual procedure navigation.
These kind of tools are going to become increasingly important as surgeries, medical devices, and pharmaceutical products become increasingly complex. The WSJ published a great piece highlighting this trend last year:
A major obstacle looms for the drugs of the future. Not enough doctors know how to administer them.
For just one rare neurodevelopmental disorder, known as Angelman syndrome, clinical trials are testing four cutting-edge therapies. Twenty more research programs are under way and could yield treatments ready to move into human testing in the next several years, according to Allyson Berent-Weisse, chief scientific officer of the Foundation for Angelman Syndrome Therapeutics, or FAST, and the mother of a child with the disorder.
These promising therapies include antisense oligonucleotides (or ASOs) which affect how proteins inside cells are made; Crispr gene editors that can cut, rewrite, or replace faulty genes; and other novel approaches.
But there’s a hitch. Some of these drugs, as well as therapies in development for other conditions, are administered through lumbar punctures, which aren’t a routine part of doctors’ daily practices. They may involve infusions of a gene packaged inside the shell of a virus, which can lead to potential immune complications and can require close monitoring of patients. Some drugs need to be stored in special conditions, which can take complicated planning and coordination that few doctors have experienced.
“The bottleneck used to be the science and finding drugs,” says Dr. Berent-Weisse. “The new bottleneck is having enough people trained to give them.”
Source: WSJ
Groups like Gilead and Roche have developed training centers to teach physicians and surgeons how to execute some of these complex procedures, training that should certainly yield greater adoption of their products and better care for patients. Integrating MR into these efforts could help accelerate learning curves on these breakthrough therapies.
the medicine of toMoRrow
An overarching question I have about adoption of MR in medical education is around urgency. What is the “why now” that drives increased usage of this nascent technology by customers? After all, VR has been around for a while and certainly has yet to see widespread uptake.
My non-scientific prediction is that many people will head to the Apple store at the mall at some point this year and try on a headset; if they recognize that MR is a big step forward in terms of usability, it might get them to start thinking about how they can incorporate this form factor into their professional work.
But considerations around what the future of medicine will look like might be a sufficient catalyst as well. Consider that a pre-med student entering university this fall is actually going to be practicing medicine as a physician like 10-15 years from now. Thus, educators have to think about investments on decades-long time horizons. And if the prevailing view becomes that MR is going to be a key technology in the delivery of care in future decades, then there will be a greater urgency to get this technology into the hands of aspiring clinicians today.
Moreover, the clinical talent shortages that plague healthcare (which I cover a lot here) will likely also drive greater adoption of MR technology for medical education. This is because as the system contends with more acute physician shortages, care delivery will naturally be forced to decentralized. As care delivery decentralizes, medical education and training will need to decentralize too.
I speculated a bit in my piece on the rapid growth of NPs and PAs within the clinical labor force that “on the job” clinical training is going to become more and more important, particularly as physician shortages inevitably increase the clinical scope of groups like NPs and PAs. Leveraging technology like clinical simulations can help augment the experience set of practicing healthcare workers of all types in support of this expanding scope.
Just this week, the Bloomberg Philanthropies announced the launch of a $250mm investment into expanding medical vocational training for students as early as high school as part of an effort to combat talent shortages.
As the healthcare industry seeks to attract prospective talent, technology will play an important role in engaging, training and uplifting that talent.
The search for a catalyst, of course, is entirely moot if this new slate of MR devices turns out to be so good that they replace computers as the primary way we engage with the digital world (and maybe even the physical world). In that case, you are all cordially invited to my birthday party this summer in the metaverse as we take yet another leap towards becoming those hover chair people from Wall-E.
Disclaimer:
This content is being made available for educational purposes only and should not be used for any other purpose. The information contained herein does not constitute and should not be construed as investment advice, an offering of advisory services, or an offer to sell or solicitation to buy any securities or related financial instruments in any jurisdiction. Certain information contained herein concerning economic trends and performance is based on or derived from information provided by independent third-party sources. The author believes that the sources from which such information has been obtained are reliable; however, the author cannot guarantee the accuracy of such information and has not independently verified the accuracy or completeness of such information or the assumptions on which such information is based.
Denotes a Sopris Portfolio Company