Dr. Benjamin Warf, a renowned neurosurgeon from Boston Children’s Hospital, has been virtually present in Brazil, aiding and mentoring residents as they perform delicate surgery on a model of a baby’s brain. This has been made possible through a digital avatar of Dr. Warf, developed by the medical simulator and augmented reality (AR) company, EDUCSIM, in conjunction with MIT.nano, the Massachusetts Institute of Technology’s nanotechnology center.
Virtual reality (VR) goggles allowed Matheus Vasconcelos, a neurosurgery resident at Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil, to watch Dr. Warf’s avatar demonstrate surgical techniques before replicating them. This innovative method of imparting neurosurgery skills bridged the gap between continents, transferring knowledge and expertise seamlessly.
The avatar project was launched as part of EDUCSIM’s endeavor to provide practical training for complex surgeries. The company, part of the 2023 cohort of START.nano, MIT.nano’s deep-tech accelerator, offers early-stage startups discounted access to MIT.nano’s laboratories. Dr. Warf’s digital model was spearheaded by Dr. Giselle Coelho, EDUCSIM’s scientific director, and a pediatric neurosurgeon.
In order to create the avatar, Coelho collaborated with Talis Reks, an AR/VR/gaming/big data IT technologist at the MIT.nano Immersion Lab, using high-fidelity and high-resolution motion-capture technology, volumetric video capture, and a range of other VR/AR technologies. Warf visited MIT.nano numerous times to be digitally “captured,” even performing an operation on a physical model while wearing sensor-embedded gloves and clothing.
A significant challenge the team encountered was developing what Coelho refers to as “holoportation”—the real-time transmission of a 3D, volumetric video capture of Warf over the internet, allowing the avatar to function in transcontinental medical training. The avatar operates in both synchronous and asynchronous modes, enabling observation, questioning, and real-time guidance.
In the synchronous mode, Warf remotely operates his avatar in real time. In the asynchronous mode, residents can observe the avatar’s demonstrations and ask it questions, receiving answers provided by AI algorithms that source from a wide array of pre-existing questions and answers offered by Warf.
The project has transformative implications, particularly for surgeons serving remote and underserved areas, such as the Amazon region of Brazil. Coelho emphasizes the potential to extend the same level of education and training to remote regions without the limitations posed by the cost and time of travel.
