Seeing is believing: Bringing virtual reality into the clinic

Seeing is believing. At the heart of this cliché lies an inherent skepticism we all share, a barrier to learning new things we cannot perceive through our familiar senses. This skepticism is amplified in clinical medicine, where stakes and emotions run high. In health care, diagnostic imaging helps to bridge the gap between knowledge and our senses by providing visual information indicating the presence or absence of disease. However, modern imaging techniques, including computed tomography (CT) and magnetic resonance imaging (MRI), can contain thousands of images, hindering our ability to concisely demonstrate visual findings to physicians, patients and families. 

Fortunately, medical imaging advancements are coursing parallel to advances in visualization technologies, including advanced rendering techniques and holographic and virtual reality displays. The missing link is converting thousands of two-dimensional (2D) images into a single composite three-dimensional (3D) image. 

This conversion is the goal of our work in the St. Jude Department of Diagnostic Imaging. I lead a team of radiologists, surgeons and advanced image processing specialists using state-of-the-art hardware and software to develop workflows that convert massive amounts of 2D image data into perceivable and understandable 3D virtual models. Once created, surgeons view these models for pre-operative planning, and families review them to better understand their child’s disease and its treatment.

The benefits of 3D modeling for pediatric cancer

At St. Jude, our patients are children, so their small size and relatively large tumors at the time of presentation make it challenging to segment (isolate) the tumor from structures it may be touching. Segmentation is the critical role of the advanced image-processing specialist. The lead image-processing specialist on our team, Chris Goode, has over a decade of experience segmenting tumors and creating virtual models and has overseen tremendous growth in visualization technologies. A combination of improved image quality stemming from advanced hardware and improved rendering software has resulted in many opportunities for displaying these 3D models.  

Complex computer-generated lighting models, originally created for the video game industry and animation studios, have made their way into medical visualization software. These photorealistic images convey dimensionality on a flat 2D display. While realistic, these 3D rendered images on 2D displays are limited by their fixed perspective; that is, the image does not change with the position of the observer — a phenomenon called parallax. Further, the lack of stereoscopic effect (illusion of depth) impedes depth perception.

Virtual reality for surgical planning

Recent advancements in visualization techniques now incorporate parallax and stereoscopy, offering a 3D experience beyond what is achievable through rendering on a 2D display. Head-mounted 3D virtual reality (VR) displays offer the greatest degree of immersion with a full-visual field, ideal to enhance focus and facilitate discussion. This is the modality used during pre-operative planning sessions before tumor resection surgery. 

This approach in preparing for a procedure benefits both the surgeon and the patient. “I feel more comfortable and confident going into an operation after reviewing the case virtually,” explains Andrew Davidoff, MD, St. Jude Department of Surgery chair.

Multiple people can use VR headsets and occupy the same virtual space, viewing and interacting with virtual models together. This, too, facilitates communication and is the technique used by Davidoff and other surgical faculty when meeting with parents to discuss the risks and benefits of a proposed procedure. Davidoff manipulates the models in virtual space while discussing it with a parent who shares his virtual view. With hand-tracking, Davidoff can virtually point, move and enlarge structures to convey key elements of the child’s tumor and plans for surgery. These sessions have quickly become the standard of care at St. Jude.

Novel technology leads the way

My team and I are excited to be at the forefront of applying this technology to pediatric oncology. However, 3D visualization is more than a mere tool; it is a field worthy of investigation. Our team has studied 3D visualization from a first principles perspective, answering questions such as how best to model the renal vein or how stereoscopic vision enhances depth perception among pediatric oncologic surgeons.

We aim to translate 3D virtual models to the operating room to assist in complex oncologic surgery. The burgeoning technology making this feasible is termed augmented reality (AR). Using AR headsets will one day allow the overlay of virtual models onto patient anatomy in the operating room, updating in real-time as surgery is carried out. The future of 3D visualization in health care is exciting, and St. Jude is leading the way.