IntraOpVSP

Intra operative virtual surgical planning using AR and Computer Vision.

A set of mixed-reality apps that allow surgeons to anatomically align and interact with CT scan data through Microsoft Hololens. The technology has been used in over 50 human surgeries. The technology has been licensed to a startup for commercialization and has been approved by FDA.

My Contributions

As part of my MS research, I implemented the technology from scratch.

  • Developed mixed reality software using Unity, Mixed Reality Toolkit, and Microsoft Hololens-2.
  • Used OpenCV, DLib, and PyTorch for 6-DoF pose estimation of human head and surgical tools.
  • Annotated 2D, 3D data for machine learning.

Skills/Tools: PyTorch, OpenCV, Unity3D, Blender, MeshLab, DLib, C++, C#, Python.

IntraOpVSP live demonstration.

The Problem

The mission is to augment every surgery. Compared to a pilot flying a plane or even a regular google-map user on his way to work/home, surgeons today have their instruments clustered behind them hanging in the wall. The Google Maps user or the pilot gets constant real-time updates regarding where they are, what to do next, and other vital data that helps them make split-second decisions. They don’t have to plan the trip for days or memorize every turn and detail of every landmark along the way. They just do it. On the other hand, surgeons today have to do rigorous surgical planning, memorize the specifics of each unique case, and all the necessary steps to ensure the safest possible surgery. Then they engage in complex procedures for several hours, with no targeting or homing devices or head-mounted displays to assist them. They have to feel their way to their objective and hope everything goes as they planned. In my research, I made the “google-maps” for surgery.

Examples:

Following are IntraOpVSP usage examples:

CT scan of the head can be aligned to the patient's head using computer vision. Purple shows the CT scan of the head before surgery. Red shows the planned cuts/segments and their intended position after the surgery. Yellow shows the base structure of the skull which will remain unmodified.
Face keypoints on the CT scan are annotated and loaded in IntraOpVSP before surgery. Using Dlib, ICP and on-board cameras in Hololens-2 the 6-DoF pose of the head is estimated. Then using Unity3D, the CT scan is aligned with the subject's head during surgery.
Using hand tracking and voice commands, the user can interact with the different parts of the aligned CT scans.
The surgeon can use their hand to slice the CT scan data and create a cross-sectional view. This view helps surgeons get an idea of how thick a section is as they often need to drill or cut.
This is a demonstration of liver surgery. Computer vision can't be used for pose estimation as the body of the subject is covered by a cloth. Thus, I implemented a semi-automatic alignment process by allowing the surgeon to place known landmarks on the subject's body during surgery. I then align the CT scan data to the subject's body using 3D rigid body pose transform estimation.
Example of liver tumor removal.
Example of an ear reconstruction surgery using IntraOpVSP.