The ARTEMIS project is an exploration into using Augmented Reality as a tool for remote surgical environments. Funded by the Navy, it addresses the issue of bringing medical expertise to remote environments. Uses AR and VR to bring a hologram of an expert surgeon based in high quality medical facility to a remote novice.
I was involved in the design, development, and management of this project. I am not responsibile for its entirety, but I have developed most of this system and have heavily contributed to each part. Project is done through the Human-Centered and Ubiquitous Computing Lab run by Dr. Nadir Weibel. This project is currently going through cadaver studies to validate and compare it against the current standard of practice.
Expert Environment (VR)
Technology
HTC Vive Pro
VR Headset with Video See-Through
Optitrack
Mocap tracking system using IR tracking of retroreflective markers
Senso Gloves
IMU gloves paired with Optitrack to determine hand location and movement
Custom Pen
A wireless mouse paired with Optitrack to use clicking to draw
Gesturing and Annotating
The Expert sees a live point cloud hologram of the patient which they can draw on top of. Any gesturing
Interfaces
Video Streaming
The Novice's environment has several cameras which stream to the Expert and are expanded when selected (see Novice information below)
Surgical Video Reference
For complex procedures, reference videos are provided which can be selected and shown to the Novice - presenting maneuvers, and where to cut.
Control Panel
Because the novice is conducting a surgery, the Expert is responsible for all control: setting visibility of various elements in the Novice's environment
Novice Environment (AR)
Technology
Microsoft HoloLens
Mixed Reality (AR) headset
Optitrack
Mocap tracking system using IR tracking of retroreflective markers
Kinect Azure
Used for gathering point cloud of patient and tracking annotations on the body
Intel RealSense
Used for various camera views of the Novice's space
Novice Perspective
The Novice sees an avatar of the Expert, the Expert's pen, and the Expert's hands. The Expert can use the pen both as gestural device - demonstrating maneuvering, and as annotational device - drawing on the patient. Annotations are visible to the Novice and appear directly on the patient's body.
The Expert can walk around the table to take on the same perspective as the novice - allowing for first person demonstration and teaching; or the Expert can stand opposite the novice for traditional third person instruction.
Prototyping ARTEMIS
Novice (right) working on patient with holographic assistance from Expert (left)
Expert (left) seeing holograms of patient and Novice (right)
Exploring the potential of using a CAVE in conjunction with AR/VR
Future goal is to make the system capable of functioning without additional cameras - becoming a standalone HoloLens application
Using PrototipAR to explore problem space.
Artemis Display _ no bleed.pdf