external website 1 and website 2

Project is in progress and undergoing cadaver studies

The ARTEMIS project is an exploration into using Augmented Reality as a tool for remote surgical environments. Funded by the Navy, it addresses the issue of bringing medical expertise to remote environments. Uses AR and VR to bring a hologram of an expert surgeon based in high quality medical facility to a remote novice.

I was involved in the design, development, and management of this project. I am not responsibile for its entirety, but I have developed most of this system and have heavily contributed to each part. Project is done through the Human-Centered and Ubiquitous Computing Lab run by Dr. Nadir Weibel. This project is currently going through cadaver studies to validate and compare it against the current standard of practice.

Expert Environment (VR)

Technology

HTC Vive Pro

VR Headset with Video See-Through

Optitrack

Mocap tracking system using IR tracking of retroreflective markers

Senso Gloves

IMU gloves paired with Optitrack to determine hand location and movement

Custom Pen

A wireless mouse paired with Optitrack to use clicking to draw

Gesturing and Annotating

The Expert sees a live point cloud hologram of the patient which they can draw on top of. Any gesturing

Interfaces

Video Streaming

The Novice's environment has several cameras which stream to the Expert and are expanded when selected (see Novice information below)

Surgical Video Reference

For complex procedures, reference videos are provided which can be selected and shown to the Novice - presenting maneuvers, and where to cut.

Control Panel

Because the novice is conducting a surgery, the Expert is responsible for all control: setting visibility of various elements in the Novice's environment

Novice Environment (AR)

Technology

Microsoft HoloLens

Mixed Reality (AR) headset

Optitrack

Mocap tracking system using IR tracking of retroreflective markers

Kinect Azure

Used for gathering point cloud of patient and tracking annotations on the body

Intel RealSense

Used for various camera views of the Novice's space

Novice Perspective

The Novice sees an avatar of the Expert, the Expert's pen, and the Expert's hands. The Expert can use the pen both as gestural device - demonstrating maneuvering, and as annotational device - drawing on the patient. Annotations are visible to the Novice and appear directly on the patient's body.

The Expert can walk around the table to take on the same perspective as the novice - allowing for first person demonstration and teaching; or the Expert can stand opposite the novice for traditional third person instruction.

Prototyping ARTEMIS

Novice (right) working on patient with holographic assistance from Expert (left)

Expert (left) seeing holograms of patient and Novice (right)

Exploring the potential of using a CAVE in conjunction with AR/VR

Future goal is to make the system capable of functioning without additional cameras - becoming a standalone HoloLens application

Using PrototipAR to explore problem space.

Artemis Display _ no bleed.pdf

Google Sites

Report abuse