Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

Florian Karner, Christina Schwarz-Gsaxner, Antonio Pepe, Jianning Li, Philipp Fleck, Clemens Arth, Jürgen Wallner, Jan Egger*

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review


Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

Original languageEnglish
Title of host publicationMultimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures
Subtitle of host publication 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings
EditorsTanveer Syeda-Mahmood, Klaus Drechsler, Hayit Greenspan, Anant Madabhushi, Alexandros Karargyris, Cristina Oyarzun Laura, Stefan Wesarg, Marius George Linguraru, Raj Shekhar, Marius Erdt, Miguel Ángel González Ballester
Number of pages11
ISBN (Print)9783030609450
Publication statusPublished - 1 Jan 2020
Event2020 Workshop on Clinical Image-Based Procedures: in Conjunction with MICCAI 2020 - Virtual, Virtuell, Peru
Duration: 4 Oct 20208 Oct 2020

Publication series

NameLecture Notes in Computer Science
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference2020 Workshop on Clinical Image-Based Procedures
Abbreviated titleCLIP 2020
Internet address


Dive into the research topics of 'Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality'. Together they form a unique fingerprint.

Cite this