Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

Florian Karner; Christina Schwarz-Gsaxner; Antonio Pepe; Jianning Li; Philipp Fleck; Clemens Arth; Jürgen Wallner; Jan Egger

doi:10.1007/978-3-030-60946-7_7

Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

Florian Karner, Christina Schwarz-Gsaxner, Antonio Pepe, Jianning Li, Philipp Fleck, Clemens Arth, Jürgen Wallner, Jan Egger^*

^*Corresponding author for this work

Institute of Computer Graphics and Vision (7100)

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Abstract

Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

Original language	English
Title of host publication	Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures
Subtitle of host publication	10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings
Editors	Tanveer Syeda-Mahmood, Klaus Drechsler, Hayit Greenspan, Anant Madabhushi, Alexandros Karargyris, Cristina Oyarzun Laura, Stefan Wesarg, Marius George Linguraru, Raj Shekhar, Marius Erdt, Miguel Ángel González Ballester
Publisher	Springer
Pages	64-74
Number of pages	11
ISBN (Print)	9783030609450
DOIs	https://doi.org/10.1007/978-3-030-60946-7_7
Publication status	Published - 1 Jan 2020
Event	2020 Workshop on Clinical Image-Based Procedures: in Conjunction with MICCAI 2020 - Virtual, Virtuell, Peru Duration: 4 Oct 2020 → 8 Oct 2020 https://miccai-clip.org/

Publication series

Name	Lecture Notes in Computer Science
Volume	12445
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	2020 Workshop on Clinical Image-Based Procedures
Abbreviated title	CLIP 2020
Country/Territory	Peru
City	Virtuell
Period	4/10/20 → 8/10/20
Internet address	https://miccai-clip.org/

Access to Document

10.1007/978-3-030-60946-7_7

Cite this

Karner, F., Schwarz-Gsaxner, C., Pepe, A., Li, J., Fleck, P., Arth, C., Wallner, J., & Egger, J. (2020). Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. In T. Syeda-Mahmood, K. Drechsler, H. Greenspan, A. Madabhushi, A. Karargyris, C. Oyarzun Laura, S. Wesarg, M. G. Linguraru, R. Shekhar, M. Erdt, & M. Á. González Ballester (Eds.), Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures : 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings (pp. 64-74). (Lecture Notes in Computer Science ; Vol. 12445 ). Springer. https://doi.org/10.1007/978-3-030-60946-7_7

Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. / Karner, Florian; Schwarz-Gsaxner, Christina ; Pepe, Antonio et al.

Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures : 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. ed. / Tanveer Syeda-Mahmood; Klaus Drechsler; Hayit Greenspan; Anant Madabhushi; Alexandros Karargyris; Cristina Oyarzun Laura; Stefan Wesarg; Marius George Linguraru; Raj Shekhar; Marius Erdt; Miguel Ángel González Ballester. Springer, 2020. p. 64-74 (Lecture Notes in Computer Science ; Vol. 12445 ).

Research output: Chapter in Book/Report/Conference proceeding › Conference paper › peer-review

Karner, F, Schwarz-Gsaxner, C , Pepe, A, Li, J, Fleck, P , Arth, C, Wallner, J & Egger, J 2020, Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. in T Syeda-Mahmood, K Drechsler, H Greenspan, A Madabhushi, A Karargyris, C Oyarzun Laura, S Wesarg, MG Linguraru, R Shekhar, M Erdt & MÁ González Ballester (eds), Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures : 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. Lecture Notes in Computer Science , vol. 12445 , Springer, pp. 64-74, 2020 Workshop on Clinical Image-Based Procedures, Virtuell, Peru, 4/10/20. https://doi.org/10.1007/978-3-030-60946-7_7

Karner F, Schwarz-Gsaxner C , Pepe A, Li J, Fleck P , Arth C et al. Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. In Syeda-Mahmood T, Drechsler K, Greenspan H, Madabhushi A, Karargyris A, Oyarzun Laura C, Wesarg S, Linguraru MG, Shekhar R, Erdt M, González Ballester MÁ, editors, Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures : 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. Springer. 2020. p. 64-74. (Lecture Notes in Computer Science ). https://doi.org/10.1007/978-3-030-60946-7_7

Karner, Florian ; Schwarz-Gsaxner, Christina ; Pepe, Antonio et al. / Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality. Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures : 10th International Workshop, ML-CDS 2020, and 9th International Workshop, CLIP 2020, Held in Conjunction with MICCAI 2020, Proceedings. editor / Tanveer Syeda-Mahmood ; Klaus Drechsler ; Hayit Greenspan ; Anant Madabhushi ; Alexandros Karargyris ; Cristina Oyarzun Laura ; Stefan Wesarg ; Marius George Linguraru ; Raj Shekhar ; Marius Erdt ; Miguel Ángel González Ballester. Springer, 2020. pp. 64-74 (Lecture Notes in Computer Science ).

@inproceedings{02b2360c55374d0c9422bf58974db371,

title = "Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality",

abstract = "Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient{\textquoteright}s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.",

author = "Florian Karner and Christina Schwarz-Gsaxner and Antonio Pepe and Jianning Li and Philipp Fleck and Clemens Arth and J{\"u}rgen Wallner and Jan Egger",

year = "2020",

month = jan,

day = "1",

doi = "10.1007/978-3-030-60946-7_7",

language = "English",

isbn = "9783030609450",

series = "Lecture Notes in Computer Science ",

publisher = "Springer",

pages = "64--74",

editor = "Tanveer Syeda-Mahmood and Klaus Drechsler and Hayit Greenspan and Anant Madabhushi and Alexandros Karargyris and {Oyarzun Laura}, Cristina and Stefan Wesarg and Linguraru, {Marius George} and Raj Shekhar and Marius Erdt and {Gonz{\'a}lez Ballester}, {Miguel {\'A}ngel}",

booktitle = "Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures",

note = "2020 Workshop on Clinical Image-Based Procedures : in Conjunction with MICCAI 2020, CLIP 2020 ; Conference date: 04-10-2020 Through 08-10-2020",

url = "https://miccai-clip.org/",

}

TY - GEN

T1 - Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

AU - Karner, Florian

AU - Schwarz-Gsaxner, Christina

AU - Pepe, Antonio

AU - Li, Jianning

AU - Fleck, Philipp

AU - Arth, Clemens

AU - Wallner, Jürgen

AU - Egger, Jan

PY - 2020/1/1

Y1 - 2020/1/1

N2 - Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

AB - Augmented reality for medical applications allows physicians to obtain an inside view into the patient without surgery. In this context, we present an augmented reality application running on a standard smartphone or tablet computer, providing visualizations of medical image data, overlaid with the patient, in a video see-through fashion. Our system is based on the registration of medical imaging data to the patient using a single 2D photograph of the patient. From this image, a 3D model of the patient’s face is reconstructed using a convolutional neural network, to which a pre-operative CT scan is automatically registered. For efficient processing, this is performed on a server PC. Finally, anatomical and pathological information is sent back to the mobile device and can be displayed, accurately registered with the live patient, on the screen. Hence, our cost-effective, markerless approach needs only a smartphone and a server PC for image processing. We present a qualitative and quantitative evaluation using real patient photos and CT from the clinical routine in facial surgery, reporting overall processing times and registration errors.

UR - http://www.scopus.com/inward/record.url?scp=85092621117&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-60946-7_7

DO - 10.1007/978-3-030-60946-7_7

M3 - Conference paper

SN - 9783030609450

T3 - Lecture Notes in Computer Science

SP - 64

EP - 74

BT - Multimodal Learning for Clinical Decision Support and Clinical Image-Based Procedures

A2 - Syeda-Mahmood, Tanveer

A2 - Drechsler, Klaus

A2 - Greenspan, Hayit

A2 - Madabhushi, Anant

A2 - Karargyris, Alexandros

A2 - Oyarzun Laura, Cristina

A2 - Wesarg, Stefan

A2 - Linguraru, Marius George

A2 - Shekhar, Raj

A2 - Erdt, Marius

A2 - González Ballester, Miguel Ángel

PB - Springer

T2 - 2020 Workshop on Clinical Image-Based Procedures

Y2 - 4 October 2020 through 8 October 2020

ER -

Single-Shot Deep Volumetric Regression for Mobile Medical Augmented Reality

Abstract

Publication series

Conference

Access to Document

Other files and links

Fingerprint