Learning Lightprobes for Mixed Reality Illumination

David Mandl; Kwang Moo Yi; Peter Mohr-Ziak; Peter M. Roth; Pascal Fua; Vincent Lepetit; Dieter Schmalstieg; Denis Kalkofen

doi:10.1109/ISMAR.2017.25

Learning Lightprobes for Mixed Reality Illumination

David Mandl^*, Kwang Moo Yi, Peter Mohr-Ziak, Peter M. Roth, Pascal Fua, Vincent Lepetit, Dieter Schmalstieg, Denis Kalkofen

^*Korrespondierende/r Autor/-in für diese Arbeit

Institut für Maschinelles Sehen und Darstellen (7100)

Publikation: Beitrag in Buch/Bericht/Konferenzband › Beitrag in einem Konferenzband › Begutachtung

Abstract

This paper presents the first photometric registration pipeline for
Mixed Reality based on high quality illumination estimation using
convolutional neural networks (CNNs). For easy adaptation and deployment
of the system, we train the CNNs using purely synthetic
images and apply them to real image data. To keep the pipeline accurate
and efficient, we propose to fuse the light estimation results
from multiple CNN instances and show an approach for caching
estimates over time. For optimal performance, we furthermore explore
multiple strategies for the CNN training. Experimental results
show that the proposed method yields highly accurate estimates for
photo-realistic augmentations.

Originalsprache	englisch
Titel	Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017
Herausgeber (Verlag)	Institute of Electrical and Electronics Engineers
Seiten	82 - 89
Seitenumfang	8
ISBN (elektronisch)	978-153862943-7
DOIs	https://doi.org/10.1109/ISMAR.2017.25
Publikationsstatus	Veröffentlicht - 9 Okt. 2017
Veranstaltung	2017 IEEE International Symposium on Mixed and Augmented Reality: ISMAR 2017 - La Cité, Nantes, Frankreich Dauer: 9 Okt. 2017 → 13 Okt. 2017 https://ismar2017.sciencesconf.org/

Konferenz

Konferenz	2017 IEEE International Symposium on Mixed and Augmented Reality
Kurztitel	ISMAR 2017
Land/Gebiet	Frankreich
Ort	Nantes
Zeitraum	9/10/17 → 13/10/17
Internetadresse	https://ismar2017.sciencesconf.org/

Zugriff auf Dokument

10.1109/ISMAR.2017.25

Dieses zitieren

Mandl, D., Yi, K. M., Mohr-Ziak, P., Roth, P. M., Fua, P., Lepetit, V., Schmalstieg, D., & Kalkofen, D. (2017). Learning Lightprobes for Mixed Reality Illumination. in Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017 (S. 82 - 89). Artikel 8115407 Institute of Electrical and Electronics Engineers. https://doi.org/10.1109/ISMAR.2017.25

Learning Lightprobes for Mixed Reality Illumination. / Mandl, David; Yi, Kwang Moo; Mohr-Ziak, Peter et al.
Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017. Institute of Electrical and Electronics Engineers, 2017. S. 82 - 89 8115407.

Publikation: Beitrag in Buch/Bericht/Konferenzband › Beitrag in einem Konferenzband › Begutachtung

Mandl, D, Yi, KM, Mohr-Ziak, P , Roth, PM, Fua, P, Lepetit, V, Schmalstieg, D & Kalkofen, D 2017, Learning Lightprobes for Mixed Reality Illumination. in Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017., 8115407, Institute of Electrical and Electronics Engineers, S. 82 - 89, 2017 IEEE International Symposium on Mixed and Augmented Reality, Nantes, Frankreich, 9/10/17. https://doi.org/10.1109/ISMAR.2017.25

@inproceedings{43fc66b5266d429c8cbe8c24f5401140,

title = "Learning Lightprobes for Mixed Reality Illumination",

abstract = "This paper presents the first photometric registration pipeline forMixed Reality based on high quality illumination estimation usingconvolutional neural networks (CNNs). For easy adaptation and deploymentof the system, we train the CNNs using purely syntheticimages and apply them to real image data. To keep the pipeline accurateand efficient, we propose to fuse the light estimation resultsfrom multiple CNN instances and show an approach for cachingestimates over time. For optimal performance, we furthermore exploremultiple strategies for the CNN training. Experimental resultsshow that the proposed method yields highly accurate estimates forphoto-realistic augmentations.",

author = "David Mandl and Yi, {Kwang Moo} and Peter Mohr-Ziak and Roth, {Peter M.} and Pascal Fua and Vincent Lepetit and Dieter Schmalstieg and Denis Kalkofen",

year = "2017",

month = oct,

day = "9",

doi = "10.1109/ISMAR.2017.25",

language = "English",

pages = "82 -- 89",

booktitle = "Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017",

publisher = "Institute of Electrical and Electronics Engineers",

address = "United States",

note = "2017 IEEE International Symposium on Mixed and Augmented Reality : ISMAR 2017, ISMAR 2017 ; Conference date: 09-10-2017 Through 13-10-2017",

url = "https://ismar2017.sciencesconf.org/",

}

TY - GEN

T1 - Learning Lightprobes for Mixed Reality Illumination

AU - Mandl, David

AU - Yi, Kwang Moo

AU - Mohr-Ziak, Peter

AU - Roth, Peter M.

AU - Fua, Pascal

AU - Lepetit, Vincent

AU - Schmalstieg, Dieter

AU - Kalkofen, Denis

PY - 2017/10/9

Y1 - 2017/10/9

N2 - This paper presents the first photometric registration pipeline forMixed Reality based on high quality illumination estimation usingconvolutional neural networks (CNNs). For easy adaptation and deploymentof the system, we train the CNNs using purely syntheticimages and apply them to real image data. To keep the pipeline accurateand efficient, we propose to fuse the light estimation resultsfrom multiple CNN instances and show an approach for cachingestimates over time. For optimal performance, we furthermore exploremultiple strategies for the CNN training. Experimental resultsshow that the proposed method yields highly accurate estimates forphoto-realistic augmentations.

AB - This paper presents the first photometric registration pipeline forMixed Reality based on high quality illumination estimation usingconvolutional neural networks (CNNs). For easy adaptation and deploymentof the system, we train the CNNs using purely syntheticimages and apply them to real image data. To keep the pipeline accurateand efficient, we propose to fuse the light estimation resultsfrom multiple CNN instances and show an approach for cachingestimates over time. For optimal performance, we furthermore exploremultiple strategies for the CNN training. Experimental resultsshow that the proposed method yields highly accurate estimates forphoto-realistic augmentations.

U2 - 10.1109/ISMAR.2017.25

DO - 10.1109/ISMAR.2017.25

M3 - Conference paper

SP - 82

EP - 89

BT - Proceedings of the 2017 IEEE International Symposium on Mixed and Augmented Reality, ISMAR 2017

PB - Institute of Electrical and Electronics Engineers

T2 - 2017 IEEE International Symposium on Mixed and Augmented Reality

Y2 - 9 October 2017 through 13 October 2017

ER -

Learning Lightprobes for Mixed Reality Illumination

Abstract

Konferenz

Zugriff auf Dokument

Fingerprint

Dieses zitieren