BP-MVSNet: Belief-Propagation-Layers for Multi-View-Stereo

Christian Sormann, Patrick Knöbelreiter, Andreas Kuhn, Mattia Rossi, Thomas Pock, Friedrich Fraundorfer

Research output: Chapter in Book/Report/Conference proceedingConference paperpeer-review

Abstract

In this work, we propose BP-MVSNet, a convolutional neural network (CNN)-based Multi-View-Stereo (MVS) method that uses a differentiable Conditional Random Field (CRF) layer for regularization. To this end, we propose to extend the BP layer [16] and add what is necessary to successfully use it in the MVS setting. We therefore show how we can calculate a normalization based on the expected 3D error, which we can then use to normalize the label jumps in the CRF. This is required to make the BP layer invariant to different scales in the MVS setting. In order to also enable fractional label jumps, we propose a differentiable interpolation step, which we embed into the computation of the pairwise term. These extensions allow us to integrate the BP layer into a multi-scale MVS network, where we continuously improve a rough initial estimate until we get high quality depth maps as a result. We evaluate the proposed BP-MVSNet in an ablation study and conduct extensive experiments on the DTU, Tanks and Temples and ETH3D data sets. The experiments show that we can significantly outperform the baseline and achieve state-of-the-art results.

Original languageEnglish
Title of host publicationProceedings - 2020 International Conference on 3D Vision, 3DV 2020
PublisherIEEEXplore
Pages394-403
Number of pages10
ISBN (Electronic)9781728181288
DOIs
Publication statusPublished - 25 Nov 2020
Event8th International Conference on 3D Vision: 3DV 2020 - Virtual, Fukuoka, Japan
Duration: 25 Nov 202028 Nov 2020
http://3dv2020.dgcv.nii.ac.jp

Publication series

NameProceedings - 2020 International Conference on 3D Vision, 3DV 2020

Conference

Conference8th International Conference on 3D Vision
Abbreviated title3DV
CountryJapan
CityVirtual, Fukuoka
Period25/11/2028/11/20
Internet address

Keywords

  • CNN
  • Conditional Random Field
  • deep learning
  • depth estimation
  • Multi View Stereo

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition

Fingerprint

Dive into the research topics of 'BP-MVSNet: Belief-Propagation-Layers for Multi-View-Stereo'. Together they form a unique fingerprint.

Cite this