Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

Alexander Grabner, Yaming Wang, Peizhao Zhang, Peihong Guo, Tong Xiao, Peter Vajda, Peter M. Roth, Vincent Lepetit

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

We present a novel 3D pose refinement approach based ondifferentiable rendering for objects of arbitrary categories in the wild. Incontrast to previous methods, we make two main contributions: First,instead of comparing real-world images and synthetic renderings in theRGB or mask space, we compare them in a feature space optimized for3D pose refinement. Second, we introduce a novel differentiable rendererthat learns to approximate the rasterization backward pass from data in-stead of relying on a hand-crafted algorithm. For this purpose, we predictdeep cross-domain correspondences between RGB images and 3D modelrenderings in the form of what we call geometric correspondence fields.These correspondence fields serve as pixel-level gradients which are ana-lytically propagated backward through the rendering pipeline to performa gradient-based optimization directly on the 3D pose. In this way, weprecisely align 3D models to objects in RGB images which results in sig-nificantly improved 3D pose estimates. We evaluate our approach on thechallenging Pix3D dataset and achieve up to 55% relative improvementcompared to state-of-the-art refinement methods in multiple metrics.
Original languageEnglish
Title of host publicationEuropean Conference on Computer Vision (ECCV)
Pages1-14
Number of pages14
Publication statusAccepted/In press - 2020
Event16th European Conference on Computer Vision - Virtuell
Duration: 23 Aug 202028 Aug 2020

Conference

Conference16th European Conference on Computer Vision
Abbreviated titleECCV 2020
CityVirtuell
Period23/08/2028/08/20

Fingerprint Dive into the research topics of 'Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild'. Together they form a unique fingerprint.

  • Cite this

    Grabner, A., Wang, Y., Zhang, P., Guo, P., Xiao, T., Vajda, P., ... Lepetit, V. (Accepted/In press). Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild. In European Conference on Computer Vision (ECCV) (pp. 1-14)