Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild

Alexander Grabner*, Yaming Wang, Peizhao Zhang, Peihong Guo, Tong Xiao, Peter Vajda, Peter M. Roth, Vincent Lepetit

*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceedingConference paper


We present a novel 3D pose refinement approach based ondifferentiable rendering for objects of arbitrary categories in the wild. Incontrast to previous methods, we make two main contributions: First,instead of comparing real-world images and synthetic renderings in theRGB or mask space, we compare them in a feature space optimized for3D pose refinement. Second, we introduce a novel differentiable rendererthat learns to approximate the rasterization backward pass from data in-stead of relying on a hand-crafted algorithm. For this purpose, we predictdeep cross-domain correspondences between RGB images and 3D modelrenderings in the form of what we call geometric correspondence fields.These correspondence fields serve as pixel-level gradients which are ana-lytically propagated backward through the rendering pipeline to performa gradient-based optimization directly on the 3D pose. In this way, weprecisely align 3D models to objects in RGB images which results in sig-nificantly improved 3D pose estimates. We evaluate our approach on thechallenging Pix3D dataset and achieve up to 55% relative improvementcompared to state-of-the-art refinement methods in multiple metrics.
Original languageEnglish
Title of host publicationComputer Vision – ECCV 2020
Place of PublicationCham
Number of pages14
ISBN (Print)978-3-030-58516-7
Publication statusPublished - 2020
Event16th European Conference on Computer Vision: ECCV 2020 - Virtual, Glasgow, United Kingdom
Duration: 23 Aug 202028 Aug 2020

Publication series

NameLecture Notes in Computer Science


Conference16th European Conference on Computer Vision
Abbreviated titleECCV 2020
CountryUnited Kingdom
CityVirtual, Glasgow


Dive into the research topics of 'Geometric Correspondence Fields: Learned Differentiable Rendering for 3D Pose Refinement in the Wild'. Together they form a unique fingerprint.

Cite this