Deep Feedback Inverse Problem Solver

Wei-Chiu Ma1,2 Shenlong Wang1,3 Jiayuan Gu1,3 Sivabalan Manivasagam1,3
Antonio Torralba2 Raquel Urtasun1,3
1Uber Advanced Technologies Group 2Massachusetts Institute of Technology
3University of Toronto
European Conference on Computer Vision (ECCV), 2020


We present an efficient, effective, and generic approach towards solving inverse problems. The key idea is to leverage the feedback signal provided by the forward process and learn an iterative update model. Specifically, at each iteration, the neural network takes the feedback as input and outputs an update on current estimation. Our approach does not have any restrictions on the forward process; it does not require any prior knowledge either. Through the feedback information, our model not only can produce accurate estimations that are coherent to the input observation but also is capable of recovering from early incorrect predictions. We verify the performance of our approach over a wide range of inverse problems, including 6-DOF pose estimation, illumination estimation, as well as inverse kinematics. Comparing to traditional optimization-based methods, we can achieve comparable or better performance while being two to three orders of magnitude faster. Compared to deep learning-based approaches, our model consistently improves the performance on all metrics.

Prior work

(a) Structured optimization approaches require hand-crafted energy/objective functions and are sensitive to initializations which makes them easy to get stuck in local optima. (b) Direct learning based methods do not utilize the available forward process as feedback to guarantee the quality of the solution. Without this feedback, the models cannot rectify the estimates effectively as shown above.

Overview of our approach

Our model iteratively updates the estimation based on the feedback signal from the forward process. We adopt a closed-loop scheme to ensure the consistency between the estimation and the observation. We neither require an objective at test time, nor have any restrictions on the forward process

Application 1: 6 DoF Pose Estimation

Observation / Ours / Neural Mesh Renderer / Soft Rasterizer

Application 2: Illumination Estimation

Observation / Ours / Neural Mesh Renderer

Application 3: Inverse Kinematics

Our approach is able to accurately predict the joint rotations within a few steps. It can also correct wrong estimations through the feedback from the forward model (see the feet/toes in the right column). Bottom right shows an example where our model fails.

Short Video (1:30)

Long Video (10:00)


Paper (link)

Supp. material (link)

Code (coming soon)


  title={Deep Feedback Inverse Problem Solver},
  author={Ma, Wei-Chiu and Wang, Shenlong and Gu, Jiayuan and Manivasagam, Sivabalan and Torralba, Antonio and Urtasun, Raquel},

Related work

This section will be updated soon. During the meanwhile, please refer to the paper/supp. material for more discussion on relevant approaches. Please also let us know if we miss any prior art!

Template borrowed from an amazing 3D folk.