Differentiable Vector Graphics Rasterization for Editing and Learning
| Tzu-Mao Li | Michal Lukáč | Michaël Gharbi | Jonathan Ragan-Kelley |
| MIT CSAIL | Adobe Research | Adobe Research | MIT CSAIL |
We introduce a differentiable rasterizer for vector graphics that bridges the raster and vector domains through backpropagation. Differentiable rasterization enables many novel vector graphics applications. (a) Interactive editing that locally optimizes for image-space metrics, such as opacity, under geometric constraints. (b) A new painterly rendering technique by fitting random Bézier curves to a target image. (c) Improving state of art image vectorization result. (d) Editing vector graphics using potentially non-differentiable raster image processing operators, such as seam carving [Avidan et al. 2007] for image retargeting. (e) Training a variational autoencoder to generate vector MNIST digits and adding stylized strokes as postprocessing.
Abstract
We introduce a differentiable rasterizer that bridges the vector graphics and raster image domains, enabling powerful raster-based loss functions, optimization procedures, and machine learning techniques to edit and generate vector content. We observe that vector graphics rasterization is differentiable after pixel prefiltering. Our differentiable rasterizer offers two prefiltering options: an analytical prefiltering technique and a multisampling anti-aliasing technique. The analytical variant is faster but can suffer from artifacts such as conflation. The multisampling variant is still efficient, and can render high-quality images while computing unbiased gradients for each pixel with respect to curve parameters.
We demonstrate that our rasterizer enables new applications, including a vector graphics editor guided by image metrics, a painterly rendering algorithm that fits vector primitives to an image by minimizing a deep perceptual loss function, new vector graphics editing algorithms that exploit well-known image processing methods such as seam carving, and deep generative models that generate vector content from raster-only supervision under a VAE or GAN training objective.
Introduction Video
SIGGRAPH Asia Presentation
Publication
Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, Jonathan Ragan-Kelley
Differentiable Vector Graphics Rasterization for Editing and Learning.
ACM Transactions on Graphics 39(6) (Proceedings of ACM SIGGRAPH Asia 2020)
BibTeX
Errata
In Equation 6 in the paper, the line integral should have measure dp(t) instead of dt (or if we are keeping the dt measure, we should include the Jacobian ||p'(t)||). In Equation 7, we should use function g instead of function f.
More precisely, the line integral at Equation 6 should read: $$ \int_{\partial A_i(\Theta)} \left(\frac{\partial p(t)}{\partial \Theta} \cdot n(t)\right) g(p(t)) d{\color{Red}p(t)} $$
And Equation 7 should read: $$ \int_{\partial A_{i}(\Theta)} \left( \nabla_{\Theta} p \cdot n \right) {\color{Red}g(p(t))} d{\color{Red}p(t)} \approx \\ \frac{1}{N} \sum_{j} \frac{\left(\nabla_{\Theta} p_j \cdot n_j \right) \left( {\color{Red}g}(p_j + \epsilon n_j) - {\color{Red}g}(p_j - \epsilon n_j) \right) }{P(p_j|c)P(c)} $$
Downloads
Acknowledgement
The main author is funded by DARPA under award HR0011-20-9-0017, from the PAPPA program. We thank Andrew Adams and Gilbert Bernstein for their helpful comments. We thank people who provided the graphics used in the figures: wikipedia users Daderot (fallingwater), Eric Guinther (flower), and David Corby (kitty), freesvg.org user OpenClipart (The Wave), and vecteezy.com users Vectorbox Studio and Graphics RF (cat and landscape).