Home Publications Research Grants Inventions & Patents Awards Additional Research Activities Faculties & Institutions Research Areas

Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation

Li C, Morel-Forster A, Vetter T, Egger B, Kortylewski A (2023)

---

Publication Type: Conference contribution

Publication year: 2023

Journal

Publisher: IEEE Computer Society

Book Volume: 2023-June

Pages Range: 372-381

Conference Proceedings Title: Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition

Event location: Vancouver, BC

ISBN: 9798350301298

DOI: 10.1109/CVPR52729.2023.00044

Abstract

In this work, we aim to enhance model-based face reconstruction by avoiding fitting the model to outliers, i.e. regions that cannot be well-expressed by the model such as occluders or makeup. The core challenge for localizing outliers is that they are highly variable and difficult to annotate. To overcome this challenging problem, we introduce a joint Face-autoencoder and outlier segmentation approach (FOCUS). In particular, we exploit the fact that the outliers cannot be fitted well by the face model and hence can be localized well given a high-quality model fitting. The main challenge is that the model fitting and the outlier segmentation are mutually dependent on each other, and need to be inferred jointly. We resolve this chicken-and-egg problem with an EM-type training strategy, where a face autoencoder is trained jointly with an outlier segmentation network. This leads to a synergistic effect, in which the segmentation network prevents the face encoder from fitting to the outliers, enhancing the reconstruction quality. The improved 3D face reconstruction, in turn, enables the segmentation network to better predict the outliers. To resolve the ambiguity between outliers and regions that are difficult to fit, such as eyebrows, we build a statistical prior from synthetic data that measures the systematic bias in model fitting. Experiments on the NoW testset demonstrate that FOCUS achieves SOTA 3D face reconstruction performance among all baselines trained without 3D annotation. Moreover, our results on CelebA-HQ and AR database show that the segmentation network can localize occluders accurately despite being trained without any segmentation annotation.

Authors with CRIS profile

Involved external institutions

How to cite

APA:

Li, C., Morel-Forster, A., Vetter, T., Egger, B., & Kortylewski, A. (2023). Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (pp. 372-381). Vancouver, BC, CA: IEEE Computer Society.

MLA:

Li, Chunlu, et al. "Robust Model-based Face Reconstruction through Weakly-Supervised Outlier Segmentation." Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2023, Vancouver, BC IEEE Computer Society, 2023. 372-381.

BibTeX: Download