Fitting a Face, and Learning When to Stop Optimizing a Noisy Target

Optimizing longer made the faces worse. That is the part of this project I still think about.

We were fitting a parametric face model to a single photograph by minimizing the distance between the model's projected landmarks and 68 landmarks found by a detector. Those detected points are estimates, and they carry error. Past a certain point the solver runs out of real geometry to explain and starts explaining the detector's mistakes instead, which lowers the loss and deforms the face. Early stopping and L2 regularization on the identity and expression coefficients were not tuning. They were a statement about how far to trust the target relative to the prior.

The setup is analysis-by-synthesis, which I have kept as a way of thinking. A morphable model is a generative prior over faces, a mean shape plus PCA bases, where 300 identity coefficients and 100 expression coefficients synthesize a specific face. Reconstruction becomes inference over those coefficients, with the regularization weights scaled by the PCA variances so the answer stays in plausible territory. You are not learning a mapping from image to geometry. You are taking something that can generate faces and fitting it to the evidence.

We refused to treat it as one large optimization. Model inspection, landmark detection, semantic correspondence, pose initialization, shape and expression fitting, texture, export, and every stage produced an artifact you could look at. Meshes exported for inspection in 3D software. Debug images from the detector. Correspondences tagged with their semantic names, so a point claiming to be the nose tip had to prove it. This matters more in an inverse problem than in ordinary software, because an upstream mistake does not crash anything. It produces a wrong answer that looks entirely reasonable.

Two stages carried more weight than their size suggests. Correspondence, because if the 2D landmarks map to the wrong vertices the optimizer will faithfully fit the wrong face. And initialization, which we solved in two steps, rotation from a Perspective-n-Point solve and then scale and translation under a weak perspective camera. Once the wireframe locked onto the jaw, nose bridge, eyes and mouth, everything downstream behaved. In a non-convex fit the initialization chooses which basin you land in.

We also wrote our own projection and gradients rather than handing everything to a solver library. It cost time and bought control: numerical stability, and the ability to enforce constraints like keeping vertices in front of the camera and bounding the coefficients. I would not make that trade every time, but when the failures are geometric and you diagnose them by looking, owning the projection math helps.

The evidence is qualitative, and I would rather say so than let a nice render imply otherwise. We projected the wireframe back onto the photograph and judged the alignment, and the report centers on the rigid pose stage. There is no geometric error measured against ground-truth scans, so the claim is that pose recovery is visually verified for both face models, not that reconstruction accuracy has been established. Occlusion and unusual poses degrade the landmarks, and everything downstream follows from there.

Mostly this project handed me the noisy-target lesson early, and it keeps coming back. A solver fitting detector error here. A prompt search optimizing a rubric that stands in for clinical quality later. A reward signal standing in for which reasoning steps mattered after that. The optimizer is always faithful to the target you give it, so the question worth asking is rarely whether the loss went down. It is whether the target still means what you think it means by the time you stop.