PCHMR: Empowering and Benchmarking Human Mesh Recovery in Privacy-Constrained Real-World Settings

Human pose and body shape are already supported by major personal-device platforms and applications. Meta Quest tracks body motion to animate VR avatars and provides pose-comparison tools for fitness exercises; Apple's Vision framework estimates 3D body joints for exercise tracking, sports analysis, and character animation; Google ML Kit supports mobile fitness-pose recognition and repetition counting; and Snapchat's Clothing Try-On uses a body mesh that follows a person's movement and shape. These systems can observe identifiable images, body geometry, and motion inside homes and other private spaces. Personalizing an HMR model may improve its fit to a particular user, camera, and environment, but collecting those images on a central server creates avoidable exposure. PCHMR studies how HMR models can be trained and adapted while the images remain on the user's device.

Training Setup

PCHMR uses Flower to simulate 10, 100, and 1000 clients. In each global round, 10 clients are sampled uniformly without replacement; in the 10-client setting, all clients participate. Each sampled client makes one pass over its local data before the server aggregates the updates. Data partitions are generated once with seed 42 and then held fixed. Every number on this page comes from a single run.

Privacy setting comparison. PCHMR treats centralized HMR as a reference and evaluates federated training, in which raw images remain on each client, plus a secure-aggregation variant.

Federated training prevents the server from receiving raw images, but the updates and the released model may still reveal information. In the secure-aggregation experiment, the server follows the protocol but may try to inspect the updates; it can see only their aggregate, not each client's update. The study does not claim differential privacy and does not evaluate membership inference, gradient inversion, or model inversion attacks.

Training Budget and Evaluation

The published centralized baselines use 60 passes over the training set. The 10/10, 100/10, and uneven-data federated runs use about 200 passes, while 1000/10 uses 60. For a direct comparison, we also report centralized runs trained for 200 epochs. Optimizer steps and wall-clock time are still not matched, so small differences in error cannot be attributed to federated aggregation.

What is and is not matched. One training-set pass means, on average, one use of every image in the 476,062-image training set.

The validation split contains 27,558 images and is evaluated with batch size 256. It is also used for checkpoint selection; there is no separate held-out test set in this comparison. The learning curves show that the centralized runs reach their best validation MPJPE around epochs 26–27 and then flatten or worsen.

DePoser Local Annotation

DePoser fits SMPL-X to each local image using Sapiens 2D keypoints and predicted joint depths. The depth term uses relative depth structure rather than metric scale, and the method has no separate metric-alignment step. The corrected SI-Log objective is

L_SI-Log = √[(1/n) Σ_i d_i² − λ((1/n) Σ_i d_i)²],   d_i = log ẑ_i − log z_i.

The implementation uses this expression and applies a numerical guard to non-positive depths.

Effect of the Number of Clients

These results must be read together with the training budget. The published centralized baselines use 60 passes over the training set, whereas 10/10 and 100/10 use 200. The 200-epoch centralized baselines match those runs in data passes. They do not match optimizer steps or wall-clock time, and every result is from one run, so the table reports the observed outcomes without assigning the differences to federated training itself.

Client-scale evaluation. Human3.6M protocol-2 validation error in millimeters. Lower is better.

The 1000/10 setting is the only federated run that matches the published 60-epoch baseline in data passes, and it has higher error for both models. The 200-epoch centralized runs provide the equal-data-pass comparison for 10/10 and 100/10, but the federated runs still use more optimizer steps and time. Together with the single-run protocol, these differences mean that the results do not show that federated aggregation itself caused the measured changes.

Uneven Client Data

These experiments vary how strongly actions and poses are concentrated on particular clients. Every uneven-data run uses the same data passes and essentially the same number of optimizer steps as the balanced 100/10 run. Training budget therefore does not explain the low error observed in the most uneven setting. Because each configuration is still a single run, this is reported as an observation rather than a general claim that less balanced data improves HMR.

Natural Partition on 3DPW

Natural scene partitions approximate real deployment, where each user's data is shaped by identity, clothing, environment, camera pose, and local data volume.

3DPW fine-tuning. The natural partition assigns each scene to a client. All errors are in millimeters; lower is better.

DePoser creates mesh labels from each user's images on that user's device, and those generated labels are used for local fine-tuning. For the in-the-wild examples below, the same labels are also used as the evaluation reference. The reported reductions therefore show that the fine-tuned model agrees more closely with DePoser; they do not establish higher 3D accuracy against independent measurements.

Personalization Results

Agreement with DePoser-generated references. All errors are in millimeters; lower values mean that the fine-tuned model is closer to the labels generated by DePoser. They do not measure accuracy against independent 3D ground truth.

DePoser component study. Errors in millimeters on the same 300-image 3DPW subset with independent 3D ground truth. All variants use the same L-BFGS schedule and loss weights. Lower is better.

Replacing OpenPose with Sapiens lowers MPJPE by 12.75 mm and PA-MPJPE by 5.04 mm. Adding the SI-Log relative-depth term lowers them by a further 17.13 mm and 10.99 mm. Both changes contribute to DePoser's result. The method does not use a separate metric-depth alignment step.