In this work we present three novelties. (I) A large-scale synthetic dataset with photo-realistic images of 80 subjects performing 70 activities and wearing diverse outfits. (II) A novel 3D body shape representation based on geometry images. (III) A multi-resolution deep generative network that, given an input image of a dressed human, predicts his/her geometry image (and thus the clothed body shape) in an end-to-end manner.
First, we present 3DPeople, a large-scale synthetic dataset with 2.5 Million photo-realistic images of 80 subjects performing 70 activities and wearing diverse outfits. We annotate the dataset with segmentation masks, skeletons, depth, normal maps, optical flow and SMPL. We can NOT share the 3D meshes for copyright reasons. All this together makes 3DPeople suitable for a plethora of tasks.
We represent the 3D shapes using 2D geometry images. To build these images we propose a novel spherical area-preserving parameterization algorithm based on the optimal mass transportation method. We show this approach to improve existing spherical maps which tend to shrink the elongated parts of the full body models such as the arms and legs, making the geometry images incomplete. In the figure: (a) Reference mesh in a tpose configuration color coded using the xyz position. (b) Spherical parameterization; (c) Octahedral parameterization; (d) Unwarping the octahedron to a planar configuration; (e) Geometry image, resulting from the projection of the octahedron onto a plane; (f) mesh reconstructed from the geometry image.
We also designed a multi-resolution deep generative network that, given an input image of a dressed human, predicts his/her geometry image (and thus the clothed body shape) in an end-to-end manner. We obtain very promising results in jointly capturing body pose and clothing shape, both for synthetic validation and on the wild images.
Qualitative results. For the synthetic images we plot our estimated results and the shape reconstructed directly from the ground truth geometry image. In all cases we show two different views. The color of the meshes encodes the xyz vertex position.
This work is supported in part by an Amazon Research Award, the Croucher Foundation and the Spanish MiNeCo under projects HuMoUR TIN2017-90086-R, ColRobTransp DPI2016-78957-R and Mar\'ia de Maeztu Seal of Excellence MDM-2016-0656. We also thank Nvidia for hardware donation under the GPU Grant Program.