Explain about parameters of Candide face model

I saw that on the function you write to read parameters from candide.npz

def load3DFaceModel(filename):
    faceModelFile = np.load(filename)
    mean3DShape = faceModelFile["mean3DShape"]
    mesh = faceModelFile["mesh"]
    idxs3D = faceModelFile["idxs3D"]
    idxs2D = faceModelFile["idxs2D"]
    blendshapes = faceModelFile["blendshapes"]
    mesh = fixMeshWinding(mesh, mean3DShape)
    return mean3DShape, blendshapes, mesh, idxs3D, idxs2D

It is seem that content of file candide.npz is different from candide file from http://www.icg.isy.liu.se/candide/candide3.wfm And can you explain the meaning of mean3DShape, mesh, idxs3D, idxs2D, blendshapes array?

[MarekKowalski]

Hi,

The content of candide.npz is basically a processed version of the original candide model you posted in your link.

• mean3DShape: this corresponds to the neutral state of the face, which is the vertex list in Candide.
• mesh: the original mesh provided by Candide as face list.
• idxs3D and idxs2D: those are manually marked indices of corresponding points between the Candide model (idxs3D) and the face alignment point set (idxs2D). An example of a corresponding point would be the left outer eye corner. Both Candide and the face alignment point set have that model marked, thus its index is probably specified in idxs3D and idxs2D.
• blendshapes: blendshapes are shapes you can add to the Candide neutral state face to modify it. An example of a blendshape would be a smile or an eye brow raise. The blendshapes are derived from the Animation Units defined in Candide.

If you have any other questions, or if that was not clear, please let me know.

Marek

[MarekKowalski]

Hi,

Congratulations on converting the app to C++! There might be several reasons for this behaviour:

• The 3D model vertices that are moved by the shape unit you added are not connected to any 2D landmark vertices. You would have to looks at idxs3D and check.
• The landmark detection does not actually represent the eyelid closure. To check that you would have to draw the detected landmarks and see if the landmarks on the eyes actually “close” when you close your eyes.
• If it is none of the two above it might be a problem with the texture warping around the eye region. For example if the eyelid region is very small on the photo you are swapping, there might be some issues, not sure about that though.

What I would suggest is that you try removing the triangles of the mesh that are located in the eye region so there are holes in those areas of the model. If you do that the swapped face will show the user’s real eyes as it does with the mouth. This might prove more realistic than the current effect.

Let me know what you did and how it went.

Marek