mm.utils package

Submodules

mm.utils.io module

This module contains functions that concern the input and output of files.

mm.utils.io.exportObj(v, c=None, vt=None, f=None, fNameIn=None, fNameOut='test.obj')

Writes vertices to an .obj file.

Parameters:
  • v (ndarray) – Vertex coordinates
  • c (ndarray) – Optional, vertex colors
  • vt (ndarray) – Optional, mapping of vertex coordinates to UV coordinates
  • f (ndarray) – Optional, array of vertex indices that correspond to each mesh face
  • fNameIn (str) – Optional, filename of .obj file to overwrite certain attributes with
  • fNameOut (str) – Optional, output filename
Returns:

.obj file
mm.utils.io.importObj(fName, dataToImport=['v', 'f'])

Returns the shape vertices and the list of vertex indices for each mesh face.

Parameters:
  • fName (str) – Filename of .obj file
  • dataToImport (list) – A list containing strings that indicate what part of the .obj file to read (v = geometric vertices, f = face indices). Note that all of the .obj files for a given 3DMM have the same face indices.
Returns:

the vertex coordinates, the vertex indices for each face, or both
Return type:

ndarray or tuple
mm.utils.io.speechProc(fName, numFrames, fps, kuro=False, return_extras=False)

Inputs an audio file and processes audio feature vectors that coincide in time with the source video frames. The audio feature vectors are based on Mel frequency cepstral coefficients (MFCCs). Because the sampling rate in the audio file is likely higher in frequency than that of the video frames, this function finds the samples of the audio features that are nearest neighbors in time to the frames of the video.

Parameters:
  • fName (str) – Filename of the audio file
  • numFrames (int) – Number of frames for the source video
  • fps (int) – The amount of frames per second for the source video
  • kuro (bool) – True if the audio file is the target, and False if it is the source
  • return_extras (bool) – True to also return the full, unsampled audio features and the time vector for the video frames in addition to the sampled audio features, and False to only return the sampled audio features.
Returns:

sampled audio features, or (sampled audio features, full audio features, time vector)
Return type:

ndarray or tuple

mm.utils.mesh module

This module contains functions that concern operations on 3DMMs. Perhaps these will be integrated into the MeshModel class later on.

mm.utils.mesh.barycentricReconstruction(vertices, pixelFaces, pixelBarycentricCoords, indexData)

Reconstructs per-pixel attributes from a barycentric combination of the vertices in the triangular face underlying the pixel.

Parameters:
  • vertices (ndarray) – An array of a certain per-vertex attribute, e.g., vertex coordinates, vertex colors, spherical harmonic bases, etc., (n, numVertices)
  • pixelFaces (ndarray) – The triangular face IDs for each pixel where the 3DMM is drawn, (numPixels,)
  • pixelBarycentricCoords (ndarray) – The barycentric coordinates of the vertices on the triangular face underlying each pixel where the 3DMM is drawn, (numPixels, 3)
  • indexData (ndarray) – An array containing the vertex indices for each face, (numFaces, 3)
Returns:

The per-pixel barycentric reconstruction of the desired per-vertex attribute
Return type:

ndarray
mm.utils.mesh.calcNormals(vertexCoord, model)

Calculates the per-vertex normal vectors for a model given shape coefficients.

Parameters:
  • vertexCoord (ndarray) – Vertex coordinates for the 3DMM, (3, numVertices)
  • model (MeshModel) – 3DMM MeshModel class object
Returns:

Per-vertex normal vectors
Return type:

ndarray
mm.utils.mesh.generateFace(param, model, ind=None)

Generates vertex coordinates based on the 3DMM eigenmodel and the shape identity parameters, the shape facial expression parameters, and the similarity transform parameters.

Parameters:
  • param (ndarray) – Contains the concatenation of the shape identity parameters, the shape facial expression parameters, the three Euler angles, the three translation vector terms in Cartesian space, and a scaling factor. The amount of shape parameters should match the number of shape eigenvectors in the 3DMM.
  • model (MeshModel) – 3DMM MeshModel class object
  • ind (ndarray) – Optional, a list of certain vertex indices in the 3DMM to return
Returns:

vertex coordinates
Return type:

ndarray
mm.utils.mesh.generateTexture(vertexCoord, texParam, model)

Generates vertex colors based on the 3DMM eigenmodel, the vertex coordinates, and the texture parameters and spherical harmonic lighting parameters.

Parameters:
  • vertexCoord (ndarray) – Vertex coordinates for the 3DMM, (3, numVertices)
  • texParam (ndarray) – Contains the concatenation of the texture parameters and a flattened version of the spherical harmonic lighting parameters such as the lighting parameters for each color channel are grouped together. The amount of texture parameters should match the number of texture eigenvectors in the 3DMM.
  • model (MeshModel) – 3DMM MeshModel class object
Returns:

vertex RGB colors
Return type:

ndarray, (3, numVertices)
mm.utils.mesh.subdivide(v, f)

Uses Catmull-Clark subdivision to subdivide a 3DMM with quadrilateral faces, increasing the number of faces by 4 times.

Parameters:
  • v (ndarray) – Vertex coordinates for the 3DMM, (4, numVertices)
  • f (ndarray) – An array containing the vertex indices for each quadrilateral face, (numFaces, 4)
Returns:

Subdivided vertex coordinates and array of vertex indices
Return type:

tuple

mm.utils.opengl module

class mm.utils.opengl.Render(width, height, meshData, indexData, indexed=False, img=None)

Bases: object

OpenGL rendering class

Parameters:
  • width (int) – Pixel width of window/viewport
  • height (int) – Pixel height of window/viewport
  • meshData (ndarray) – 3DMM vertex coordinates and RGB values, concatenated vertically
  • indexData (ndarray) – 3DMM vertex indices for each triangular face
  • indexed (bool) – Determines whether or not to do indexed OpenGL drawing
  • img (ndarray, (height, width, 1 or 3)) – Optional background image for rendering
width

int – Pixel width of window/viewport

height

int – Pixel height of window/viewport

zNear

int – Nearside clipping plane

zFar

int – Farside clipping plane

meshData

ndarray – 3DMM vertex coordinates and RGB values, concatenated vertically

indexData

ndarray – 3DMM vertex indices for each triangular face

numVertices

int – Number of 3DMM vertices

numFaces

int – Number of 3DMM triangular faces

vertexDim

int – Dimensionality of 3DMM vertices

indexed

bool – Determines whether or not to do indexed OpenGL drawing

img

ndarray, (height, width, 1 or 3) – Optional background image for rendering

window

int – Instance of OpenGL GLUT context

shaderDict

dict – Dictionary of OpenGL shader strings

numStoredVertices

int – Number of 3DMM vertices stored in the VBO

configureShaders()

Modifies the window-to-clip space transform matrix in the vertex shader, but you can use this to configure your shaders however you’d like, of course.

grabRendering(return_info=False)

Reads the rendered pixels from the FBO into an array.

Parameters:return_info (bool) – Optional, can choose whether or not to return the barycentric coordinates and triangular face IDs for each pixel. You can only get this extra information when doing non-indexed OpenGL drawing. If False, it only returns the rendering. If True, it returns a tuple containing the rendering, the pixel coordinates where the 3DMM is drawn, the triangular face IDs for each pixel where the 3DMM is drawn, and the barycentric coordinates of the triangular face underlying each pixel where the 3DMM is drawn.
Returns:ndarray or tuple
initializeContext()

Intializes an OpenGL context for rendering.

initializeFramebufferObject()

Creates an FBO and assign a texture to it for the purpose of offscreen rendering. Also assigns textures to hold the barycentric coordinates and face IDs for each pixel during the rendering.

initializeShaders()

Compiles each shader defined in shaderDict, attaches them to a program object, and links them (i.e., creates executables that will be run on the vertex, geometry, and fragment processors on the GPU). This is more-or-less boilerplate.

initializeVertexArray()

Creates the VAO to store the VBOs for the mesh data and the index data and assigns the vertex attributes for the OpenGL shaders.

initializeVertexBuffer(faceID=None)

Assigns the triangular mesh data and the triplets of vertex indices that form the triangles (index data) to VBOs

Parameters:faceID (ndarray) – Optional, an array of triangular face ID numbers for each vertex. This is only used with non-indexed OpenGL drawing, where vertices are replicated in the VBO so that each set of three vertices corresponding to a triangular face can have a vertex attribute representing the index number of the triangular face.
render()

Renders the objects defined in the VAO to the FBO.

resetFramebufferObject()

Erases any drawn objects from the FBO.

updateVertexBuffer(meshData)

Updates the VBO with new 3DMM vertex coordinates and RGB colors without having to reinitialize the VBO.

Parameters:meshData (ndarray) – 3DMM vertex coordinates and RGB values, concatenated vertically
mm.utils.opengl.windowToClip(width, height, zNear, zFar)

Creates elements for an OpenGL-style column-based orthographic transformation matrix that maps homogenous coordinates from window space to clip space.

Parameters:
  • width (int) – Pixel width of window/viewport
  • height (int) – Pixel height of window/viewport
  • zNear (int) – Nearside clipping plane
  • zFar (int) – Farside clipping plane
Returns:

Vectorized transformation matrix
Return type:

ndarray, (16,)

mm.utils.transform module

mm.utils.transform.PCA(data, numPC=80)

Return the top principle components of some data. Input (1) the data as a 2D NumPy array, where the observations are along the rows and the data elements of each observation are along the columns, and (2) the number of principle components (numPC) to keep.

mm.utils.transform.perspectiveTransformKinect(d, inverse=False)

Transformation between pixel indices (u, v) of depth map to real-world coordinates in mm (x, y) for Kinect v1 depth camera (640x480 resolution). Depth values z are in mm. In the forward direction, go from (u, v, z) to (x, y, z). In the inverse direction, go from (x, y, z) to (u, v, z).

mm.utils.transform.rotMat2angle(R)

Conversion between 3x3 rotation matrix and Euler angles psi, theta, and phi in radians (rotations about the x, y, and z axes, respectively). If the input is 3x3, then the output will return a size-3 array containing psi, theta, and phi. If the input is a size-3 array, then the output will return the 3x3 rotation matrix.

mm.utils.transform.sh9(x, y, z)

First nine spherical harmonics as functions of Cartesian coordinates

mm.utils.transform.sph2cart(el, az)

Unit sphere elevation and azumuth angles to Cartesian coordinates

mm.utils.visualize module

Module contents