How to use facial landmarks obtained from dlib

Now, sometimes we need to do more than just detecting a face like check if the eyes are closed, or see if the tongue is out.

To do this, we’d need to extract, crop, measure specific parts of the face.

Let’s see step by step how to achieve this:

Step 1:

Use dlib to get facial landmarks on any image containing a face.

This article is more about how to use the result from the dlib face detection results, so I won't go deep into the initial face detection part.

Here’s the python code for it:

This is the result we’d get by running this code:

dlib facial landmarks

Now, in terms of code, the below line gives us the rectangle for each face:

rects = detector(gray, 1)

Where rects is a list of rectangles having one element for each face in the image

shape contains the list of coordinated corresponding to each facial part

We convert it into a NumPy array to access each part:

from imutils import face_utils

shape = face_utils.shape_to_np(shape)

Step 2

Now, here is the guide for us to know what index will be what part of the face:

Reference for indices in “shape” NumPy array

Let’s say we want to just extract the “mouth” part.

As we can see in this image above, indices 49 through 68 corresponds to the mouth.

We can fetch these indices manually :

shape[48:67] #(index starts from 0)

OR

use face_utils(from imutils):

face_utils.FACIAL_LANDMARKS_IDXS.items()

The face_utils.FACIAL_LANDMARKS_IDXS.items() is a dictionary constant with the value shown below:

FACIAL_LANDMARKS_68_IDXS = OrderedDict([(“mouth”, (48, 68)), (“inner_mouth”, (60, 68)), (“right_eyebrow”, (17, 22)),(“left_eyebrow”, (22, 27)), (“right_eye”, (36, 42)), (“left_eye”, (42, 48)),(“nose”, (27, 36)), (“jaw”, (0, 17))])

Loop through this dictionary and get the indices (x and y) for entry where value is “mouth”, get the coordinates, and create a bounding box to extract that part from the image.

The result from the above code:

Extracted “mouth” part

Similarly, you can extract other parts:

inner mouth

jaw

left eye

left eyebrow

right eye

right eyebrow

nose

The python script for this one is available on my GitHub: https://github.com/mansikataria/tongue-tip-detection-and-tracking/blob/main/facial_landmarks.py

That’s it for this one folk!

Ciao….

Feel free to connect with me on Twitter or LinkedIn and drop me a note if you have any questions. If you like this article, follow me right here on Medium.

Mansi Kataria

Email: mkataria920@gmail.com
LinkedIN: https://www.linkedin.com/in/mansirkataria/
Twitter: https://twitter.com/_mansi___
Medium: https://zoomout.medium.com