From an early age and throughout one’s lifespan, emotional skills are essential to communicate our emotions to others and adapt our behavior according to both our internal feelings and the reactions of those around us. Understanding how we feel and showing empathy towards others are essential skills for individuals of any age. Visual signals, such as facial expressions, are crucial components for people to understand as emotional signals. For children, understanding facial expressions and developing empathy for others are crucial for their social-communication development. However, children with Autism Spectrum Conditions (ASC) have major difficulties in detecting and recognizing the emotional and mental states of other people's facial expressions (Baron-Cohen).

The problem that we are trying to tackle involves deducing human emotions, but specifically, detecting what facial expressions represent in terms of how an individual may feel is not considered an easy task. Our focus will be on specifically assisting children with Autism Spectrum Conditions (ASC) to learn how to detect facial expressions in their parents and peers of all races and genders. While our focus is on children, anyone would be able to use this to learn about facial expressions and what they represent.

Distinguishing between different facial expressions has its difficulty as even people disagree on what emotion a face may be portraying. This brings up the question, if people have potential to disagree, how could a computer be able to be 100% accurate?

Facial recognition has not reached the point of perfection, yet it has become a technology that is now widely used for many applications such as for smart technology like Face ID, security surveillance, access control, etc. We decided to focus on the problem involving the deduction of human facial emotion. This especially assists children who are affected by ASC in getting more familiar in detecting certain emotions.

These are two test images from the ChildEFES database, a set of facial expressions focusing on children (Negrão). One of these test images is fear while the other is surprise, can you guess which one is which?

Emotion recognition can be applied in many different situations. Being able to detect facial expressions can give an insight into how people may be feeling, without any speech or actions involved. For children, especially those with Autism Spectrum Conditions (ASC), recognizing facial expressions can be very difficult, and not building this skill early on can potentially stunt their ability to form social skills and relationships later in life.

The current state of the art for facial expression detection involves machine learning algorithms. There are many different kinds of algorithms that attempt to do facial detection. There are several examples of these algorithms such as knowledge-based detection, template matching, and feature-based detection. Knowledge-based detection is where there are rules defined in order to parse individual’s facial features. Template matching is more self explanitory where they use predefined templates to compare facial details, especially focusing on edge detection.

Feature-based detection extracts structural features of the face and attempts to parse facial landmarks as emotions are largely shown through the eyes, eyebrows, lips, and other movements in the face. Facial detection algorithms aren’t limited to these we listed, however they are commonly used approaches when it comes to putting a face together. Covariance pooling is also used among learning algorithms to determine where the facial landmarks are located on an individual's face, while dimensionality reduction helps reduce the “noise” in the face that isn’t relevant to determine what type of emotion is recognized. There are many different ways to approach facial expression detection and each method comes with its pros and cons. Many algorithms have fatal flaws when it comes to things like variance in lighting, skin color or angle that the image was taken from. We are trying to use an implementation that works especially with the change in skin color or lighting as we plan to focus on a diverse subset of individuals to test on.

We attempted to recreate an implementation for emotion detection by starting out with detecting faces in a given image.

We took advantage of OpenCV along with Python and were able to detect any faces in an image and then make it greyscale by:

We then were able to use OpenCV, specifically the cv2 python library in order to set a cascade to detect any potential faces in the image with:

These were the initial steps in order to find a face in a given image.

We had several options for facial detection yet ended up going with Appearance-based detection.

From this point, we attempted to create models that could be run over test images, which would over time get closer to correctly determining where to look and parse the facial landmarks of our found face.

However, we were unable to create working models that fit our liking, so we went with an alternate approach, of taking advantage of both the DeepFace and Facial Emotion Recognizer aka FER libraries in Python. With these libraries, we were able to test their capabilities with our test image database and came to some meaningful conclusions that are mentioned later.

We decided to test existing implementations on our dataset by using the DeepFace and Facial Expression Recognizer libraries in python. These are several results versus expected test images using these libraries and we realized that certain emotions are problematic to point out. Scroll down to see the results from our findings!

We found that it was easier to detect the emotion of happiness than any other emotion. In contrast, the hardest emotion to detect was disgust with every disgust test image resulting in "angry". We did not see disgust come up as a result even once during our research.

We also compared some test images against each other using these libraries and we realized that certain emotions were harder to point out. For example, test images including emotions such as fear versus surprise led to less accurate results in comparison to images that contained happiness versus sadness. For those with a darker complexion, the results tended to be less accurate than those with a lighter complexion.

Our hypothesis is that with emotions that are based around more subtle facial movements, such as movement of the eyes or the brows, DeepFace struggled to detect them. Meanwhile, for emotions that are more based around the movement of the mouth such as happiness, it was much easier to detect.

Throughout this project, we learned that detecting facial expressions is not an easy task. There are already tools out there that have the ability to find facial expressions, yet even those have their downfalls especially when it comes to different races or low-quality images.

While we focused on children with ASC, there are many applications for this technology in the future. Facial and emotional recognition is bound to become more accurate in the near future, as it is still a topic that many people are still focusing on finding ways to improve. Here are a few examples of what this can look like.

Firstly, facial and emotion recognition can help in fields such as law enforcement and help detect criminals. Secondly, this technology can contribute to various forms of research technology, including pattern recognition networks. Another industry this technology can help is the entertainment industry– specifically, the video game industry. There are already games such as Left 4 Dead 2 that have experimented with biometrics directly by introducing them into special builds (Szwoch). Lastly, emotional recognition technology is being researched today for automotive AI. Using this technology, automakers are trying to understand the driver’s status, like their emotions, tiredness level and attention, and responses to the drive systems and ride quality as a way to make roads safer and prevent accidents (Sahu).