Masks are making it hard for facial recognition software to recognize people

You’re not the only one having a hard time making sense of who’s who behind their masks. It turns out, even computers find it difficult, too.

A preliminary report by the U.S. National Institue of Standards and Technology revealed that even the best commercial facial recognition systems have error rates as high as 50% in identifying people wearing masks.

The NIST says that the report is the first in its investigation to better understand how masks and other face coverings affect the performance of facial recognition software.

“With the arrival of the pandemic, we need to understand how face recognition technology deals with masked faces,” explains Mei Ngan, a computer scientist at NIST and lead investigator for the report.

In their report, Ngan and her team looked at the accuracy of facial recognition software developed before the COVID-19 pandemic—in particular, how these are affected by subjects wearing masks. The team plans to conduct a separate test for algorithms that have masked faces in mind.

The study used two large datasets of photos to test the effectiveness of 89 pre-COVID-19 algorithms supplied by tech firms and academic labs. The team looked at how the algorithms performed “one-to-one” matching, wherein a person is verified using two different photos. One-to-one matching is often used for verification, including unlocking a smartphone or checking a passport. One dataset contained photos from visa and immigration benefit applications, while the other dataset contained photos taken from U.S. border crossings. To note, the NIST says that it scanned around 6.2 million photos of about 1 million people for the study.

The researchers then digitally applied masks to the photos and tested the algorithms’ performance. To mimic how people wear masks, the team came up with nine variants, which included differences in shape, color and nose shape. With people donning masks everywhere they go, the study attempts to recreate an operational scenario where officials have to verify a masked person’s identity using a visa or passport photo.

The findings revealed that face coverings significantly affected the accuracy of even the best facial recognition systems. Under ideal conditions, the NIST said that these systems have a failure rate of only about 0.3%. However, when masks are added, the failure rate increases to 5% or worse. Lower quality facial recognition systems fared worse, failing to match photos up to 50% of the time.

“We can draw a few broad conclusions from the results, but there are caveats,” Ngan added. “None of these algorithms were designed to handle face masks, and the masks we used are digital creations, not the real thing.”

A redo won’t do much to help the algorithms, said the researchers. For one, the NIST says the error is most likely caused by a problem with the algorithm and not the person. Without a mask, a person can adjust his posture or even his facial expression for the algorithm to get a positive match—something that can’t be done when a mask is on.

The shape and color of the masks can also affect the accuracy of the software, with masks that covering more of the face leading to higher failure rates.

Masks that covered the nose were challenging for algorithms. This finding is particularly important, say the researchers, since covering your nose is critical to mitigating the spread of COVID-19.

Here’s a video from the World Health Organization on how to wear a medical mask, for reference:

The NIST team also stressed the limitations of the report. While the digitally created masks sought to represent the many ways people wear masks, the truth is, masks in the real world come in nearly endless shapes and sizes.

The team also didn’t include how masks will affect the accuracy of algorithms for people wearing glasses, as well as how it affects people of various demographic groups.

“With respect to accuracy with face masks, we expect the technology to continue to improve,” Ngan concluded. “Users should get to know the algorithm they are using thoroughly and test its performance in their own work environment.”