In a report published today by the National Institutes of Science and Technology (NIST), a physical sciences laboratory and non-regulatory agency of the U.S. Department of Commerce, researchers attempted to evaluate the performance of facial recognition algorithms on faces partially covered by protective masks. They report that even the best of the 89 commercial facial recognition algorithms tested had error rates between 5% and 50% in matching digitally applied face masks with photos of the same person without a mask.
“With the arrival of the pandemic, we need to understand how face recognition technology deals with masked faces,” said Mei Ngan, a NIST computer scientist and an author of the report, in a statement. “We have begun by focusing on how an algorithm developed before the pandemic might be affected by subjects wearing face masks. Later this summer, we plan to test the accuracy of algorithms that were intentionally developed with masked faces in mind.”
The study — part of a series from NIST’s Face Recognition Vendor Test (FRVT) program conducted in collaboration with the Department of Homeland Security’s Science and Technology Directorate, the Office of Biometric Identity Management, and Customs and Border Protection — explored how well each of the algorithms was able to perform “one-to-one” matching, where a photo is compared with a different photo of the same person. (NIST notes that this sort of tech is often used to unlock smartphones and check passports.) The team tested the algorithms on a set of about 6 million photos used in previous FRVT studies, but they didn’t test the algorithms’ ability to perform “one-to-many” matching, which is used to determine whether a person in a photo matches any in a database of known images.
Because real-world masks differ, the researchers came up with nine mask variants to test, which included differences in shape, color, and nose coverage. The digital masks were black or a light blue that is approximately the same color as a blue surgical mask, while the shapes included round masks that cover the nose and mouth and a larger type as wide as the wearer’s face. These wider masks had high, medium, and low variants that covered the nose to different degrees.
According to the researchers, algorithm accuracy with masked faces declined “substantially” across the board. Using unmasked images, the most accurate algorithms failed to authenticate a person about 0.3% of the time. Masked images raised even these top algorithms’ failure rate to about 5%, while many otherwise competent algorithms failed between 20% to 50% of the time.
In addition, masked images more frequently caused algorithms to be unable to process a face, meaning they couldn’t extract a face’s features well enough to make an effective comparison. The more of the nose a mask covers, the lower the algorithm’s accuracy — accuracy degraded with greater nose coverage. Algorithm error rates were generally lower with round masks and black masks as opposed to surgical blue ones. And while false negatives increased, false positives remained stable or modestly declined. (A “false negative” indicates an algorithm failed to match two photos of the same person, while a “false positive” indicates it incorrectly identified a match between photos of two different people.)
“With respect to accuracy with face masks, we expect the technology to continue to improve,” continued Ngan. “But the data we’ve taken so far underscores one of the ideas common to previous FRVT tests: Individual algorithms perform differently. Users should get to know the algorithm they are using thoroughly and test its performance in their own work environment.”
The results of the study align with VentureBeat’s survey earlier this year, which found that facial recognition algorithms used by Google and Apple struggled to recognize users. But crucially, NIST doesn’t take into account systems designed specifically to identify mask wearers, like those developed by Chinese company Hanwang and researchers from Wuhan University. In an op-ed earlier in April, Northeastern University professor Woodrow Hartzog said face masks are a temporary speed bump for facial recognition. While they present a challenge, he believes face masks will not stand in the way of increased facial recognition use in the age of COVID-19.
Perhaps in recognition of this, this summer, NIST plans to test algorithms created with face masks in mind and conduct tests with one-to-many searches and other variations.