Facial morphology analysis and eyewear recommendation algorithms

Do you have a round face? At first sight, the shape of glasses that would suit you best would be angular and stretched. Do you have a square face? Try butterfly glasses! Depending on the shape of your face, certain frame types will suit you better than others. While virtual eyewear fitting plug-ins have been flourishing on opticians' websites for some years now, they don't offer any features to help users find the frames best suited to their morphology.

The VESPA project, led by Jacques Boonaert and Stéphane Lecoeuche, within IMT Lille Douai's Computer Science and Control research unit, aims to develop, in collaboration with Pascal Mobuchon from virtual fitting plug-in supplier Acep Trylive, a machine learning algorithm capable of determining the shape of the user's face and recommending types of eyewear based on its morphological characteristics. " Acep Trylive has developed a technology for placing key points on users' faces, filmed with their webcam during the virtual eyewear fitting, " explains Stéphane Lecoeuche. " The aim of the VESPA project is to develop algorithms capable of analyzing the placement of key points on faces to determine the user's morphology. " All data used in the project is anonymized, and hosted on the secure, neutral and sovereign TeraLab platform. TeraLab also provides researchers with tools for processing these datasets.

Algorithms capable of determining facial morphology

The algorithms developed by the researchers follow a supervised learning logic. " This means that the algorithms are given a set of images labeled by human experts, who determine whether the person's face is round, square, oval, etc.," explains Jacques Boonaert.

In addition, the key points automatically positioned on the user's face by Acep Trylive's software provide the algorithm with a set of descriptors. A descriptor can be, for example, a measurement of key points on the face: it could be the height of the forehead, the shape of the chin, the width of the face or jaw ... From the data labeled by the human experts, the algorithm will determine which descriptors are most relevant for recognizing the user's morphology. " We tested subsets of descriptors. In all, there are over 20," explains Stéphane Lecoeuche. " The algorithms will then weight the influence of the descriptors to best characterize the morphology.

There are three morphological classification algorithms. One focuses on jaw shape, the second on face shape and the third on forehead width. Numerical descriptors of the user's hair, eye and skin color are also used to suggest the most suitable frame colors. All this data is then merged to produce eyewear recommendations tailored to the user's needs.

Recommending products to users based on their morphology

By tracking consumer behavior and history, researchers have been able to determine which glasses each web user prefers: " Someone tries on a first pair of glasses, tries on a second, goes back to the first, tries on a third, goes back to the first again... By observing this sequence of use, we can determine which product the person preferred! " explains Jacques Boonaert.

The learning algorithm is based on these fitting histories, and statistically groups users' preferred eyewear by morphology. " Using data from thousands or even hundreds of thousands of fitting sessions, the algorithm makes the link between face shape and products tried on," explains Stéphane Lecoeuche. So, for each new user using the application, the morphological analysis algorithms will determine their face shape, and, based on the choices of users with similar morphology, the recommendation engine will propose a ranking of products most likely to please them.

A project that needs a partner company to develop

" We also had to work on geometric characterization on the product side. The problem was that we couldn't access data from opticians' catalogs, which classify eyewear by shape, style, color..." notes Stéphane Lecoeuche. The researchers' objective was to work on the association between the customer's morphological characteristics and the geometric characteristics of the products. While algorithmic analysis of facial morphology yielded good results, the lack of frame data limited their objectives.

" The other difficulty is that we didn't have access to users' final purchasing decisions either ," adds Jacques Boonaert. " We are aware that this is sensitive data for companies. That's why we'd like to set up a solid partnership with an optician, in order to perfect this project. " The researchers would like to set up a second phase of experimentation with a partner company, where the algorithm could integrate users' purchasing decisions and the geometric characterization of products. While waiting for a company to join the project consortium, the research team is continuing its machine learning work for industry and commerce.