Virtual try-on tool for makeup products

Our CV specialists developed a virtual try-on tool for makeup products. The system consists of a face detection and segmentation model and an algorithm that allows recoloring objects without losing their original texture.

The model aimed to detect a person’s face in the picture and separate it into different segments - eyes, lips, cheeks, nose, etc.

These are the stages of processing that the system applies:

• Face detection, a search of positions for key points of the face;
• Creation of the 3d model of the face;
• Initial assessment of regions (segmentation);
• A refinement in the precision of the masks for the segmentation;

The first step (detection) is a multipurpose convolutional network model that is implemented through a Caffe module. This neural network allows for detecting faces and 5 of its essential key points. In the next step, two Caffe models were used to find initial estimates for the boundaries of the face and its regions. In the last stage, the regions are refined according to their specifics. As a starting point, 11 segments are used. Refinement algorithms are based on the analysis of colours of different parts of the region and the geometric shape and position of parts of the face.

The colouring of the object is implemented through a Python module that works with a selected part of the image. The principle of DRM (Dichromatic reflection model) was taken as the basis, which allows for determining the colour of the object, the colour of the lighting, and the relationship between them at each point of the object matrix. Based on this data, the most natural change in the shade of objects is selected.