Automatic image semantics

As online video-on-demand platforms flourish, so do the associated recommendation algorithms, capable of taking into account - among other things - viewers' tastes according to the genre, actors and theme of the program. In short, to increase the chances of getting it right. Artificial intelligence now makes it possible to go even further: it's now possible to specify the location of the plot, the type of shots and actions, or even the sequence of scenes.

This is the objective on which teams from France Télévisions and Télécom SudParis have been working since October 2019 and the launch of the joint IA TV laboratory. They are focusing on automating the analysis of fictional video content. " Today, our recommendation rules are very basic.If a viewer enjoyed a piece of content, a program, a film, a documentary, we don't know much about why they enjoyed it, or even the characteristics of the content itself. For a fictional program, was it the era, the cast, theplot? There are so many dimensions that could have seduced him ," emphasizes Matthieu Parmentier.

AI applied to fictional content

The aim of the partnership is precisely to explore these dimensions. Using deep learning, a technique based on neural networks, the researchers run their algorithm through a massive quantity of videos. Successive layers of neurons extract and analyze increasingly complex features of a visual scene: the first layer takes the pixels of the image, the last provides the labels.

" Thanks to this technology, we are able to categorize content, i.e. to classify each sequence, each scene, in order to know, for example, whether it is shot outdoors or indoors, to recognize the characters/actors involved, to identify objects or places of interest as well as the relationships between these different elements, or even to extract emotional or aesthetic characteristics. Our aim is to make the machine capable of automatically deriving a semantically similar interpretation of a scene to that of humans," explains Titus Zaharia.

Researchers have already achieved convincing results. Does the scene take place in a car? In a park? Inside a bus? The tool suggests the most relevant categories in order of probability. Their algorithm also manages to determine the shot values of the sequences analyzed: wide shot, long shot, close-up. " This was not yet available on the market," enth uses Matthieu Parmentier. " And in addition to detecting these shot changes, the algorithm is able to identify those that belong to the same scene.

For France Télévisions, the applications will be numerous. Firstly, the automatic extraction of key frames, i.e., for each sequence and according to aesthetic criteria, the most representative image to illustrate a fictional content; secondly, the identification in a program of the "ideal" moments between which to introduce a commercial. "However, we are currently working on still video shots. One of our next objectives is to be able to characterize moving shots such as zooms, travellings or panoramas. For us, this could be very interesting in terms of editing assistance or content reuse," adds Matthieu Parmentier.

Multimodal AI solutions

Teams from France Télévisions and Télécom SudParis have been working together for over five years to adapt to the new digital uses of television viewers. They have contributed to the creation of artificial intelligence solutions and tools applied to digital images, but also to other forms of content, text and sound. In 2014, the two structures launched the collaborative project Média4Dplayer, a prototype media player designed for all four screens (TV, PC, tablet and smartphone), accessible to all and particularly to aging populations or those with disabilities. A few months later, they turned their attention to automatic subtitle generation. Here, the interests are manifold: equal access to content or the possibility of viewing a video without sound.

" In the case of television news, for example, subtitles are generated live by small hands, professional scribes. We've all had experience of this, and it can sometimes lead to errors, but above all to a discrepancy between what we hear and what we read," explains Titus Zaharia, a professor at Télécom SudParis specializing in AI applied to multimedia content. The solution developed by the two teams has enabled this synchronization to be produced automatically for France TV's Replay offering. After two and a half years of development, they have filed a joint patent for this technology. " Eventually, we hope to be able to offer perfectly synchronized subtitles a few seconds after the broadcast of any type of live program," continues Matthieu Parmentier, head of the Data & AI department at France Télévisions.

" France Télévisions has many unresolved scientific research problems, particularly in the field of artificial intelligence. What we're interested in is developing tools that they can use and industrialize rapidly, but which will be sufficiently generic and methodologically promising to find other fields of application in the future ," concludes Titus Zaharia.