OpenFace integration

OpenFace 1

AUParserFileReader

To reproduce the facial expressions and head movements extracted via OpenFace 1, it is possible use the AUParserFileReader module. It takes as input a csv file containing the timestamp, the values of all AUs, gaze, head pose and head rotation. The output is a video that reproduce frame by frame the values contained in the csv file (example csv: https://github.com/isir/greta/blob/master/bin/Examples/AUParser/AUParser_example.csv).

Modules Connections

The module has to be connected to the SimpleAUPerformer to send the AUs to the agent and MPEG4Animatable to send the BAP values for the Head rotation. One more connection that has to be create is from the ParserCaptureControllerAU to the AUParserFileReader. (see figure below)

How it works

1. Pushing the button Open it is possible to select the directory where are stored the csv files.

2. Then pushing the button Send, the list of all variables in the csv files will appear in the window below (Headers).

3. Select the variables you want to reproduce, chekcing the related checkbox (or just push the button Select All).

4. Push the button Set. After that you are ready to record.

5. To start the recording go to the ParserCaptureControllerAU interface and push the button Record all files

Some info

• the frame rate for the video is computed taking the difference between the timestamp value of the first two rows

• keep the “Real Time” checkbox selected if you want the duration of the video be as long as the real time duration of the file

OpenFace 2

OpenFace 2, developed at the CMU MultiComp Lab, is a facial behavior analysis toolkit capable of real-time performance. The system is capable of performing a number of facial analysis tasks:

• Facial Landmark Detection

• Facial Landmark and head pose tracking (links to YouTube videos)

• Facial Action Unit Recognition

• Gaze tracking (image of it in action)

• Facial Feature Extraction (aligned faces and HOG features)

It can forward all facial expressions, head pose, and gaze to any other software

It can either :

• read the standard output file from the standard OpenFace2 program. Reading is dynamic, synchronous with OpenFace writing.

• connect via ZeroMQ (https://zeromq.org/) protocol on port 5000. Using the OpenFaceOfflineZeroMQ application (built with this branch of OpenFace 2 (https://github.com/isir/OpenFace/tree/feature/streaming-AU-ZeroMQ))

In Greta use “OpenFace2 Output Stream Reader” module to listen to OpenFace inputs and connect them to the specific Greta modules handling AUs and headpose.

The module allows :

1) OpenFace data Input selection 2 Signal processing 3) To forward any information to a debug application using the OSC protocol

OpenFace data Input selection

The UI allows dynamic and easy selection of the OpenFace 2 information. It can be used to use only a specific set of AUs.

Signal processing

A Facial action unit is composed of:

*a continuous signal : how much is feature is activated * a discrete signal : is feature detected or not

Raw signal :

Mask :

Hence the need to filter the signal processing filter.

Filtered signal (kernel size of 5, weight function with power 0.5) :

A dynamically sized kernel processing approach where the most recent signal value is the last index of the kernel. Each kernel weight is valued with the mathematical “pow” function which conveniently grows from 0 to 1 for x =[0-1]. So the most recent values have the most weight.

12 normalized kernel values for different power values

Demonstration videos :

• AU4-brow Lowerer

• AU2-outer brow raiser

• AU26-jaw drop