OpenFace

Download OpenFace (choose 1):

The modified version has ZeroMQ added to it’s interface (C#), which is needed to get the AU values into FACSvatar. The GUI is only available for Windows, therefore real-time animation requires a Windows PC (you can still use e.g. Ubuntu for the other modules if you follow the Multi-machines setup instructions).

Note

Requested: Help from a C++ programmer to implement a ZeroMQ component in the C++ part of OpenFace. That would allow OpenFace to be used on any computer OS. Starting point: https://github.com/TadasBaltrusaitis/OpenFace/issues/492#issuecomment-641814378 Please make an issue on GitHub if you want to do this.

Warning

The output of the Real-time approach is slightly different from the Offline approach. Once OpenFace is done analyzing a video, it applies some post-processing, meaning the Offline approach is slightly more accurate. Therefore, only use the Real-time approach for real-time use cases. Otherwise, use the generated .csv.

Warning

All copyright of OpenFace belongs to Carnegie Mellon University. By using this software, you agree to their licensing terms. Check the License info page to find out more (especially if you want to use it in commercial projects).

Download OpenFace tracker models

TODO Execute download scripts

Note

Additional OpenFace instructions can be found here: https://github.com/TadasBaltrusaitis/OpenFace/wiki#installation

Real-time GUI

  1. Double click OpenFaceOffline.exe –> menu: File –> Open Webcam

Windows 7, 8 and 10 Home version <2004 - only

  1. Navigate inside folder openface_x.x.x_zeromq

  2. (Windows 7/8/10 Home version <2004 - only) Get Docker machine ip by opening a 2nd terminal and execute: docker-machine ip (likely to be 192.168.99.100)

  3. (Windows 7/8/10 Home version <2004 - only) Open config.xml, change <IP>127.0.0.1</IP> to <IP>machine ip from step 3</IP> (<IP>192.168.99.100</IP>) and save and close.

  4. Double click OpenFaceOffline.exe –> menu: File –> Open Webcam

Warning

Requires a Windows PC (for now). Also, it’s pretty heavy on the CPU of the PC.

Offline

If you analyze a video with OpenFace, it will output a .csv file with the analysis results. FACSvatar only requires the AU values and ignores

Python

  1. conda activate facsvatar

  2. cd *your_path*/FACSvatar/modules/input_facsfromcsv

  3. python main.py

    • Execute python main.py -h to see more possible arguments, including explanations

Docker

1.. Go inside Docker container: docker-compose exec facsvatar_facsfromcsv bash (same as Quickstart) #. python main.py

Use your own videos

If you want to use your own FACS values extracted from a video do:

  1. OpenFaceOffline.exe –> menu: Recording settings –> Output location

  2. OpenFaceOffline.exe –> menu: File –> Open Video.

  3. Wait for analysis to be completed and the .csv to be created.

Python

  1. Copy .csv from your output location to: *your_path*/FACSvatar/modules/input_facsfromcsv/openface/*some_folder*

    Warning

    NOT in a folder with _clean at the end (e.g. not default_clean). A some_folder_clean is automatically generated after e.g. some_folder/OF_output.csv has been been passed as an argument to main.py.

  2. Open a terminal and follow the steps under Offline, but tell it to check your “some_folder”: python main.py --csv_folder some_folder --csv_arg -1

Docker

If you’re using Docker to run FACSvatar, you have to do 1 step more to give the container access to your new file.

Copy into container:

  1. Follow the Quickstart

  2. Copy the .csv into the container: docker cp foo.csv facsvatar_facsfromcsv:/openface/*your_folder*/foo.csv

  3. Open a terminal and follow the Docker steps under Offline, but tell it to check your “some_folder”: python main.py --csv_folder some_folder --csv_arg -1

TODO: Give Docker access to folder on disk through mounting that folder via the Docker file

Build OpenFace with ZeroMQ

Warning

Last tested with OpenFace v2.1.0

  1. Overwrite MainWindow.xaml.cs in OpenFaceguiOpenFaceOffline with openface/MainWindow.xaml.cs from FACSvatar GitHub

  2. Open visual studios:

    • Open OpenFace/OpenFace.sln with Visual Studio 2015

    • Open OpenFace_vs2017.sln with Visual Studio 2017 (didn’t work for me so far)

  3. (In visual studio) Right click in “Solution Explorer” on “OpenFaceOffline” –> Manage NuGet Packages…

  4. Browse and search for netmq; install NetMQ by NetMQ with version v4.0.0.1 (AsyncIO.0.1.26) note: Search under “Browse” not “Installed”

    • Don’t update AsyncIO to a newer version (v0.1.40)

  5. Search for json; Install Newtonsoft.Json by James Newton-King v11.0.2

  6. Select OpenFaceOffline –> Release, x64, OpenFaceOffline –> Build –> (Re)build OpenFaceOffline

  7. Copy config.xml from FACSvatar GitHub and put it at OpenFacex64Releaseconfig.xml # DON’T FORGET - otherwise crashes at startup