Metadata-Version: 2.4
Name: libreface
Version: 0.2.0
Summary: LibreFace model for facial analysis
Home-page: https://boese0601.github.io/libreface
Author: IHP-Lab
Author-email: achaubey@usc.edu
License: USC Research Licence 
        This software is Copyright © 2020 The University of Southern California. All Rights Reserved.
        
        Permission to use, copy, modify, and distribute this software and its documentation for educational, research and non-profit purposes, without fee, and without a written agreement is hereby granted, provided that the above copyright notice, this paragraph and the following three paragraphs appear in all copies.
        
        Permission to make commercial use of this software may be obtained by contacting:
        
        USC Stevens Center for Innovation
        
        University of Southern California
        
        1150 S. Olive Street, Suite 2300
        
        Los Angeles, CA 90115, USA
        
        Email: accounting@stevens.usc.edu
        
        This software program and documentation are copyrighted by The University of Southern California. The software program and documentation are supplied "as is", without any accompanying services from USC. USC does not warrant that the operation of the program will be uninterrupted or error-free. The end-user understands that the program was developed for research purposes and is advised not to rely exclusively on the program for any reason.
        
        IN NO EVENT SHALL THE UNIVERSITY OF SOUTHERN CALIFORNIA BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF THE UNIVERSITY OF SOUTHERN CALIFORNIA HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THE UNIVERSITY OF SOUTHERN CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS ON AN "AS IS" BASIS, AND THE UNIVERSITY OF SOUTHERN CALIFORNIA HAS NO OBLIGATIONS TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.
Classifier: Programming Language :: Python
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: Implementation :: CPython
Classifier: Programming Language :: Python :: Implementation :: PyPy
Requires-Python: >=3.9
Description-Content-Type: text/x-rst
License-File: LICENSE.rst
Requires-Dist: bleach!=5.0.0
Requires-Dist: cmake==3.30.3
Requires-Dist: dlib==19.24.6
Requires-Dist: gdown==5.2.0
Requires-Dist: importlib-metadata==8.4.0
Requires-Dist: importlib-resources==6.4.5
Requires-Dist: imutils==0.5.4
Requires-Dist: mediapipe==0.10.5
Requires-Dist: oauthlib>=3.0.0
Requires-Dist: opencv-python==4.10.0.84
Requires-Dist: pandas==2.0.3
Requires-Dist: pyyaml==6.0.2
Requires-Dist: requests==2.32.3
Requires-Dist: scikit-learn==1.3.2
Requires-Dist: timm==1.0.9
Requires-Dist: torch==2.0.0
Requires-Dist: torchaudio==2.0.1
Requires-Dist: torchvision==0.15.1
Requires-Dist: tqdm==4.66.5
Dynamic: author
Dynamic: author-email
Dynamic: classifier
Dynamic: description
Dynamic: description-content-type
Dynamic: home-page
Dynamic: license
Dynamic: license-file
Dynamic: requires-dist
Dynamic: requires-python
Dynamic: summary

=====================================
Libreface
=====================================

`GitHub Repo`_ - https://github.com/ihp-lab/LibreFace

.. _`GitHub Repo`: https://github.com/ihp-lab/LibreFace

|badge1| |badge2|


.. |badge1| image:: https://img.shields.io/badge/version-0.2.0-blue
   :alt: Static Badge


.. |badge2| image:: https://img.shields.io/badge/python-%3D%3D3.9-green
   :alt: Static Badge


This is the Python package for `LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis`_.
LibreFace is an open-source and comprehensive toolkit for accurate and real-time facial expression analysis with both CPU and GPU acceleration versions.
LibreFace eliminates the gap between cutting-edge research and an easy and free-to-use non-commercial toolbox. We propose to adaptively pre-train the vision encoders with various face datasets and then distill them to a lightweight ResNet-18 model in a feature-wise matching manner.
We conduct extensive experiments of pre-training and distillation to demonstrate that our proposed pipeline achieves comparable results to state-of-the-art works while maintaining real-time efficiency. LibreFace2 uses a lightweight RepVGG backbone for the AU detection, AU intensity, and facial expression heads.

**LibreFace2** additionally supports gaze estimation using a MediaPipe landmark-based MLP pipeline. Gaze yaw and pitch (in degrees) are returned alongside the existing facial attribute outputs.

.. _`LibreFace: An Open-Source Toolkit for Deep Facial Expression Analysis`: https://boese0601.github.io/libreface



Dependencies
============

- Python 3.9
- You should have `cmake` installed in your system.
    - **For Linux users** - :code:`sudo apt-get install cmake`. If you run into trouble, consider upgrading to the latest version (`instructions`_).
    - **For Mac users** - :code:`brew install cmake`.

.. _`instructions`: https://askubuntu.com/questions/355565/how-do-i-install-the-latest-version-of-cmake-from-the-command-line


Installation
============
You can first create a new Python 3.9 environment using `conda` and then install this package using `pip` from the PyPI hub:

.. code-block:: bash
    
    conda create -n libreface_env python=3.9
    conda activate libreface_env
    pip install --upgrade libreface


Usage
=====

Commandline
----------------

You can use this package through the command line using the following command.

.. code-block:: bash

    libreface --input_path="path/to/your_image_or_video"

Note that the above script would save results in a CSV at the default location - :code:`sample_results.csv`. If you want to specify your own path, use the :code:`--output_path`  command line argument,

.. code-block:: bash

    libreface --input_path="path/to/your_image_or_video" --output_path="path/to/save_results.csv"

To change the device used for inference, use the :code:`--device` command line argument,

.. code-block:: bash

    libreface --input_path="path/to/your_image_or_video" --device="cuda:0"

To save intermediate files, :code:`libreface` uses a temporary directory that defaults to ./tmp. To change the temporary directory path,

.. code-block:: bash

    libreface --input_path="path/to/your_image_or_video" --temp="your/temp/path"

For video inference, our code processes the frames of your video in batches. You can specify the batch size and the number of workers for data loading as follows,

.. code-block:: bash

    libreface --input_path="path/to/your_video" --batch_size=256 --num_workers=2 --device="cuda:0"

Note that by default, the :code:`--batch_size` argument is 256, and :code:`--num_workers` argument is 2. You can increase or decrease these values according to your machine's capacity.

**Examples**

Download a `sample image`_ from our GitHub repository. To get the facial attributes for this image and save to a CSV file, simply run,

.. _`sample image`: https://github.com/ihp-lab/LibreFace/blob/pypi_wrap/sample_disfa.png

.. code-block:: bash

    libreface --input_path="sample_disfa.png"

Download a `sample video`_ from our GitHub repository. To run the inference on this video using a GPU and save the results to :code:`my_custom_file.csv` run the following command,

.. _`sample video`: https://github.com/ihp-lab/LibreFace/blob/pypi_wrap/sample_disfa.avi

.. code-block:: bash
    
    libreface --input_path="sample_disfa.avi" --output_path="my_custom_file.csv" --device="cuda:0"

Note that for videos, each row in the saved CSV file corresponds to individual frames in the given video.

Python API
--------------

Here’s how to use this package in your Python scripts.

To assign the results to a Python variable,

.. code-block:: python

    import libreface 
    detected_attributes = libreface.get_facial_attributes(image_or_video_path)

To save the results to a csv file, 

.. code-block:: python

    import libreface 
    libreface.get_facial_attributes(image_or_video_path,
                                    output_save_path = "your_save_path.csv")

To change the device used for inference, use the :code:`device` parameter,

.. code-block:: python

    import libreface 
    libreface.get_facial_attributes(image_or_video_path,
                                    device = "cuda:0") # can be "cpu" or "cuda:0", "cuda:1", ...

To save intermediate files, libreface uses a temporary directory that defaults to :code:`./tmp`. To change the temporary directory path,

.. code-block:: python

    import libreface 
    libreface.get_facial_attributes(image_or_video_path,
                                    temp_dir = "your/temp/path")

For video inference, our code processes the frames of your video in batches. You can specify the batch size and the number of workers for data loading as follows, 

.. code-block:: python

    import libreface 
    libreface.get_facial_attributes(video_path,
                                    batch_size = 256,
                                    num_workers = 2)

Note that by default, the :code:`batch_size` is 256, and :code:`num_workers` is 2. You can increase or decrease these values according to your machine's capacity.

Gaze Estimation
----------------

Gaze yaw and pitch are returned by default in the :code:`get_facial_attributes` output (keys :code:`gaze_yaw` and :code:`gaze_pitch`, or columns of the same name for video).

.. image:: https://github.com/ihp-lab/LibreFace/blob/libreface2/main_code/media/gaze_bias_example.png?raw=true
   :alt: Gaze prediction example: biased (raw) vs unbiased (bias-corrected) vs ground truth on a Gaze360 frame.
   :width: 720px
   :align: center

The figure above shows a Gaze360 example with three panels: **biased (raw)** model output (red, :code:`yaw=+11.07°, pitch=+11.43°`), **unbiased (bias-corrected)** prediction (green, :code:`yaw=+4.51°, pitch=+6.44°`), and **ground truth** (blue, :code:`yaw=-7.68°, pitch=+2.61°`). Bias correction subtracts the systematic offset measured on the held-out test set (:code:`+6.55°` yaw, :code:`+5.05°` pitch).


If you only need gaze and want to skip AU and expression inference, call :code:`estimate_gaze` (image) or :code:`estimate_gaze_video` (list of aligned frames) directly. Both expect an **already aligned** face image — use :code:`libreface.get_aligned_image` to produce one from a raw image.

.. code-block:: python

    import libreface

    aligned_image_path, _, _ = libreface.get_aligned_image("path/to/your_image.png", temp_dir="./tmp")
    gaze = libreface.estimate_gaze(aligned_image_path, device="cpu")
    print(gaze)  # {'gaze_yaw': <float>, 'gaze_pitch': <float>}

For videos, pass a list of aligned frame paths:

.. code-block:: python

    import libreface

    gaze_df = libreface.estimate_gaze_video(aligned_frames_path_list,
                                            device="cuda:0",
                                            batch_size=256,
                                            num_workers=2)
    # gaze_df has columns "gaze_yaw" and "gaze_pitch", one row per frame.

Downloading Model Weights
================================

Weights of the model are automatically downloaded at :code:`./libreface_weights/` directory. If you want to download and save the weights to a separate directory, please specify the parent folder for weights as follows,

.. code-block:: python

    import libreface 
    libreface.get_facial_attributes(image_or_video_path,
                                    weights_download_dir = "your/directory/path")

Output Format
==================

For an image processed through LibreFace, we save the following information in the CSV file,

- :code:`lm_mp_idx_x`, :code:`lm_mp_idx_y`, :code:`lm_mp_idx_z` :  x, y, z co-ordinate of the 3D landmark indexed at :code:`idx` (total 478) obtained from mediapipe. Refer to the `mediapipe documentation`_ for getting the index to landmark map.

- :code:`pitch`, :code:`yaw`, :code:`roll` : contains the angles in degrees for the 3D head pose for the person.

- :code:`facial_expression` : contains the detected facial expression. Can be "Neutral", "Happiness", "Sadness", "Surprise", "Fear", "Disgust", "Anger", or "Contempt".

- :code:`au_idx` : contains the output of our action unit (AU) detection model, which predicts whether an action unit at index :code:`idx` is activated. 0 means not activated, and 1 means activated. We detect AU at the indices :code:`[1, 2, 4, 6, 7, 10, 12, 14, 15, 17, 23, 24]`.

- :code:`au_idx_intensity` : contains the output of our action unit (AU) intensity prediction model, which predicts the intensity of an action unit at index :code:`idx` between 0 and 5. 0 is least intensity and 5 is maximum intensity. We predict AU intensities for the AU indices :code:`[1, 2, 4, 5, 6, 9, 12, 15, 17, 20, 25, 26]`.

- :code:`gaze_yaw`, :code:`gaze_pitch` : estimated gaze angles in degrees, predicted by a MediaPipe landmark-based MLP. Yaw is the horizontal angle (left/right) and pitch is the vertical angle (up/down).

.. _`mediapipe documentation`: https://github.com/google-ai-edge/mediapipe/blob/7c28c5d58ffbcb72043cbe8c9cc32b40aaebac41/mediapipe/modules/face_geometry/data/canonical_face_model_uv_visualization.png

For a video, we save the same features for each frame in the video at index :code:`frame_idx` and timestamp :code:`frame_time_in_ms`.

Inference Speed
====================

LibreFace is able to process long-form videos at :code:`~30 FPS`, on a machine that has a :code:`13th Gen Intel Core i9-13900K` CPU and a :code:`NVIDIA GeForce RTX 3080` GPU. Please note that the default code runs on CPU and you have to use the :code:`device` parameter for Python or the :code:`--device` command line option to specify your GPU device ("cuda:0", "cuda:1", ...).

Contributing
============

We welcome contributions! Here’s how you can help:

1. Fork the GitHub repository_.
2. Create a new branch for your feature (`git checkout -b feature-branch`).
3. Commit your changes (`git commit -m 'Add some feature'`).
4. Push to the branch (`git push origin feature-branch`).
5. Create a Pull Request.

.. _repository: https://github.com/ihp-lab/LibreFace

If you have questions about this package, please direct them to achaubey@usc.edu or achaubey@ict.usc.edu

License
=======
Please refer to our github repo for License_

.. _license : https://github.com/ihp-lab/LibreFace/blob/main/LICENSE.txt
