Python bindings: update & adding new libraries

Automated bindings: introduction

The bindings are generated automatically thanks to a sophisticated generator, which is based on srcML.

The generator in itself is located in the litgen module provided by srcmlcpp.

Installing the generator

Step 1: install srcML

Get and install srcML from its official site.

Step 2: install the generator and the required libraries

You will need to install the following python packages (see requirements-dev.txt):

srcmlcpp @ git+https://github.com/pthom/srcmlcpp
black
mypy
pytest
opencv-contrib-python

Quick information about the generator

litgen (aka "Literate Generator") is the package provided by srcmlcpp that will generate the python bindings.

Its source code is available here.

It is heavily configurable by a wide range of options.

Folders structure

In order to work on bindings, it is essential to understand the folders structure inside Dear ImGui Bundle. Please study the dedicated doc.

Study of a bound library generation

Let’s take the example of the library ImCoolBar.

Tip
The processing of adding a new library from scratch is documented in Adding a new library to the bindings. It uses ImCoolBar as an example

Here is how the generation works for the library. The library principal files are located in external/ImCoolBar:

external/ImCoolBar/                        # Root folder for ImCoolBar
├── ImCoolBar/                             # ImCoolBar submodule
│         ├── CMakeLists.txt               # ImCoolBar code
│         ├── ImCoolbar.cpp
│         ├── ImCoolbar.h
│         ├── LICENSE
│         └── README.md
└── bindings/                               # Scripts for the bindings generations & bindings
    ├── generate_imcoolbar.py               # This script reads ImCoolbar.h and generates:
    |                                       #     - binding C++ code in ./pybind_imcoolbar.cpp
    |                                       #     - stubs in
    |                                       #          bindings/imgui_bundle/im_cool_bar_pyi
    ├── im_cool_bar.pyi -> ../../../bindings/imgui_bundle/im_cool_bar.pyi   # this is a symlink!
    └── pybind_imcoolbar.cpp

The actual stubs are located here:

imgui_bundle/bindings/imgui_bundle/
├── im_cool_bar.pyi              # Location of the stubs
├── __init__.pyi                 # Main imgui_bundle stub file, which loads im_cool_bar.pyi
├── __init__.py                  # Main imgui_bundle python module which loads
|                                # the actual im_cool_bar module
├── ...

And the library is referenced in a global generation script:

imgui_bundle/external/bindings_generation/
├── autogenerate_all.py          # This script will call generate_imcoolbar.py (among many others)
├── all_external_libraries.py    # ImCoolBar is referenced here
├── ...

Update existing bindings

The process for updating bindings for a given library is straightforward:

  1. Update the library submodule in external/LIBNAME/LIBNAME

  2. Run the generation script in external/LIBNAME/generate_LIBNAME.py

  3. Compile and test python bindings (carefully study that nothing was broken)

  4. Commit and push

For example with ImCoolBar, in order to update the bindings for ImCoolBar, one needs to run:

python external/ImCoolBar/bindings/generate_imcoolbar.py
Tip
This video demonstrates from starts to finish the process of updating imgui and its bindings (17 minutes).

Adding a new library to the bindings

This example is based on the addition of ImCoolBar, which was added in Oct 2023.

Step 1: Reference the new library

Tip
All the modifications done in step 1 can be seen in this commit.
Step 1-a: Add needed folders, files and submodules inside external/

Add the library as a submodule in external/lib_name/lib_name

If the library can be included without adaptations for inclusion inside ImGui Bundle, you can add it directly as a submodule.

mkdir external/ImCoolBar
git submodule add https://github.com/aiekick/ImCoolBar.git external/ImCoolBar/ImCoolBar

However, if it requires adaptations, you need to create a fork (it was the case for ImCoolBar): So, the following actions were done separately:

  • ImCoolBar was cloned into github.com/pthom/ImCoolBar.git

  • a branch imgui_bundle was created and pushed to github. It will contain the adaptations and bug corrections for imgui_bundle.

Then, we add this fork as a submodule.

git submodule add https://github.com/pthom/ImCoolBar.git external/ImCoolBar/ImCoolBar
cd external/ImCoolBar/ImCoolBar
git checkout imgui_bundle
cd -

Create the folder external/lib_name/bindings/

Copy the folder external/bindings_generation/bindings_generator_template into external/lib_name/bindings/

cp -r external/bindings_generation/bindings_generator_template external/ImCoolBar/bindings

Rename files in external/lib_name/bindings

After having copied the template files, we need to rename them. In the example of ImCoolbar, we will rename them as follows:

 mv external/ImCoolBar/bindings/generate_LIBNAME.py external/ImCoolBar/bindings/generate_imcoolbar.py
mv external/ImCoolBar/bindings/pybind_LIBNAME.cpp external/ImCoolBar/bindings/pybind_imcoolbar.cpp
# im_cool_bar will be the final name of the python module: imgui_bundle.im_cool_bar
mv external/ImCoolBar/bindings/LIBNAME.pyi external/ImCoolBar/bindings/im_cool_bar.pyi

Move external/ImCoolBar/bindings/im_cool_bar.pyi to bindings/imgui_bundle/

The stub file (*.pyi) must be inside bindings/imgui_bundle. In order to facilitate development, we will create a symlink to it inside external/ImCoolBar/bindings/

mv external/ImCoolBar/bindings/im_cool_bar.pyi bindings/imgui_bundle/im_cool_bar.pyi
cd external/ImCoolBar/bindings/
ln -s ../../../bindings/imgui_bundle/im_cool_bar.pyi .
cd -

Final folder structure

We end up with the following structure:

external/ImCoolBar/
├── ImCoolBar/                   # Note that the submodule is inside
│         ├── CMakeLists.txt     # external/ImCoolBar/ImCoolBar/ !!!
│         ├── ImCoolbar.cpp
│         ├── ImCoolbar.h
│         ├── LICENSE
│         └── README.md
└── bindings/
    ├── im_cool_bar.pyi              # We will edit and rename those files later
    ├── generate_imcoolbar.py -> symlink to ../../../bindings/imgui_bundle/im_cool_bar.pyi
    └── pybind_imcoolbar.cpp

Step 1-b: Update python generator manager

Update external/bindings_generation/all_external_libraries.py

Add a function that returns info about this new library:

def lib_imcoolbar() -> ExternalLibrary:
    return ExternalLibrary(
        name="ImCoolBar",
        official_git_url="https://github.com/aiekick/ImCoolBar.git",
        official_branch="master",
        fork_git_url="https://github.com/pthom/ImCoolBar.git",
        fork_branch="imgui_bundle"
    )
ALL_LIBS = [
    lib_imgui(),  # must be first as it declare bindings used by the next ones
    # ...
    lib_imcoolbar(),  # Add the lib here
    # ...
Step 1-c: Update the C++ sources to include the new lib binding generation

In external/CMakeLists.txt: Add a cmake directive to compile the new library.

# If the library is "simple" to compile you can use `add_simple_external_library_with_sources`
add_simple_external_library_with_sources(imcoolbar ImCoolBar)

In external/bindings_generation/cpp/all_pybind_files.cmake:

add external/ImCoolBar/bindings/pybind_imcoolbar.cpp

Note
the script external/bindings_generation/autogenerate_all.py will also regenerate this file from scratch.

In external/bindings_generation/cpp/pybind_imgui_bundle.cpp:

Add the bindings

// ... Near the start of the file, add a new function declaration
void py_init_module_imgui_command_palette(py::module& m);
void py_init_module_implot_internal(py::module& m);
void py_init_module_imcoolbar(py::module& m);         // added this line
//  ...


void py_init_module_imgui_bundle(py::module& m)
{
    // ...

    // At the end of py_init_module_imgui_bundle, register your new python module
    auto module_imcooolbar = m.def_submodule("im_cool_bar"); // the python module will be known as imgui_bundle.im_cool_bar
    py_init_module_imcoolbar(module_imcooolbar);

Now, run cmake.

Step 1-d: Edit and adapt the generation scripts

Edit the 3 files inside external/ImCoolBar/bindings and replace occurrences of LIBNAME with appropriate values.

Step 1-e: Edit and adapt the imgui_bundle init scripts

In bindings/imgui_bundle/init.py, this line was added:

from imgui_bundle._imgui_bundle import im_cool_bar as im_cool_bar

In bindings/imgui_bundle/init.pyi, this line was added:

from . import im_cool_bar as im_cool_bar

Step 2: fine tune the generation options and write a demo

Tip
All the modifications done in step 2 can be seen in this commit.
Step 2-a: Edit and run external/ImCoolBar/bindings/generate_imcoolbar.py:

Edit and re-run it until the generated code fits the expected needs.

In the case of ImCoolBar, two simple changes were made:

def main():
    # ...
    # ...

    # Configure options
    options = litgen.LitgenOptions()
    options.namespace_root__regex = "ImGui"
    options.srcmlcpp_options.functions_api_prefixes = "IMGUI_API"

Each time you run the code generation, look at external/ImCoolBar/bindings/im_cool_bar.pyi and external/ImCoolBar/bindings/pybind_imcoolbar.cpp to see if they seem OK. Also run a compilation.

Step 2-b: Fix syntax issues in external/ImCoolBar/bindings/im_cool_bar.pyi:

You can add some code before the autogenerated code to fix the syntax issues. For example, this was added:


import enum

from imgui_bundle.imgui import ImVec2, WindowFlags, WindowFlags_
ImCoolBarFlags = int
ImGuiWindowFlags = WindowFlags
ImGuiWindowFlags_None = WindowFlags_.none


# !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!  AUTOGENERATED CODE !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
# <litgen_stub> // Autogenerated code below! Do not edit!
####################    <generated_from:ImCoolbar.h>    ####################
Step 2-c: Write a nice looking demo

It should demo the library, and act as a tutorial, in python and C++.