Update (October 4, 2023)

It's been a while since I wrote this and Poetry has become my Python package manager of choice due to pip missing several features I need.

I've added a section below on how the build process can be integegrated into a the Poetry project structure.

Introduction

PyBind11 is a fantastic header only library that allows you to integrate C++11 into Python. For example, lets say you have the following C++ source file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
#include <iostream>
#include <vector>

#include <pybind11/pybind11.h>
#include <pybind11/stl.h>

namespace py = pybind11;

std::vector<float> my_wrapper_function(std::vector<float> my_vector) {
  for (int i = 0; i < my_vector; ++i) {
    std::cout << my_vector[i] << ' ' << std::endl;
  }
  return my_vector;
}

PYBIND11_MODULE(my_wrapper, m) {
  m.def("my_wrapper_function", &my_wrapper_function);
}

You can can then compile it with the following flags:

1
> c++ -O3 -Wall -shared -std=c++11 -fPIC `python3 -m pybind11 --includes` my_wrapper.cpp -o example`python3-config --extension-suffix`

This produces a my_wrapper.cpython-36m-x86_64-linux-gnu.so, shared object
file, that you can import from a Python shell or script.

1
2
3
4
5
6
7
8
9
> python3.6
Python 3.6.3 (default, Oct  3 2017, 21:45:48)
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import my_wrapper
>>> result = my_wrapper.my_wrapper_function([0, 1, 2, 3, 4])
0 1 2 3 4
>>> result
[0, 1, 2, 3, 4]

Integration into the Standard Python Package Structure

When I began using PyBind11, I initially struggled to find a good way to import the shared object. One solution was to place make the wrapper it's own package and install it to the global or local site-packages.

However, in most cases I wanted to keep my project self-contained and have the wrapper in the same package as the rest of my sub-modules. The following is my solution to not using weird path manipulating import magic.

Lets say we have the following package layout:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
my_project_package/
  my_python_module/
    __init__.py
    my_module_script_0.py
    my_module_script_1.py
    my_python_submodule/
      __init__.py
      my_submodule_script_0.py
      my_submodule_script_1.py
    my_wrapper/
      __init__.py
      my_wrapper_source.cpp
  tests/
    test_0.py
    test_1.py
  images/
    image_0.png
  setup.py

To build the Python/C++ binding we can make our setup.py reflect the following:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
#!/usr/bin/env python3

from setuptools import setup, Extension, find_packages
import sysconfig
import glob
import os
import shutil


OUTPUT_FILE_NAME = 'my_wrapper'
PATH_TEMPLATE = '{}/{}{}'
SUFFIX = sysconfig.get_config_var('EXT_SUFFIX')


class get_pybind_include(object):
    """Helper class to determine the pybind11 include path

    The purpose of this class is to postpone importing pybind11
    until it is actually installed, so that the ``get_include()``
    method can be invoked. """

    def __init__(self, user=False):
        self.user = user

    def __str__(self):
        import pybind11
        return pybind11.get_include(self.user)


my_wrapper_extension = Extension(
    OUTPUT_FILE_NAME,
    sources=['my_wrapper/my_wrapper_source.cpp'],
    include_dirs=[
        get_pybind_include(),
        get_pybind_include(user=True)
    ],
    libraries=[],
    extra_compile_args=['-std=c++11'],
    language='c++'
)


setup(
    name='My Package',
    ext_modules=[my_wrapper_extension],
    packages=find_packages(),
    install_requires=[
        'pybind11'
    ]
)


def move_to_submodule():
    if os.path.isdir('build'):
        build_directory = glob.glob('build/lib*')[0]
        build_path = PATH_TEMPLATE.format(build_directory, OUTPUT_FILE_NAME, SUFFIX)

        if os.path.isfile(build_path):
            shutil.copy(build_path, 'my_python_module/my_wrapper/')
        else:
            print('Could not find shared object.')


move_to_submodule()

Without the move_to_submodule() method when you run the command python setup.py build, your wrapper will be placed in a directory called build in the top level of your package. So in my example, at this path: my_project_package/build.

This makes importing the shared object from within your package difficult because it is above the top level module. My solution to this is to copy the shared object to the same directory as the source and to make the wrapper a module by creating an __init__.py file. Both of the actions are reflected in the above directory listing and by the contents of setup.py.

Poetry

The pyproject.toml file:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
[tool.poetry]
name = "my_python_module"
version = "0.1.0"
description = ""
authors = ["drew_gunderson <drewgun@icloud.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"