CMake and distutils

Imagine you are writing a nice C library, together with some nice Python bindings and build this with CMake. Now, you would also like to deploy pure Python packages that use the nice bindings. You could just use the distutils module, hack a setup.py together and let the poor soul using your software figure out how to install that. However, there are some good reasons to let CMake build and install the package:

  • CMake knows if a Python interpreter is installed.
  • The setup.py script can be configured with central information like version numbers.
  • Each time the library changes, your package can be run and tested against it.
  • The poor soul can type sudo make install and forget everything else.

To avoid any misunderstandings, let’s assume the following project layout

.
|-- CMakeLists.txt
|-- src
|   |-- CMakeLists.txt
|   |-- foo.c
|   `-- foo.h
`-- python
    |-- CMakeLists.txt
    |-- foo
    |   `-- __init__.py
    `-- setup.py.in

and that ${PACKAGE_VERSION} is already defined in the top-level CMakeLists.txt. setup.py.in contains the usual mumbo jumbo spiced with variables that are replaced during the configure step:

from distutils.core import setup

setup(name='foo',
      version='${PACKAGE_VERSION}',
      package_dir={ '': '${CMAKE_CURRENT_SOURCE_DIR}' },
      packages=['foo'])

Because, CMake can build from any directory, setup.py will be generated nowhere near the package root directory. For this, we set the general package root dir via the package_dir parameter to ${CMAKE_CURRENT_SOURCE_DIR} (which is python/ in our example).

To build and install the Python package with CMake, python/CMakeLists.txt looks something like this:

find_program(PYTHON "python")

if (PYTHON)
    set(SETUP_PY_IN "${CMAKE_CURRENT_SOURCE_DIR}/setup.py.in")
    set(SETUP_PY    "${CMAKE_CURRENT_BINARY_DIR}/setup.py")
    set(DEPS        "${CMAKE_CURRENT_SOURCE_DIR}/module/__init__.py")
    set(OUTPUT      "${CMAKE_CURRENT_BINARY_DIR}/build/timestamp")

    configure_file(${SETUP_PY_IN} ${SETUP_PY})

    add_custom_command(OUTPUT ${OUTPUT}
                       COMMAND ${PYTHON} ${SETUP_PY} build
                       COMMAND ${CMAKE_COMMAND} -E touch ${OUTPUT}
                       DEPENDS ${DEPS})

    add_custom_target(target ALL DEPENDS ${OUTPUT})

    install(CODE "execute_process(COMMAND ${PYTHON} ${SETUP_PY} install)")
endif()

What’s so special about that? Well, the target that we add to the build system needs a way to tell when it’s time to run. For this, the add_custom_command specifies one or more outputs on which the target depends. But the output from python setup.py build is not a good candidate because it is placed in architecture- and platform-specific directories like lib.linux-x86_64-2.6. And you don’t want to test on that in a sane and portable way.

What we can do is to generate a file in the build root, each time our source dependencies change. The simplest way to do this, is to touch a stamp file and thus update its modification time. The dependency graph then looks like this:

<pre>+--------+ depends +-------+ depends +--------+ | Target |--------->| Stamp |--------->| Source | +--------+ +-------+ +--------+ </pre>

Last but not least, we install the package with the CODE form of the install command. install(CODE) is tied to the install target and expects a CMake command. For our purposes this is execute_process that in turn runs python setup.py install.


Discussion

nic

That looks neat, but do you have an example of how to set up the two other CMakeLists.txt files?…

Matthias

Sorry for the late reply, I’ve been on hiatus.

Well, the top-level CMakeLists.txt just drives the others by calling add_subdirectory and the one in src just building the library. That’s pretty a common way of using CMake.

Matěj

Thanks for the tutorial!

I build a C++ extension with the Extension class and I’ve found that it doesn’t care about the package_dir arg that’s required for the CMake out of source build. To find the C++ sources hardcoding the absolute path in the setup.py.in seems to work:

ext = Extension(‘ext’, sources = [’${CMAKE_CURRENT_SOURCE_DIR}/python/ext.cpp’], …) setup (name = ‘Ext’, description = ‘Sample extension’, ext_modules = [ext])

I also fixed the distutils installation to use CMake installation prefix:

install(CODE “execute_process(COMMAND ${PYTHON} ${SETUP_PY} install –prefix=${CMAKE_INSTALL_PREFIX})”)

Matthias

Thanks Matěj, I will update the post ASAP!

Stu

I’d like to get to a place where something can be installed from pypi, I think this might involve a bit of reshuffling ?

  • Basically it should build from setup.py, (unless on windows where there would be a .whl)
matthias

Well, if you want to install from PyPI you have to prepare the setup.py beforehand and trigger build of the library from there, i.e. exactly the other way around than presented here.