Embedding Python with Boost.Python Part 1

Embedding Python with Boost.Python Part 1
by Howard "SiCrane" Jeng

Building Applications that Embed Python

Python is an increasingly popular programming language used in a variety of contexts. Python has been called a language optimized for development speed. This puts it in contrast with compiled languages like C and C++, which can be called languages optimized for execution speed.

This contrast between Python and C/C++ often fuels a development methodology in Python circles: code the application in Python, profile the application, and rewrite the performance critical parts in C or C++. This makes the topic of developing hybrid systems by extending Python very well covered in various Python documentation. Less well covered is the situation where you have a C++ program where you want to embed Python. Python's ease of use make it well suited in game development where the ability to quickly modify code is more important than execution speed. This includes areas such as artificial intelligence, event scripting or level design.

This article series covers embedding Python in a C++ application with Boost.Python, one of the Boost project's many useful libraries. This particular article focuses on the basics of embedding, specifically on the process of building applications that embed Python.

Embedding Python with Boost.Python Part 2 focuses more on the interface between Python and C++ code, including an extensive example of embedding based on an SDL application where the majority of the core logic is done in Python. The last article shows the flexibility of Python in an embedded context by further extending the application presented in the second article, with what may be considered "bells and whistles", including incorporating an interactive Python interpreter in the application.

This article series assumes familiarity with the basics of Python and intermediate knowledge of C++.1) Also, the series relies on Boost.Python which in turn requires Python 2.2 or higher. The code in these articles have been tested with Boost 1.32.0, 1.33.0 and 1.33.1 as well as Python 2.3 and 2.4.

The First Embedded Application

The absolute minimal application that you can create that can claim to have embedded Python looks like:2)

#include <Python/Python.h>int main(int, char **) {  Py_Initialize();    Py_Finalize();  return 0;}

All this program does is initialize Python and then cleans it up. However, even a program as simple as this can be instructive. For one thing, unless you've already set up Python for your C++ development environment, it shouldn't compile or link.

In order to get this program to build you need to have Python installed on your computer. Your computer may already have Python installed. For example, recent versions of Mac OS X ship with Python as a framework. If not, there are pre-built binaries for Windows and Mac available from http://www.python.org, and source archives available for other operating systems. The source archive builds and installs much like you would normally expect for a automake project:./configure ; make ; make install. Among the outputs is the libpythonx.y.a archive (where x.y is the version number, e.g. libpython2.4.a), which is required for linking against for Python extending or embedding. It is put in the root directory of the source archive. 3)

Once you have Python installed, you still need to get the headers recognized by your compiler. I put the headers in a path so that they reside in thePython/ virtual directory, which is also how the Python framework for Mac sets things up by default. This is in contrast with normal Python extending/embedding sources, which assumes that the Python headers are put in a place that you can access without a directory specification (i.e. #include <Python.h> instead of#include <Python/Python.h>). For the purposes of this article the difference in directory structure is minor, since for the most part we will be includingboost/python.hpp instead of the Python.h header directly. You will also need to add the Python lib directory to your linker path or otherwise directly add the Python library to your project (eitherpythonxy.lib, libpythonx.y.a or adding the Python framework in OS X).

Also, in debug mode, with a default Python installation, the program will not link in MSVC. This is because the Python MSI installer comes withpythonxy.lib, but not the pythonxy_d.lib requested as an input file to the linker via a header pragma. You can get around this by includingboost/python.hpp instead of Python.h. boost/python.hpp does some preprocessor magic to change the pragma to link the pythonxy.lib instead. However, don't include both. That may cause your program to try to link against both pythonxy.lib and pythonxy_d.lib. 4)

One final thing to note is that with older Python versions, on Mac platforms you needed to usePyMac_Initialize() instead of Py_Initialize(). This should only matter if you are using Python 2.1 or earlier. However, this article covers Boost.Python, which requires Python 2.2 or later, so this should be a non-issue.

Once you can get it to compile and link, try running the program. Hopefully it will do what it's supposed to do: have no apparent effect.5)

An Embedded Program That Does Something

Now let's modify the program to actually do something. Tradition dictates that the next step be Hello World. So let's create a Python file,hello.py with the contents:

def hello():  print "Hello World!"

Then modify our main() function to:

#include <Python/Python.h>int main(int, char **) {  Py_Initialize();  PyRun_SimpleString("import hello");  PyRun_SimpleString("hello.hello()");    Py_Finalize();  return 0;}

This does more or less what you would expect if you ran the two statements in Python.6) PyRun_SimpleString() can be used to run a variety of Python code; however, it has several limitations. First off,PyRun_SimpleString() can only be used to run statements, not evaluate expressions. It also doesn't communicate its result to the calling code. In order to adjust the inputs to the function, you need to alter the C string used to callPyRun_SimpleString(), which can be cumbersome. Finally, the Python code is always run in the namespace of module__main__.

For this Hello World program these limitations are not a big issue. Now let's look at something where we want to extract a value from Python. First we need to put a value into Python in order to extract it. So first we changePyRun_SimpleString() to take an assignment statement. Then in order to get at the assigned variable, we use the low level Python API:

int main(int, char **) {  Py_Initialize();    PyRun_SimpleString("result = 5 ** 2");    PyObject * module = PyImport_AddModule("__main__"); // borrowed reference  assert(module);                                     // __main__ should always exist  PyObject * dictionary = PyModule_GetDict(module);   // borrowed reference  assert(dictionary);                                 // __main__ should have a dictionary  PyObject * result    = PyDict_GetItemString(dictionary, "result");     // borrowed reference  assert(result);                                     // just added result  assert(PyInt_Check(result));                        // result should be an integer  long result_value = PyInt_AS_LONG(result);          // already checked that it is an int    std::cout << result_value << std::endl;    Py_Finalize();  return 0;}

The first thing you might notice is the strange comments about borrowed references. In the Python C API, most functions returnPyObject pointers, which are reference counted. These pointers can either be new references or borrowed references. A new references means that the Python runtime has already incremented the reference count for you before returning thePyObject pointer. When you receive a new reference from the API you need to eventually callPy_DECREF() or Py_XDECREF() to decrement the reference count after you are finished with it. For borrowed references you don't need to decrement the reference count; however, this also means that since the reference count hasn't been incremented, the Python runtime may garbage collect the object while you still have a pointer to it. For these borrowed references, the objects should last all the way until thePy_Finalize() call, so incrementing the reference count is not necessary.

The actual function calls are a bit more straightforward. It's equivalent to the Python code:

result = 5 ** 2import __main____main__.__dict__["result"]

With the result then being transferred to the C++ program to be outputted. Now let's try the other half of the equation: setting a variable in Python from the C++ program. We'll just add a few lines between thecout and Py_Finalize() call.

  std::cout << result_value << std::endl;    PyObject * twenty = PyInt_FromLong(20);             // new reference  PyDict_SetItemString(dictionary, "result", twenty);  Py_DECREF(twenty);                                  // release reference count  PyRun_SimpleString("print result");    Py_Finalize();

This creates a new Python integer object with the value of 20 and assigns it to the dictionary.7) In this case instead of getting a borrowed reference from Python, we get a new reference that we need to track, so we callPy_DECREF() after finishing with the object.

Introducing Boost.Python

At this point, all the uses of raw pointers as well as explicit management of reference counts may be triggering instincts of wrappingPyObject pointers with smart pointers. Fortunately, Boost.Python does that for you as well as wrapping some of the API calls so that you don't need to deal with the low level API directly as much.

Unfortunately, Boost.Python still seems mostly geared toward writing Python extensions rather than embedding, so we still need to use some API calls directly. A direct translation of the previous program looks like:

#include <boost/python.hpp>#include <iostream>int main(int, char **) {  using namespace boost::python;  Py_Initialize();    try {    PyRun_SimpleString("result = 5 ** 2");        object module(handle<>(borrowed(PyImport_AddModule("__main__"))));    object dictionary = module.attr("__dict__");    object result = dictionary["result"];    int result_value = extract<int>(result);        std::cout << result_value << std::endl;        dictionary["result"] = 20;    PyRun_SimpleString("print result");  } catch (error_already_set) {    PyErr_Print();  }    Py_Finalize();  return 0;}

The call to PyImport_AddModule() becomes slightly more complex, as what was originally a comment about the borrowed reference becomes part of the code itself. However, once we have a reference to the__main__ module, instead of using an API call, we use the attr() member function to get at the__dict__ attribute of the __main__ module, which more closely resembles the equivalent Python code. Similarly, getting the reference to theresult variable is done with operator[], and we can get at the value with theextract<>() function, whose usage resembles a cast.

Also added is a slightly better error handling scheme than before. Instead of assuming that everything works properly, now it traps Python exceptions inside a try/catch block.

Now the challenge is to get this to build. For directions on how to get Boost.Python built with Boost.Jam go tohttp://www.boost.org/more/getting_started.html. If you can't get that to work, I've included an appendix to this article about building a project with Boost.Python without bjam. Even if you can get bjam to work, there are a couple of notes for building your project against the Boost.Python library. Firstly, exception handling and RTTI are required to be enabled for Boost.Python to function properly. Secondly, under MSVC, you should build your project with the Multithread DLL versions of the C runtime library or you can get some very subtle and strange runtime errors.

Hopefully once you get this built and running, it will display 25 and 20 as expected.

Creating a Module

The last topic for this article is creating a module in the application that can be imported from the embedded Python interpreter. To define a Python module with Boost.Python you declare the module with theBOOST_PYTHON_MODULE() macro, and put the various initializers inside the curly braces that follow it. The next article deals more with the initializers and how to export classes. Right now just focus on the one commented line in the following source.

#include <boost/python.hpp>using namespace boost::python;int add_five(int x) {  return x + 5;}BOOST_PYTHON_MODULE(Pointless){    def("add_five", add_five);}int main(int, char **) {  Py_Initialize();     try {    initPointless(); // initialize Pointless      PyRun_SimpleString("import Pointless");    PyRun_SimpleString("print Pointless.add_five(4)");  } catch (error_already_set) {    PyErr_Print();  }    Py_Finalize();  return 0;}

This program shouldn't require any new measures to build or run properly. The important detail here is that in order for the module to be properly registered with the embedded Python interpreter, theinit function for the module needs to be called. The function is calledinit followed by the module name. In this case it is initPointless(). So if you declare a module withBOOST_PYTHON_MODULE(Foo) you should call initFoo() sometime afterPy_Initialize(), but before being potentially imported. 8)

Wrapping Up

This should cover all the minor details required to successfully build an application with an embedded Python interpreter with and without Boost.Python. In the next article I cover more details on how to perform actual integration between Python and C++.

References

• Extending and Embedding the Python Interpreter
• Python/C API Reference Manual
• Boost.Python
• Getting Started [with Boost]

Appendix

Setting up embedded Boost.Python projects if you can't (or don't want to) get Boost.Jam to work.

While later versions of Boost have been very good about installing properly with bjam, sometimes it's hard to get it to work right if you don't know how to set up the right environment variables, or if you have an otherwise eccentric build system. Alternately, you may not want to dynamically link against the Boost.Python library. For those cases, here are a couple of examples of setting up projects without using Boost.Jam to put together an executable that statically links against Boost.Python. I've chosen the two IDEs I've seen people have the most trouble getting bjam to work with: MSVC .NET 2003 for Windows and XCode for Max OS X.

These instructions have been tested with both Boost 1.32, 1.33.0 and 1.33.1 with Python 2.3 and 2.4, though it should work with Python 2.2.

In Windows with MSVC .NET 2003

The Windows environment presents special challenges in getting Boost.Python to build if you can't get bjam to work. The primary problem is that Boost.Python on Windows is designed to be built as a dynamic link library, so simply including the Boost.Python source files in your project isn't sufficient, you also need to sprinkle in the correct pre-processor macros.

The first step is to download Boost and Python. Boost just needs to be unpacked. For Windows, there is an MSI installer for Python that contains the necessary headers and link libraries for use with MSVC. On my computer I put theboost_1_32_0/ directory in C:\include, and the C header files from theinclude/ directory of the Python installation in the C:\include\Python directory.

Then create an empty console project in MSVC, and add the boost_1_32_0/ directory and the directory containing the Python header directory to your include path in project settings. Next, add some Boost.Python test code as a main.cpp file:

#include <boost/python.hpp>#include <iostream>int main(int, char **) {  using namespace boost::python;  Py_Initialize();    try {    PyRun_SimpleString("result = 5 ** 2");        object module(handle<>(borrowed(PyImport_AddModule("__main__"))));    object dictionary = module.attr("__dict__");    object result = dictionary["result"];    int result_value = extract<int>(result);        std::cout << result_value << std::endl;        dictionary["result"] = 20;    PyRun_SimpleString("print result");  } catch (error_already_set) {    PyErr_Print();  }    Py_Finalize();  return 0;}

Next, you need to add BOOST_PYTHON_STATIC_LIB to the list of preprocessor defines in the project settings. This prevents Boost.Python from defining functions as__declspec(dllimport) or __declspec(dllexport). Don't forget to do this for both Debug and Release mode. Also Boost.Python need RTTI enabled to function properly, so turn that on, and switch to the multi-threaded DLL versions of the C runtime library. You can get very strange and annoying runtime errors if you neglect either of those steps.

If you follow the same structure for your include directory hierarchy you then get a number of unable to open header errors. You just need to prefixPython/ to the header file names in the problem files. So #include <patchlevel.h> becomes#include <Python/patchlevel.h>. Alternately you can add the actual Python header directory to your include path and not follow the way I do things.

While we're messing with the Boost files, you should uncomment the #define BOOST_ALL_NO_LIB line inboost/config/user.hpp to tell Boost that we aren't trying to use the bjam-built DLLs.

Now add all the files in the boost_1_32_0/libs/python/src directory to your project. You may get a few more of the unable to open header errors, but that has the same fix as before.

After all that, you can run the program and it should produce the number 25 and the number 20 on the console.

As a side note, on MSVC it is important not to include Python.h directly. Doing this in debug mode may cause the linker to try to load a version of the Python link library that doesn't come with the Python MSI installation package. Even if you have the Python source and have built both versions of the Python link library, it may cause your program to try link against both versions of the Python library simultaneously, which will cause no end of unhappy linker messages.

In OS X with XCode

Building an application that embeds Python for Mac OS X with Boost.Python is slightly different. In OS X 10.4 (Tiger), Python 2.3 comes as a Framework. The first step is to actually download the boost library and uncompress and untar the archive.

In XCode create a new project as a "C++ Command Line Tool". Now add some Boost.Python test code to the main.cpp file:

#include <boost/python.hpp>#include <iostream>int main(int, char **) {  using namespace boost::python;  Py_Initialize();    try {    PyRun_SimpleString("result = 5 ** 2");        object module(handle<>(borrowed(PyImport_AddModule("__main__"))));    object dictionary = module.attr("__dict__");    object result = dictionary["result"];    int result_value = extract<int>(result);        std::cout << result_value << std::endl;        dictionary["result"] = 20;    PyRun_SimpleString("print result");  } catch (error_already_set) {    PyErr_Print();  }    Py_Finalize();  return 0;}

Now go to the project window and Add/Existing Framework/Python.Framework. Then in source files, add a new group, call it something like "Boost Python Sources" and thenAdd / Existing Files / boost_1_32_0/libs/python/src, highlight all the files and the folders in the directory and hitAdd. Select the radio button for "Recursively create groups for added folders" and hitAdd again.

Now go to Project Settings and add boost_1_32_0/ to the Header Search Paths under Styles. (At this point I also disable ZeroLink, but that's a personal preference.)

At this point it should give you several build errors in the boost headers. In order to build properly when using the Python framework, the boost headers that give errors should have #includes that reference python headers have thePython/ prefix appended to the name. So #include <patchlevel.h> becomes#include <Python/patchlevel.h>. Alternately you can add the directory for the Python frameworks header file to the list of Header Search Paths under Styles in the Project Settings.

Either way, once that it is finished you should be able to build the project, and 25 and 20 show up in the log window as expected.

As a note, the project should also have RTTI enabled, which XCode should do by default.

Of course, you can use a non-framework version of Python as well. To do so, I suggest building Python from source as normal../configure ; make ; make install will build and install Python to your computer. To build an embedded Python project instead of adding the Python Framework, add thelibpythonx.y.a (ex: libpython2.4.a) from the root of the Python source directory to your project instead of the Python Framework. You may also need to add the/usr/local/include directory to your header search path, and you may need to add a symlink to thepython2.4/ directory to Python/ to get the headers to be found. Or just add/usr/local/include/python2.4 to your header search path.

---

1) For those unfamiliar with Python and reading this article to evaluate embedding Python as a scripting language, I recommendDive into Python as an introduction to the Python programming language.

2) Technically you can also leave off the Py_Finalize(), though that would be bad manners.

3) Unless you chose a different method when invoking theconfigure script.

4) The preprocessor contortions that Boost.Python uses to link to the python library in debug mode also prevents it from compiling on MSVC .NET 2005 with Boost 1.32 or 1.33.0. This seems to have been fixed with Boost 1.33.1, and release mode builds will also work with Boost 1.32 or 1.33.0.

5) As opposed to generating a runtime assertion, dumping core or otherwise generating a runtime error.

6) This includes compiling the code and generating the hello.pyc file.

7) In actuality, the current C implementation of the Python interpreter pre-creates integer objects for the range of -1 to 100. So if you create an integer value in that range you get a reference to one of the pre-existing integer objects. The Python/C API mentions that it would be theoretically possible to change the value of 20 via this returned object; the resulting behavior being undefined.

8) This seems to be one of the most frequently encountered problems with embedding Python with Boost.