Python binding generation tools

[bastien]

We were having a discussion about what tool we should use for the next Python binding: http://www.sfml-dev.org/forum/viewtopic.php?t=3451
Since this discussion is off-topic, I felt it would be better to create a new topic.

So I've played with Cython and made the hello world example work for 1.6, all the code is here with the example:https://bitbucket.org/bastienleonard/pysfml-cython/src

So basically it involves writing declarations for all the SFML methods and attributes, and then you write the wrapper classes in a language similar to Python. It should be pretty fast since it compiles to pure C++ code, and you can customize the wrappers in any way you want since you write them yourself. I guess the downside is that you need to write all this code, but it's fairly compact and I guess that fully automated generation wouldn't allow to tweak the end result as easily.

[Laurent]

Looks good. I like the idea of being able to fully customize the resulting API.

Have you made some performance tests?

[bastien]

Looks good. I like the idea of being able to fully customize the resulting API.

Have you made some performance tests?

No, I can't display anything except text currently, and I'd rather let Tank test it first, since he also tested SWIG and others.

[Tank]

Interesting, looks very good so far. Didn't know Cython (god, it's hard as hell to write that, or is it only me?) is able to do such things.

However there's one thing I'm skeptical about, which isn't really related to Cython but a Python naming convention that's being discussed about a lot: Properties. The claim of many Python programmers is that public attributes and getters and setters are pure evil. Public attributes mostly are evil because of side-effects or wrong usage, but getters and setters are only annoying to write and bloat up the code. Python's properties circumvent that "problem" by giving the developer the possibility to firstly create public properties and encapsulate those into getters and setters later without breaking the public API.

To be honest, I think that's a very bad idea, since assignments are treated similar in most programming languages: When I do "a = 5", I expect that 5 is in a, and when I do "person.firstname = 'john'", I expect the same. But a developer may decide that all firstnames must be capitalized, so they throw in a property and define a setter which capitalizes the new value. So when I do "person.firstname = 'john'", it ends up being "John". In my opinion this can lead to critical and hard to find bugs and may also cause confusion.

To summarize: At a first glance Cython looks very promising and I think we should prepare a binding for SFML 1.6 that's able to open an sf::RenderWindow, load an sf::Image to use in an sf::Sprite to be able to benchmark and compare it with the other two working options (Python C API and Boost.Python). On the other hand we need to discuss about conventions.

Thanks for pointing it out, bastien.


Edit: Ah, I forgot to answer the SWIG thing: My SWIG binding builds but is NOT working because of a threading issue I can't really localize. Honestly I'd prefer other code generators over SWIG if we decided to use one to purely wrap C++ classes and methods (for example with pybindgen). But I think Cython is a cool and flexible approach here, we just have to make sure that speed won't be an issue; unfortunately this has been a show-stopper for at least one great option, namely Boost.Python.
Edit²: Here are my pybindgen results so far: https://github.com/TankOs/SFML2/tree/pybindgen . Keep in mind it's mostly a wrapper generator, not as flexible as Cython. I put it in here just for the case of interest.

[bastien]

About getters/setters:
A public attribute isn't wrong, because you can make it a property later if needed. But in our case we can't use them anyway, since we want to map attribute access to methods calls.

Personally I became a huge fan of properties after using GUI toolkits with code like:

Code: [Select]

window.title = 'Title'
window.width = 640

instead of:

Code: [Select]

window.set_title('Title')
window.set_width(640)

Here you really don't care whether title and width are real attributes or their access needs to execute some code, you precisely use a GUI toolkit to abstract this kind of stuff and their retrieval is probably nearly as costly as a direct attribute access.

So in my opinion we should use properties every time the user doesn't care how the value is used internally and simply wants to able to get and set it. Normally all SFML get/set methods are used in this way.
We should avoid creating properties that a) have a non-obvious behavior or b) have an expensive behavior (because the user expects attribute access to be fast).

It does raise some questions though. For example, sf::RenderWindow has getWidth(), getHeight() and setSize(). It seems inconsistent to me, but it didn't seem to be a problem in C++.
However, in Python, as long as width and height are properties, I feel I should write setters as well as it's much more intuitive. And width should probably be available as well since the C++ documentation will probably be used by most PySFML users and they will expect an equivalent for it. But it's totally redundant, and it's not a property. (Although we could make it a property if we pass a single argument, i.e. a tuple.)

[Tank]

Like I said, most developers expect a copy only when writing an assignment. I would not expect that an assignment does anything else than assigning value a to value b, and Python properties may do it differently. There're even cases when it's dangerous, because assignments (better: the underlying setter of the property) may throw an exception and I bet not many users would expect that.

What happens when I do the following?

Code: [Select]

window.width = -1

The pure possibility to write such code propagates that it's possible to directly modify member data, however this is not true because internally the assignment is redirected to a setter method. But how would I, as the user, know that? When using literals like above, I may recognize the dumb stuff I wrote , but you don't always have literals, so the assignment may throw an exception, for example, because width = -1 is invalid. Personally I would not surround assignments with exception handling, it's just weird.

Another point is one you already mentioned: SFML API compliance. I fully agree with that a Python binding for any library should be as much Pythonish as possible, but I've never touched properties often in the past and I guess it's not a very well-known and used convention. More importantly things like Python lists, tuples, dicts, yielding etc. should be used to accomplish a Python-like binding style. However, in my opinion the best solution would be to stick to the original SFML API as close as possible, without moving too much to the C++ side.

[bastien]

What happens when I do the following?

Code: [Select]

window.width = -1

Why would you do this if -1 is invalid? Python isn't a language where you try to forbid your users from doing anything potentially wrong, e.g. there is no private attribute. It's part of the design of the language.

The pure possibility to write such code propagates that it's possible to directly modify member data, however this is not true because internally the assignment is redirected to a setter method.

As I said, setters should be semantically similar to attribute writing, i.e. it should just quickly change the state of the object without doing anything long or non-obvious.

But how would I, as the user, know that? When using literals like above, I may recognize the dumb stuff I wrote , but you don't always have literals, so the assignment may throw an exception, for example, because width = -1 is invalid. Personally I would not surround assignments with exception handling, it's just weird.

How is that different from set_width()? In both cases you can get problems by setting invalid values. What would help you in this case is that the API throws an exception with a helpful message.

Another point is one you already mentioned: SFML API compliance. I fully agree with that a Python binding for any library should be as much Pythonish as possible, but I've never touched properties often in the past and I guess it's not a very well-known and used convention. More importantly things like Python lists, tuples, dicts, yielding etc. should be used to accomplish a Python-like binding style. However, in my opinion the best solution would be to stick to the original SFML API as close as possible, without moving too much to the C++ side.

It's not used everywhere because it's quite recent (2.5 I think). For example, the threading module uses properties, although it uses Java conventions (which don't include properties, since Java doesn't support them). wxPython and PyGTK also use properties.

[bastien]

Now supports basic sprites:


By the way, I'm not sure what is best for constants like sf::Style::Fullscreen.
Currently I just define top-level values like STYLE_FULLSCREEN. It could be placed inside a dummy class sf.Style as well.

[Kaoron]

But how would I, as the user, know that?

Try. Fail. Accept the new convention. . Profit.

Edit : One thing I've been stuck onto with cython is wrapping existing C++ instances (like one returned by a function/method). bastien, did you try to implement a method such as VideoMode::GetMode ?

Edit : Oh, and - care : your indentation is done with 5 spaces instead of 4. Baaad.

[bastien]

Edit : One thing I've been stuck onto with cython is wrapping existing C++ instances (like one returned by a function/method). bastien, did you try to implement a method such as VideoMode::GetMode ?

I guess an ugly way to do it would be to create a dummy Python instance, delete its C++ instance and replace it with the one returned by GetMode(). There's probably a better way, but I don't know enough about the C API and how Cython interacts with it to do something more clever.
But currently I can't get to call GetMode(), I thought I'd declare it this way, since static methods apparently aren't directly supported:

Code: [Select]

cdef extern from "SFML/Graphics.hpp" namespace "sf::VideoMode":
    VideoMode GetMode(unsigned long Index)

But I get “Function argument cannot have C name specification”.

Edit : Oh, and - care : your indentation is done with 5 spaces instead of 4. Baaad.

Mmh yeah, weird. I don't know why all python-mode decides to use 5 spaces with these files.






I just tried a simple benchmark:

Code: [Select]

#! /usr/bin/env python2
# -*- coding: utf-8 -*-

import sf


def main():
    window = sf.RenderWindow(sf.VideoMode(640, 480), 'Benchmark')
    image = sf.Image()
    image.load_from_file('python-logo.png')
    sprite = sf.Sprite()
    sprite.image = image
    running = True
    event = sf.Event()

    for i in xrange(100):
        window.clear()
        window.draw(sprite)
        window.display()

    window.close()


if __name__ == '__main__':
    main()

Cython:

Code: [Select]

$ time ./benchnew.py

real 0m2.236s
user 0m0.413s
sys 0m0.087s

Current binding:

Code: [Select]

$ time ./benchold.py

real 0m2.219s
user 0m0.423s
sys 0m0.087s

C++ (GCC, -O3):

Code: [Select]

$ time ./essai

real 0m1.908s
user 0m0.120s
sys 0m0.060s

Apparently the program spends more time waiting than anything else, so I'm not sure it means anything. But it seems to suggest that PySFML itself wouldn't be a bottleneck.

[Laurent]

Guys, I think you should have a look at how I write the .Net binding. I answered all the questions that you're currently asking, such as using properties and returning instances -- maybe some of my solutions are not the best, but maybe it can help you.

[bastien]

Apparently there's a bug when calling C++ static methods: http://groups.google.com/group/cython-users/browse_thread/thread/78dda025a24a651b

That's why I can't call GetMode(). Heh, I've never used it anyway.

Guys, I think you should have a look at how I write the .Net binding. I answered all the questions that you're currently asking, such as using properties and returning instances -- maybe some of my solutions are not the best, but maybe it can help you.

Python doesn't directly support method overloading, so providing multiple constructors is messy (it only clutters the binding itself though, so it can still be considered).
Personally I really prefer factory methods, but then we must decide if there should still be a default constructor for some reason.

[bastien]

Well wouldn't it be much easier to bind CSFML actually?

[Laurent]

Python doesn't directly support method overloading, so providing multiple constructors is messy (it only clutters the binding itself though, so it can still be considered).

Do you mean that we could write specific code in the binding to work around that? Can you show me an example?

Well wouldn't it be much easier to bind CSFML actually?

I don't know if binding a C API is easier than a C++ one, but it would at least have one drawback: SFML can be linked statically, whereas CSFML must be linked dynamically. This doesn't matter on Unix systems, but on Windows it's not a bad thing to have the binding as a self-contained binary.

[bastien]

Python doesn't directly support method overloading, so providing multiple constructors is messy (it only clutters the binding itself though, so it can still be considered).

Do you mean that we could write specific code in the binding to work around that? Can you show me an example?

We can inspect the number of arguments and their types at runtime and decide what to do based on that.
In pure Python it looks something like that, should be pretty similar in Cython:

Code: [Select]

class Image(object):
    def __init__(self, width=None, height=None, color=None, pixels=None, filename=None, memory_file=None):
        # Check that not too many parameters have been supplied

        if width is not None and height is not None:
            # ...
        elif pixels is not None:
            # ...
        # And so on

And you would use it like this:

Code: [Select]

Image(640, 480, sf.Color(0, 0, 0))
Image(pixels=[])
Image(filename='filename')

You could also not require keywords arguments and deduce what to do based on their types, but then it gets even more complicated for no reason in my opinion.
On the other hand static methods would have an obvious meaning.