Metaprogramming in Python

来源：互联网发布：网络侵权案例编辑：程序博客网时间：2024/06/05 03:36

Metaprogramming

Metaprogramming is the ability for a program to reason about itself or to modify.

Metaprogramming is the writing of computer programs that write or manipulate other programs (or themselves) as their data, or that do part of the work at compile time that would otherwise be done at runtime.
– Definition from the Wikipedia, emphasis added by me.“

Metaprogramming is about creating functions and classes whose main goal is to manipulate code (e.g., modifying, generating, or wrapping existing code). The main features for this include decorators, class decorators, and metaclasses. However, a variety of other useful topics—including signature objects, execution of code with exec(), and inspecting the internals of functions and classes . The main purpose of this article is to explore various metaprogramming techniques and to give examples of how they can be used to customize the behavior of Python to your own needs.

Objects are created by other objects: special objects called “classes” that we can set up to spit out objects that are configured to our liking.

Classes are just objects, and they can be modified the same way:

1
2
3
4
5
6
7
8
9
10
11
12
>>>classFoo:pass
...
>>>Foo.field=42
>>>x=Foo()
>>>x.field
42
>>>Foo.field2=99
>>>x.field2
99
>>>Foo.method=lambdaself:"Hi!"
>>>x.method()
'Hi!'

To modify a class, you perform operations on it like any other object. You can add and subtract fields and methods, for example. The difference is that any change you make to a class affects all the objects of that class, even the ones that have already been instantiated.

What creates these special “class” objects? Other special objects, called metaclasses.

The default metaclass is called type and in the vast majority of cases it does the right thing. In some situations, however, you can gain leverage by modifying the way that classes are produced – typically by performing extra actions or injecting code. When this is the case, you can use metaclass programming to modify the way that some of your class objects are created.

It’s worth re-emphasizing that in the vast majority of cases, you don’t need metaclasses, because it’s a fascinating toy and the temptation to use it everywhere can be overwhelming. Some of the examples in this article will show both metaclass and non-metaclass solutions to a problem, so you can see that there’s usually another (often simpler) approach.

Some of the functionality that was previously only available with metaclasses is now available in a simpler form using class decorators. It is still useful, however, to understand metaclasses, and certain results can still be achieved only through metaclass programming.

Why Would You Care?

Extensively used in frameworks and libraries.
Better understanding of how Python works.
It solves a practical problem

Basic Metaprogramming

So metaclasses create classes, and classes create instances. Normally when we write a class, the default metaclass type is automatically invoked to create that class, and we aren’t even aware that it’s happening.

It’s possible to explicitly code the metaclass’ creation of a class. type called with one argument produces the type information of an existing class; type called with three arguments creates a new class object. The arguments when invoking type are the name of the class, a list of base classes, and a dictionary giving the namespace for the class (all the fields and methods). So the equivalent of:

1
classC:pass

is:

1
C=type('C',(),{})

Classes are often referred to as “types,” so this reads fairly sensibly: you’re calling a function that creates a new type based on its arguments.

We can also add base classes, fields and methods:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
defHello(self,you):
    print("Hello, "+you)
 
lst=type('lst',(list,),dict(x=42,Hello=Hello))
 
ml=lst()
ml.append("John")
print(ml)
print(ml.x)
ml.Hello("Doe")
 
print(ml.__class__.__class__)
 
#['John']
#42
#Hello, Doe
#<class 'type'>

Note that printing the class of the class produces the metaclass.

The ability to generate classes programmatically using type opens up some interesting possibilities.

The Metaclass Hook

So far, we’ve only used the type metaclass directly. Metaclass programming involves hooking our own operations into the creation of class objects. This is accomplished by:

Writing a subclass of the metaclass type.
Inserting the new metaclass into the class creation process using the metaclass hook.

In Python 2.x, the metaclass hook is a static field in the class called __metaclass__. In the ordinary case, this is not assigned so Python just uses type to create the class. But if you define __metaclass__to point to a callable, Python will call __metaclass__() after the initial creation of the class object, passing in the class object, the class name, the list of base classes and the namespace dictionary.

Python 2.x also allows you to assign to the global __metaclass__ hook, which will be used if there is not a class-local __metaclass__ hook (is there an equivalent in Python 3?).

Thus, the basic process of metaclass programming looks like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
classDemoMetaclass(type):
    def__init__(cls,name,bases,nmspc):
        super(DemoMetaclass,cls).__init__(name,bases,nmspc)
        cls.uses_metaclass=lambdaself:"Yes!"
 
classDemo(object):
    __metaclass__=DemoMetaclass
    deffoo(self):pass
    @staticmethod
    defbar():pass
 
simple=Demo()
print([mformindir(simple)ifnotm.startswith('__')])
# A new method has been injected by the metaclass:
printsimple.uses_metaclass()
 
#['bar', 'foo', 'uses_metaclass']
#Yes!

By convention, when defining metaclasses cls is used rather than self as the first argument to all methods except __new__() (which uses mcl, for reasons explained later). cls is the class object that is being modified.

Note that the practice of calling the base-class constructor first (via super()) in the derived-class constructor should be followed with metaclasses as well.

__metaclass__ only needs to be callable, so in Python 2.x it’s possible to define __metaclass__ inline:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
classDemo2(object):
    class__metaclass__(type):
        def__init__(cls,name,bases,nmspc):
            # This won't work:
            # super(__metaclass__, cls).__init__(name, bases, nmspc)
            # Less-flexible specific call:
            type.__init__(cls,name,bases,nmspc)
            cls.uses_metaclass=lambdaself:"Yes!"
 
classDemo3(Demo2):pass
simple=Demo3()
printsimple.uses_metaclass()
 
#Yes!

The compiler won’t accept the super() call because it says __metaclass__ hasn’t been defined, forcing us to use the specific call to type.__init__().

Because it only needs to be callable, it’s even possible to define __metaclass__ as a function:

1
2
3
4
5
6
7
8
9
10
classDemo4(object):
    def__metaclass__(name,bases,nmspc):
        cls=type(name,bases,nmspc)
        cls.uses_metaclass=lambdaself:"Yes!"
        returncls
 
simple=Demo4()
printsimple.uses_metaclass()
 
#Yes!

As you’ll see, Python 3 doesn’t allow the syntax of these last two examples. Even so, the above example makes it quite clear what’s happening: the class object is created, then modified, then returned.

The Metaclass Hook in Python 3

Python 3 changes the metaclass hook. It doesn’t disallow the __metaclass__ field, but it ignores it. Instead, you use a keyword argument in the base-class list:

1
2
classSimple1(object,metaclass=SimpleMeta1):
    ...

This means that none of the (clever) alternative ways of defining __metaclass__ directly as a class or function are available in Python 3. All metaclasses must be defined as separate classes. This is probably just as well, as it makes metaclass programs more consistent and thus easier to read and understand.

Practical Example of Metaclasses with Python3:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
importcollections
 
classAutoSlotProperties(type):
    def__new__(mcl,classname,bases,dictionary):
        slots=list(dictionary.get("__slots__",[]))
        forgetter_namein[keyforkeyindictionaryifkey.startswith("get_")]:
            ifisinstance(dictionary[getter_name],collections.Callable):
                name=getter_name[4:]
                slots.append("__"+name)
                getter=dictionary.pop(getter_name)
                setter_name="set_"+name
                setter=dictionary.get(setter_name,None)
                if(setterisnotNoneand
                    isinstance(setter,collections.Callable)):
                    deldictionary[setter_name]
                dictionary[name]=property(getter,setter)
        dictionary["__slots__"]=tuple(slots)
        returnsuper().__new__(mcl,classname,bases,dictionary)
 
classProduct(metaclass=AutoSlotProperties):
    def__init__(self,barcode,description):
        self.__barcode=barcode
        self.description=description
 
    defget_barcode(self):
        returnself.__barcode
 
    defget_description(self):
        returnself.__description
 
    defset_description(self,description):
        ifdescriptionisNoneorlen(description)<3:
            self.__description="<Invalid Description>"
        else:
            self.__description=description
 
product=Product("101110110","8mm Stapler")
print(product.barcode,product.description)
#101110110 8mm Stapler
 
product.description="8mm Stapler (long)"  
print(product.barcode,product.description)  
#101110110 8mm Stapler (long)
 
product.barcode="1111111"
print(product.barcode,product.description)
#AttributeError: cant set attribute

A metaclass’s __new__() class method is called with the metaclass,and the class name, base classes, and dictionary of the class that is to be created. We must use a reimplementation of __new__() rather than __init__() because we want to change the dictionary before the class is created. We begin by copying the __slots__ collection, creating an empty one if none is present, and making sure we have a list rather than a tuple so that we can modify it. For every attribute in the dictionary we pick out those that begin with “get_” and that are callable, that is, those that are getter methods. For each getter we add a private name to the slots to store the corresponding data; for example, given getter get_name() we add __name to the slots. We then take a reference to the getter and delete it from the dictionary under its original name (this is done in one go using dict.pop()).We do the same for the setter if one is
present, and then we create a new dictionary item with the desired property name as its key; for example, if the getter is get_name() the property name is name.We set the item’s value to be a property with the getter and setter (which might be None) that we have found and removed from the dictionary.

For this example we didn’t need to write an __init__() method because we have
done all the work in __new__(), but it is perfectly possible to reimplement both
__new__() and __init__() doing different work in each.

Using `init` vs. `new` in Metaclasses

It can be confusing when you see metaclass examples that appear to arbitrarily use __new__ or__init__ – why choose one over the other?

__new__ is called for the creation of a new class, while __init__ is called after the class is created, to perform additional initialization before the class is handed to the caller:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
frompprintimportpprint
 
classTag1:pass
classTag2:pass
classTag3:
    deftag3_method(self):pass
 
classMetaBase(type):
    def__new__(mcl,name,bases,nmspc):
        print('MetaBase.__new__n')
        returnsuper(MetaBase,mcl).__new__(mcl,name,bases,nmspc)
 
    def__init__(cls,name,bases,nmspc):
        print('MetaBase.__init__n')
        super(MetaBase,cls).__init__(name,bases,nmspc)
 
classMetaNewVSInit(MetaBase):
    def__new__(mcl,name,bases,nmspc):
        # First argument is the metaclass ``MetaNewVSInit``
        print('MetaNewVSInit.__new__')
        forxin(mcl,name,bases,nmspc):pprint(x)
        print('')
        # These all work because the class hasn't been created yet:
        if'foo'innmspc:nmspc.pop('foo')
        name+='_x'
        bases+=(Tag1,)
        nmspc['baz']=42
        returnsuper(MetaNewVSInit,mcl).__new__(mcl,name,bases,nmspc)
 
    def__init__(cls,name,bases,nmspc):
        # First argument is the class being initialized
        print('MetaNewVSInit.__init__')
        forxin(cls,name,bases,nmspc):pprint(x)
        print('')
        if'bar'innmspc:nmspc.pop('bar')# No effect
        name+='_y'# No effect
        bases+=(Tag2,)# No effect
        nmspc['pi']=3.14159# No effect
        super(MetaNewVSInit,cls).__init__(name,bases,nmspc)
        # These do work because they operate on the class object:
        cls.__name__+='_z'
        cls.__bases__+=(Tag3,)
        cls.e=2.718
 
classTest(object):
    __metaclass__=MetaNewVSInit
    def__init__(self):
        print('Test.__init__')
    deffoo(self):print('foo still here')
    defbar(self):print('bar still here')
 
t=Test()
print('class name: '+Test.__name__)
print('base classes: ',[c.__name__forcinTest.__bases__])
print([mformindir(t)ifnotm.startswith("__")])
t.bar()
print(t.e)
 
"""
MetaNewVSInit.__new__
<class '__main__.MetaNewVSInit'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'foo': <function foo at 0x7ed30>}
 
MetaBase.__new__
 
MetaNewVSInit.__init__
<class '__main__.Test_x'>
'Test'
(<type 'object'>,)
{'__init__': <function __init__ at 0x7ecf0>,
'__metaclass__': <class '__main__.MetaNewVSInit'>,
'__module__': '__main__',
'bar': <function bar at 0x7ed70>,
'baz': 42}
 
MetaBase.__init__
 
Test.__init__
class name: Test_x_z
('base classes: ', ['object', 'Tag1', 'Tag3'])
['bar', 'baz', 'e', 'tag3_method']
bar still here
2.718
"""

The primary difference is that when overriding __new__() you can change things like the ‘name’, ‘bases’ and ‘namespace’ arguments before you call the super constructor and it will have an effect, but doing the same thing in __init__() you won’t get any results from the constructor call.

One special case in __new__() is that you can manipulate things like __slots__, but in __init__()you can’t.

Note that, since the base-class version of __init__() doesn’t make any modifications, it makes sense to call it first, then perform any additional operations. In C++ and Java, the base-class constructormust be called as the first operation in a derived-class constructor, which makes sense because derived-class constructions can then build upon base-class foundations.

In many cases, the choice of __new__() vs __init__() is a style issue and doesn’t matter, but because__new__() can do everything and __init__() is slightly more limited, some people just start using__new__() and stick with it. This use can be confusing – I tend to hunt for the reason that __init__()has been chosen, and if I can’t find it wonder whether the author knew what they were doing. I prefer to only use __new__() when it has meaning – when you must in order to change things that only__new__() can change.

Class Methods and Metamethods

A metamethod can be called from either the metaclass or from the class, but not from an instance. A classmethod can be called from either a class or its instances, but is not part of the metaclass.

Intercepting Class Creation

This example implements Singleton using metaclasses, by overriding the __call__() metamethod, which is invoked when a new instance is created:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
classSingleton(type):
    instance=None
    def__call__(cls,*args,**kw):
        ifnotcls.instance:
            cls.instance=super(Singleton,cls).__call__(*args,**kw)
        returncls.instance
 
classASingleton(object):
    __metaclass__=Singleton
 
a=ASingleton()
b=ASingleton()
assertaisb
print(a.__class__.__name__,b.__class__.__name__)
 
classBSingleton(object):
    __metaclass__=Singleton
 
c=BSingleton()
d=BSingleton()
assertcisd
print(c.__class__.__name__,d.__class__.__name__)
assertcisnota
 
#('ASingleton', 'ASingleton')
#('BSingleton', 'BSingleton')

By overriding __call__() in the metaclass, the creation of instances are intercepted. Instance creation is bypassed if one already exists.

Note the dependence upon the behavior of static class fields. When cls.instance is first read, it gets the static value of instance from the metaclass, which is None. However, when the assignment is made, Python creates a local version for the particular class, and the next time cls.instance is read, it sees that local version. Because of this behavior, each class ends up with its own class-specificinstance field (thus instance is not somehow being “inherited” from the metaclass).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
defsingleton(klass):
    "Simple replacement of object creation operation"
    defgetinstance(*args,**kw):
        ifnothasattr(klass,'instance'):
            klass.instance=klass(*args,**kw)
        returnklass.instance
    returngetinstance
 
defsingleton(klass):
    """
    More powerful approach: Change the behavior
    of the instances AND the class object.
    """
    classDecorated(klass):
        def__init__(self,*args,**kwargs):
            ifhasattr(klass,'__init__'):
                klass.__init__(self,*args,**kwargs)
        def__repr__(self):returnklass.__name__+" obj"
        __str__=__repr__
    Decorated.__name__=klass.__name__
    classClassObject:
        def__init__(cls):
            cls.instance=None
        def__repr__(cls):
            returnklass.__name__
        __str__=__repr__
        def__call__(cls,*args,**kwargs):
            printstr(cls)+" __call__ "
            ifnotcls.instance:
                cls.instance=Decorated(*args,**kwargs)
            returncls.instance
    returnClassObject()
 
@singleton
classASingleton:pass
 
a=ASingleton()
b=ASingleton()
print(a,b)
printa.__class__.__name__
printASingleton
assertaisb
 
@singleton
classBSingleton:
    def__init__(self,x):
        self.x=x
 
c=BSingleton(11)
d=BSingleton(22)
assertcisd
assertcisnota
 
"""
ASingleton __call__
ASingleton __call__
(ASingleton obj, ASingleton obj)
ASingleton
ASingleton
BSingleton __call__
BSingleton __call__
"""

The `prepare()` Metamethod

One of the things you can’t do with class decorators is to replace the default dictionary. In Python 3 this is enabled with the __prepare__() metamethod:

1
2
3
@classmethod
def__prepare__(mcl,name,bases):
    returnodict()

For an example of using both __prepare__() and __slots__ in metaclasses, see Michele Simionato’s article.

Metaclass Conflicts

Note that the metaclass argument is singular – you can’t attach more than one metaclass to a class. However, through multiple inheritance you can accidentally end up with more than one metaclass, and this produces a conflict which must be resolved.

SOLVING THE METACLASS CONFLICT

Metaprogramming in Python