A first look at descriptors

De­scrip­tors are one of the most pow­er­ful fea­tures of Python. The rea­son why they’re so pow­er­ful is be­cause they en­able us to con­trol the core op­er­a­tions (get, set, delete) [1], of an at­tribute in a giv­en ob­jec­t, so that we can hook a par­tic­u­lar code, con­trolled by us, in or­der to mod­i­fy, change, or ex­tend the o­rig­i­nal op­er­a­tion.

A descriptor is an object that implements either __get__, __set__, or __delete__.

As of Python 3.6+ [2] the de­scrip­tor pro­to­col en­tails these meth­od­s:

__get__(self, instance, owner)
__set__(self, instance, value)
__delete__(self, instance)
__set_name__(self, instance, name)

We’ll un­der­stand bet­ter what the pa­ram­e­ters mean, once we’ve seen some ex­am­ples of de­scrip­tors and how they’re used.

How to use them

In or­der to use de­scrip­tors we need at least two class­es: one for the de­scrip­tor it­self, and the class that is go­ing to use the de­scrip­tor ob­ject­s (often re­ferred to as the man­aged class).

Getting Data

Con­sid­er this ba­sic ex­am­ple on which I have a fic­tion­al man­ag­er for video out­put, that can han­dle mul­ti­ple de­vices. Each de­vice is set with a par­tic­u­lar res­o­lu­tion, pro­vid­ed by a us­er. How­ev­er, if for some rea­son one of the de­vices ­does not have a ren­der­ing res­o­lu­tion set, we want to use a de­fault one, spec­i­fied on the class def­i­ni­tion.

A pos­si­ble im­ple­men­ta­tion could look like this.

de­scrip­tors0_get0.py (Source)

class Resolution:
    """Represents the resolution for a video display. In case there is no
    resolution set, return a default value, previously indicated.
    """
    def __init__(self, attr_name, default_resolution):
        self.attr_name = attr_name
        self.default_resolution = default_resolution

    def __get__(self, instance, owner):
        if instance is None:
            return self
        return self.default_resolution


class VideoDriver:
    """Contains multiple display devices, each one with a resolution
    configured. If a resolution is not set for a device, return a default one,
    provided by this class, as a fallback.

    >>> media = VideoDriver()
    >>> media.tv
    (1024, 768)
    >>> media.tv = (4096, 2160)
    >>> media.tv
    (4096, 2160)
    >>> del media.tv
    >>> media.tv
    (1024, 768)
    >>> media.screen
    (1920, 1080)
    >>> VideoDriver.tv  # doctest: +ELLIPSIS
    <__main__.Resolution object at 0x...>
    """
    tv = Resolution('tv', (1024, 768))
    screen = Resolution('screen', (1920, 1080))


if __name__ == '__main__':
    import doctest
    doctest.testmod()

In this case resolution is a descriptor that implements only __get__(). If an instance of the display manager, has a resolution set, it will retrieve just that one. On the other hand, if it does not, then when we access one of the class attributes like media.tv, what actually happens is that Python calls:

VideoDriver.tv.__get__(media, VideoDriver)

Which executes the code in the __get__() method of the descriptor, which in this case returns the default value, previously passed.

In gen­er­al [4] a code like:

<instance>.<descriptor>

Will be trans­lat­ed to:

type(<instance>).<descriptor>.__get__(<instance>, type(<instance>))

When the de­scrip­tor is called from the class, and not the in­stance, the val­ue of the pa­ram­e­ter “in­stance” is None, but the “own­er” is still a ref­er­ence to the class be­ing in­voked (that’s prob­a­bly one of the rea­sons why these are two sep­a­rate pa­ram­e­ter­s, in­stead of just let the us­er de­rive the class from the in­stance, it al­lows even more flex­i­bil­i­ty).

For this rea­son, is com­mon to han­dle this case, and re­turn the de­scrip­tor it­self, which is the ra­tio­nale be­hind the line:

if instance is None:
    return self

That is why when you de­fine a prop­er­ty in a class, and call it from an in­stance ob­jec­t, you’ll get the re­sult of the com­pu­ta­tion of the method. How­ev­er, if y­ou call the prop­er­ty from the class, you get the prop­er­ty ob­jec­t.

Setting Data

Ex­am­ple: imag­ine we want to have some at­tributes in an ob­ject that are go­ing to be traced, by oth­er at­tributes that keep track, of how many times their val­ues changed. So, for ex­am­ple, for ev­ery at­tribute <x> on the ob­jec­t, there would be a cor­re­spond­ing coun­t_<x> one, that will keep count of how many times x changed its val­ue. For sim­plic­i­ty let’s as­sume at­tributes start­ing with coun­t_<­name>, can­not be mod­i­fied, and those on­ly cor­re­spond to the count of at­tribute <name>.

There may be several ways to address this problem. One way could be overriding __setattr__(). Another option, could be by the means of properties (getters and setters) for each attribute we want to track. Or, we can use descriptors.

Both the properties, and __setattr__() approaches, might be subject to code repetition. Their logic should be repeated for several different properties, unless a property function builder is created (in order to reuse the logic of the property across several variables). As per the __setattr__() strategy, if we need to use this logic in multiple classes we would have to come up with some sort of mixin class, in order to achieve it, and if one of the classes already overrides this method, things might get overcomplicated.

These two op­tions seem rather con­vo­lut­ed. De­scrip­tors it is, then.

de­scrip­tors0_set0.py (Source)

class TracedProperty:
    """Keep count of how many times an attribute changed its value"""

    def __set_name__(self, owner, name):
        self.name = name
        self.count_name = f'count_{name}'

    def __set__(self, instance, value):
        try:
            current_value = instance.__dict__[self.name]
        except KeyError:
            instance.__dict__[self.count_name] = 0
        else:
            if current_value != value:
                instance.__dict__[self.count_name] += 1

        instance.__dict__[self.name] = value


class Traveller:
    """
    >>> tourist = Traveller('John Smith')
    >>> tourist.city = 'Barcelona'
    >>> tourist.country = 'Spain'

    >>> tourist.count_city
    0
    >>> tourist.count_country
    0

    >>> tourist.city = 'Stockholm'
    >>> tourist.country = 'Sweden'
    >>> tourist.count_city
    1
    >>> tourist.count_country
    1
    >>> tourist.city = 'Gothenburg'
    >>> tourist.count_city
    2
    >>> tourist.count_country
    1
    >>> tourist.country = 'Sweden'
    >>> tourist.count_country
    1
    """
    city = TracedProperty()
    country = TracedProperty()

    def __init__(self, name):
        self.name = name


if __name__ == '__main__':
    import doctest
    doctest.testmod()

The doc­string on the Trav­eller class, pret­ty much ex­plains its in­tend­ed use. The im­por­tant thing about this, is the pub­lic in­ter­face: it’s ab­so­lute­ly ­trans­par­ent for the us­er. An ob­ject that in­ter­acts with a Trav­eller in­stance, gets a clean in­ter­face, with the prop­er­ties ex­posed, with­out hav­ing ­to wor­ry about the un­der­ly­ing im­ple­men­ta­tion.

So, we have two class­es, with dif­fer­ent re­spon­si­bil­i­ties, but re­lat­ed, be­cause they in­ter­act to­wards a com­mon goal. Trav­eller has two class at­tributes that, are ob­ject­s, in­stances of the de­scrip­tor.

Now let’s take a look at the oth­er side of it, the in­ter­nal work­ing of the de­scrip­tor.

Un­der this schema, Python will trans­late a call like:

traveller = Traveller()
traveller.city = 'Stockholm'

To the one using the __set__ method in the descriptor, like:

Traveller.city.__set__(traveller, 'Stockholm')

Which means that the __set__ method on the de­scrip­tor is go­ing to re­ceive the in­stance of the ob­ject be­ing ac­cessed, as a first pa­ram­e­ter, and then the ­val­ue that is be­ing as­signed.

More gen­er­al­ly we could say that some­thing like:

obj.<descriptor> = <value>

Trans­lates to:

type(obj).__set__(obj, <value>)

With these two pa­ram­e­ter­s, we can ma­nip­u­late the in­ter­ac­tion any way we wan­t, which makes the pro­to­col re­al­ly pow­er­ful.

In this example, we are taking advantage of this, by querying the original object’s attribute dictionary (instance.__dict__), and getting the value in order to compare it with the newly received one. By reading this value, we calculate another attribute which will hold the count of the number of times the attribute was modified, and then, both of them are updated in the original dictionary for the instance.

An im­por­tant con­cept to point out is that this im­ple­men­ta­tion not on­ly work­s, but it al­so solves the prob­lem in a more gener­ic fash­ion. In this ex­am­ple, it was the case of a trav­eller, of whom we want­ed to know how many times changed of lo­ca­tion, but the ex­act same ob­ject could be used for ex­am­ple to mon­i­tor ­mar­ket stock­s, vari­ables in an equa­tion, etc. This ex­pos­es func­tion­al­i­ty as a ­sort of li­brary, toolk­it, or even frame­work. In fac­t, many well-­known frame­works in Python use de­scrip­tors to ex­pose their API.

Deleting Data

The __delete__() method is going to be called when an instruction of the type del <instance>.<descriptor> is executed. See the following example.

de­scrip­tors0_delete0.py (Source)

"""An example of a descriptor with a ``__delete__()`` method.
The code is for illustration purposes only, and it does not correspond to any
actual implementation.
"""


class ProtectedAttribute:
    """A class attribute that can be protected against deletion"""

    def __set_name__(self, owner, name):
        self.name = name

    def __set__(self, instance, value):
        instance.__dict__[self.name] = value

    def __delete__(self, instance):
        raise AttributeError(f"Can't delete {self.name} for {instance!s}")


class ProtectedUser:
    """
    >>> usr = ProtectedUser('jsmith', '127.0.0.1')
    >>> usr.username
    'jsmith'
    >>> del usr.username
    Traceback (most recent call last):
    ...
    AttributeError: Can't delete username for ProtectedUser[jsmith]
    >>> usr.location
    '127.0.0.1'
    >>> del usr.location
    >>> usr.location
    Traceback (most recent call last):
    ...
    AttributeError: 'ProtectedUser' object has no attribute 'location'
    """
    username = ProtectedAttribute()

    def __init__(self, username, location):
        self.username = username
        self.location = location

    def __str__(self):
        return f"{self.__class__.__name__}[{self.username}]"


if __name__ == '__main__':
    import doctest
    doctest.testmod()

In this ex­am­ple, we just want a prop­er­ty in the ob­jec­t, that can­not be delet­ed, and de­scrip­tors, again, pro­vide one of the mul­ti­ple pos­si­ble im­ple­men­ta­tion­s.

Caveats and recommendations

  • Re­mem­ber that de­scrip­tors should al­ways be used as class at­tributes.
  • Data should be stored in each original managed instance, instead of doing data bookkeeping in the descriptor. Each object should have its data in its __dict__.
  • Preserve the ability of accessing the descriptor from the class as well, not only from instances. Mind the case when instance is None, so it can be called as type(instance).descriptor.
  • Do not override __getattribute__(), or they might lose effect.
  • Mind the dif­fer­ence be­tween da­ta and non-­da­ta de­scrip­tors [3].
  • Im­ple­ment the min­i­mum re­quired in­ter­face.

Food for thought

De­scrip­tors pro­vide a frame­work for ab­stract­ing away repet­i­tive ac­cess log­ic. The term frame­work here is not a co­in­ci­dence. As the read­er might have no­ticed, by us­ing de­scrip­tors, there is an in­ver­sion of con­trol (IoC) on the code, be­cause Python will be call­ing the log­ic we put un­der the de­scrip­tor meth­od­s, when ac­cess­ing these at­tributes from the man­aged in­stance.

Un­der this con­sid­er­a­tions it is cor­rect to think that it be­haves as a frame­work.

Summary

De­scrip­tors pro­vide an API, to con­trol the core ac­cess to an ob­jec­t’s data ­mod­el, at its low-lev­el op­er­a­tions. By means of de­scrip­tors we can con­trol the ex­e­cu­tion of an ob­jec­t’s in­ter­face, be­cause they pro­vide a trans­par­ent lay­er ­be­tween the pub­lic in­ter­face (what is ex­posed to user­s), and the in­ter­nal rep­re­sen­ta­tion and stor­age of da­ta.

They are one of the most powerful features of Python, and their possibilities are virtually unlimited, so in this post we’ve only scratched the surface of them. More details, such as exploring the different types of descriptors with their internal representation or data, the use of the new __set_name__ magic method, their relation with decorators, and analysis of good implementations, are some of the topics for future entries.

[1] Python Cookbook (3rd edition) - David Beazley & Brian K. Jones
[2] https://docs.python.org/3.6/reference/datamodel.html#descriptors
[3] More details about this, will come in a future post.
[4] https://docs.python.org/3.6/howto/descriptor.html#invoking-descriptors