|
| 1 | +--- |
| 2 | +title: "Weak References" |
| 3 | +date: 2021-07-06T21:16:42-03:00 |
| 4 | +weight: 60 |
| 5 | +draft: false |
| 6 | +tags: |
| 7 | +- standard library |
| 8 | +- chapter 6 |
| 9 | +--- |
| 10 | + |
| 11 | +=== Introduction |
| 12 | + |
| 13 | +The presence of references is what keeps an object alive in memory. When the reference count of an object reaches zero, the garbage collector disposes of it. But sometimes it is useful to have a reference to an object that does not keep it around longer than necessary. A common use case is a cache. |
| 14 | + |
| 15 | +Weak references to an object do not increase its reference count. The object that is the target of a reference is called the _referent_. Therefore, we say that a weak reference does not prevent the referent from being garbage collected. |
| 16 | + |
| 17 | +Weak references are useful in caching applications because you don't want the cached objects to be kept alive just because they are referenced by the cache. |
| 18 | + |
| 19 | +<<ex_weakref_ref>> shows how a `weakref.ref` instance can be called to reach its referent. If the object is alive, calling the weak reference returns it, otherwise `None` is returned. |
| 20 | + |
| 21 | +[TIP] |
| 22 | +==== |
| 23 | +<<ex_weakref_ref>> is a console session, and the Python console automatically binds the `_` variable to the result of expressions that are not `None`. This interfered with my intended demonstration but also highlights a practical matter: when trying to micro-manage memory we are often surprised by hidden, implicit assignments that create new references to our objects. The `_` console variable is one example. Traceback objects are another common source of unexpected references. |
| 24 | +==== |
| 25 | + |
| 26 | +[[ex_weakref_ref]] |
| 27 | +.A weak reference is a callable that returns the referenced object or None if the referent is no more |
| 28 | +==== |
| 29 | +[source, pycon] |
| 30 | +---- |
| 31 | +>>> import weakref |
| 32 | +>>> a_set = {0, 1} |
| 33 | +>>> wref = weakref.ref(a_set) <1> |
| 34 | +>>> wref |
| 35 | +<weakref at 0x100637598; to 'set' at 0x100636748> |
| 36 | +>>> wref() <2> |
| 37 | +{0, 1} |
| 38 | +>>> a_set = {2, 3, 4} <3> |
| 39 | +>>> wref() <4> |
| 40 | +{0, 1} |
| 41 | +>>> wref() is None <5> |
| 42 | +False |
| 43 | +>>> wref() is None <6> |
| 44 | +True |
| 45 | +---- |
| 46 | +==== |
| 47 | +<1> The `wref` weak reference object is created and inspected in the next line. |
| 48 | +<2> Invoking `wref()` returns the referenced object, `{0, 1}`. Because this is a console session, the result `{0, 1}` is bound to the `_` variable. |
| 49 | +<3> `a_set` no longer refers to the `{0, 1}` set, so its reference count is decreased. But the `_` variable still refers to it. |
| 50 | +<4> Calling `wref()` still returns `{0, 1}`. |
| 51 | +<5> When this expression is evaluated, `{0, 1}` lives, therefore `wref()` is not `None`. But `_` is then bound to the resulting value, `False`. Now there are no more strong references to `{0, 1}`. |
| 52 | +<6> Because the `{0, 1}` object is now gone, this last call to `wref()` returns `None`. |
| 53 | + |
| 54 | +The http://docs.python.org/3/library/weakref.html[`weakref` module documentation] makes the point that the `weakref.ref` class is actually a low-level interface intended for advanced uses, and that most programs are better served by the use of the `weakref` collections and `finalize`. In other words, consider using `WeakKeyDictionary`, `WeakValueDictionary`, `WeakSet`, and `finalize` (which use weak references internally) instead of creating and handling your own `weakref.ref` instances by hand. We just did that in <<ex_weakref_ref>> in the hope that showing a single `weakref.ref` in action could take away some of the mystery around them. But in practice, most of the time Python programs use the `weakref` collections. |
| 55 | + |
| 56 | +The next subsection briefly discusses the `weakref` collections. |
| 57 | + |
| 58 | +=== The WeakValueDictionary Skit |
| 59 | + |
| 60 | +The class `WeakValueDictionary` implements a mutable mapping where the values are weak references to objects. When a referred object is garbage collected elsewhere in the program, the corresponding key is automatically removed from `WeakValueDictionary`. This is commonly used for caching. |
| 61 | + |
| 62 | +Our demonstration of a `WeakValueDictionary` is inspired by the classic _Cheese Shop_ skit by Monty Python, in which a customer asks for more than 40 kinds of cheese, including cheddar and mozzarella, but none are in stock.footnote:[`cheeseshop.python.org` is also an alias for PyPI--the Python Package Index software repository--which started its life quite empty. At the time of this writing, the Python Cheese Shop has 314,556 packages.] |
| 63 | + |
| 64 | +<<ex_cheeseshop_class>> implements a trivial class to represent each kind of cheese. |
| 65 | + |
| 66 | +[[ex_cheeseshop_class]] |
| 67 | +.Cheese has a kind attribute and a standard representation |
| 68 | +==== |
| 69 | +[source, py] |
| 70 | +---- |
| 71 | +include::./content/extra/weak-references/cheese.py[tags=CHEESE_CLASS] |
| 72 | +---- |
| 73 | +==== |
| 74 | + |
| 75 | +In <<ex_cheeseshop_demo>>, each cheese is loaded from a `catalog` to a `stock` implemented as a `WeakValueDictionary`. However, all but one disappear from the `stock` as soon as the `catalog` is deleted. Can you explain why the Parmesan cheese lasts longer than the others?footnote:[Parmesan cheese is aged at least a year at the factory, so it is more durable than fresh cheese, but this is not the answer we are looking for.] The tip after the code has the answer. |
| 76 | + |
| 77 | +[[ex_cheeseshop_demo]] |
| 78 | +.Customer: "Have you in fact got any cheese here at all?" |
| 79 | +==== |
| 80 | +[source, pycon] |
| 81 | +---- |
| 82 | +>>> import weakref |
| 83 | +>>> stock = weakref.WeakValueDictionary() <1> |
| 84 | +>>> catalog = [Cheese('Red Leicester'), Cheese('Tilsit'), |
| 85 | +... Cheese('Brie'), Cheese('Parmesan')] |
| 86 | +... |
| 87 | +>>> for cheese in catalog: |
| 88 | +... stock[cheese.kind] = cheese <2> |
| 89 | +... |
| 90 | +>>> sorted(stock.keys()) |
| 91 | +['Brie', 'Parmesan', 'Red Leicester', 'Tilsit'] <3> |
| 92 | +>>> del catalog |
| 93 | +>>> sorted(stock.keys()) |
| 94 | +['Parmesan'] <4> |
| 95 | +>>> del cheese |
| 96 | +>>> sorted(stock.keys()) |
| 97 | +[] |
| 98 | +---- |
| 99 | +==== |
| 100 | +<1> `stock` is a `WeakValueDictionary`. |
| 101 | +<2> The `stock` maps the name of the cheese to a weak reference to the cheese instance in the `catalog`. |
| 102 | +<3> The `stock` is complete. |
| 103 | +<4> After the `catalog` is deleted, most cheeses are gone from the `stock`, as expected in `WeakValueDictionary`. Why not all, in this case? |
| 104 | + |
| 105 | +[TIP] |
| 106 | +==== |
| 107 | +A temporary variable may cause an object to last longer than expected by holding a reference to it. This is usually not a problem with local variables: they are destroyed when the function returns. But in <<ex_cheeseshop_demo>>, the `for` loop variable `cheese` is a global variable and will never go away unless explicitly deleted. |
| 108 | +==== |
| 109 | + |
| 110 | +A counterpart to the `WeakValueDictionary` is the `WeakKeyDictionary` in which the keys are weak references. The |
| 111 | +https://docs.python.org/3/library/weakref.html?highlight=weakref#weakref.WeakKeyDictionary[`weakref.WeakKeyDictionary` documentation] |
| 112 | +hints on possible uses: |
| 113 | + |
| 114 | +[quote] |
| 115 | +____ |
| 116 | +[A `WeakKeyDictionary`] can be used to associate additional data with an object owned by other parts of an application without adding attributes to those objects. This can be especially useful with objects that override attribute accesses. |
| 117 | +____ |
| 118 | + |
| 119 | +The `weakref` module also provides a `WeakSet`, simply described in the docs as "Set class that keeps weak references to its elements. An element will be discarded when no strong reference to it exists any more." If you need to build a class that is aware of every one of its instances, a good solution is to create a class attribute with a `WeakSet` to hold the references to the instances. Otherwise, if a regular `set` was used, the instances would never be garbage collected, because the class itself would have strong references to them, and classes live as long as the Python process unless you deliberately delete them. |
| 120 | + |
| 121 | +These collections, and weak references in general, are limited in the kinds of objects they can handle. The next section explains. |
| 122 | + |
| 123 | +=== Limitations of Weak References |
| 124 | + |
| 125 | +Not every Python object may be the target, or referent, of a weak reference. Basic `list` and `dict` instances may not be referents, but a plain subclass of either can solve this problem easily: |
| 126 | + |
| 127 | +[source, python3] |
| 128 | +---- |
| 129 | +class MyList(list): |
| 130 | + """list subclass whose instances may be weakly referenced""" |
| 131 | +
|
| 132 | +a_list = MyList(range(10)) |
| 133 | +
|
| 134 | +# a_list can be the target of a weak reference |
| 135 | +wref_to_a_list = weakref.ref(a_list) |
| 136 | +---- |
| 137 | + |
| 138 | +A `set` instance can be a referent, and that's why a `set` was used in <<ex_weakref_ref>>. User-defined types also pose no problem, which explains why the silly `Cheese` class was needed in <<ex_cheeseshop_demo>>. But `int` and `tuple` instances cannot be targets of weak references, even if subclasses of those types are created. |
| 139 | + |
| 140 | +Most of these limitations are implementation details of CPython that may not apply to other Python interpreters. They are the result of internal optimizations, some of which are discussed in the following section. |
0 commit comments