{"id":4987,"date":"2024-07-13T21:28:11","date_gmt":"2024-07-14T01:28:11","guid":{"rendered":"https:\/\/ikriv.com\/blog\/?p=4987"},"modified":"2024-07-13T22:09:11","modified_gmt":"2024-07-14T02:09:11","slug":"python-dynamic-methods-and-object-copying-a-tale-of-one-bug","status":"publish","type":"post","link":"https:\/\/ikriv.com\/blog\/?p=4987","title":{"rendered":"Python: dynamic methods and object copying, a tale of one bug"},"content":{"rendered":"

Executive Summary<\/h1>\n

Avoid using callable instance attributes that call back into the object that contains them. If you must, make the object uncopyable by overriding __copy__<\/code> and __deepcopy__<\/code> and raising an error.<\/p>\n

Reason: in Python,foo.method<\/code> is permanently bound to foo<\/code>. If bar<\/code> is another instance of the same class, and we do bar.dynamic_method = foo.method<\/code>, bar.dynamic_method()<\/code> operates on foo<\/code>, not on bar<\/code>. This may cause interesting bugs.<\/p>\n

Simple Example<\/h1>\n

Suppose we have a class that prints introductions in different languages. We also add functionality to introduce in default language, as follows:<\/p>\n

\r\nclass Foo:\r\n    def __init__(self, name, default_language):\r\n        self.name = name\r\n        self.introduce = (self.introduce_in_english \r\n            if default_language == "English" \r\n            else self.introduce_in_french)\r\n\r\n    def introduce_in_english(self):\r\n        print(f"I am {self.name}")\r\n\r\n    def introduce_in_french(self):\r\n        print(f"Je m'apelle {self.name}")\r\n        \r\nfoo = Foo("FOO", "English")\r\nfoo.introduce()               # I am FOO\r\nfoo.introduce_in_french()     # Je m'apelle FOO\r\n\r\nbar = Foo("BAR", "French")\r\nbar.introduce = foo.introduce # (BAD IDEA)\r\nfoo.introduce_in_english()    # I am BAR\r\nbar.introduce_in_french()     # Je m'apelle BAR\r\nbar.introduce()               # I am FOO <--- BUG!\r\n<\/pre>\n
If we use copy()<\/code>, the copying happens implicitly, without the user realizing the averse effects. Given the same definition of `Foo` as above:<\/p>\n
\r\nfrom copy import copy\r\nfoo = Foo("FOO", "English")\r\nbar = copy(foo)\r\nbar.name = "BAR"\r\nbar.introduce()               # I am FOO <--- BUG!\r\n<\/pre>\n
Fun fact<\/b>: if we use deepcopy()<\/code> instead of copy()<\/code>, the bug disappears. The explanation is rather subtle, we address it in the next post.<\/p>\n
Solution for the simple example<\/h2>\n
The solution is to avoid having callable attributes that are bound to the instance that contains them. Other callables are fine. This may lead to somewhat longer code, but it is free of the copying bug. For example, we can do:<\/p>\n
\r\ndef introduce_in_english(name):\r\n    print(f"I am {name}")\r\n    \r\ndef introduce_in_french(name):\r\n    print(f"Je m'apelle {name}")\r\n\r\nclass Foo:\r\n    def __init__(self, name, default_language):\r\n        self.name = name\r\n        self.greeter = (introduce_in_english \r\n            if default_language == "English" \r\n            else introduce_in_french)\r\n        \r\n    def introduce(self):\r\n        self.greeter(self.name)\r\n\r\n    def introduce_in_english(self):\r\n        introduce_in_english(self.name)  # call global function\r\n \r\n    def introduce_in_french(self):\r\n        introduce_in_french(self.name)   # call global function\r\n<\/pre>\n
We replace dynamic methods with dynamic free-standing functions, and it avoids the bug. There are lots of other ways to accomplish the same thing, but the key point is to avoid storing a method, or another callable that is bound to the object itself, in an object attribute.<\/p>\n
Real Life Example: Decorators<\/h1>\n
The actual bug we encountered in production was about applying decorators to a method. I can’t copy real production code for intellectual property protection reasons, but conceptually it went something like this:<\/p>\n
\r\nfrom copy import deepcopy\r\n \r\nclass RangeValidator:\r\n    def __init__(self, max_value, decorator=None):\r\n        self.max_value = max_value\r\n        if decorator:\r\n            self.validate = decorator(self.validate)\r\n             \r\n    def set_max_value(self, value):\r\n        self.max_value = value\r\n        return self\r\n             \r\n    def validate(self, number):\r\n        return number < self.max_value\r\n         \r\ndef negate(validator):\r\n    def inner(number):\r\n        return not validator(number)\r\n    return inner\r\n    \r\ndef double(validator):\r\n    return deepcopy(validator).set_max_value(2*validator.max_value)\r\n     \r\nat_least_100 = RangeValidator(100, negate)              # Rejects anything that is under 100\r\nat_least_200 = double(at_least_100)                     # Supposed to reject anything under 200, but doesn't\r\nprint(at_least_100.validate(150))                       # True\r\nprint(at_least_200.validate(150))                       # Also True <-- BUG\r\nprint(at_least_200.max_value)                           # 200\r\n<\/pre>\n
The source of the bug is similar to the simple case, with two differences:<\/p>\n