{"id":5037,"date":"2024-07-14T23:09:42","date_gmt":"2024-07-15T03:09:42","guid":{"rendered":"https:\/\/ikriv.com\/blog\/?p=5037"},"modified":"2024-07-14T23:09:42","modified_gmt":"2024-07-15T03:09:42","slug":"python-methods-vs-callable-attributes-and-deepcopy","status":"publish","type":"post","link":"https:\/\/ikriv.com\/blog\/?p=5037","title":{"rendered":"Python methods vs callable attributes and (deep)copy"},"content":{"rendered":"

This post describes more fundamental underpinnings of how copy<\/code> and deepcopy<\/code> interact with callable attributes. See previous post<\/a> for some practical examples of unexpected behavior.<\/p>\n

Methods vs Instance Attributes<\/h1>\n

Consider the following code. Does statement self.method = self.method<\/code> alter the object state? Surprisingly, the answer is yes.<\/p>\n

\r\nclass TestClass1:\r\n    def method(self, arg):\r\n        print(self)\r\n        print(arg)\r\n    \r\n    def update(self):\r\n        self.method = self.method\r\n<\/pre>\n
In Python, methods are attributes of a class. Class TestClass1<\/code> literally contains a free-standing function that takes two arguments: self<\/code> and arg<\/code>. We can even call it directly, and the first argument may not be of type TestClass1<\/code>:<\/p>\n
\r\nTestClass1.method("Fake self", 42)\r\n\r\nOutput:\r\nFake self\r\n42\r\n<\/pre>\n
Now, consider this code:<\/p>\n
\r\ninstance = TestClass1()\r\ninstance.method(99) \r\n\r\nOutput:\r\n<__main__.TestClass1 object at 0x7c840fd12b90>\r\n43\r\n<\/pre>\n
What happens when we refer to instance.method<\/code>? There is no attribute named \"method\"<\/code> in instance<\/code>. The answer is, Python uses the “descriptor protocol”.<\/p>\n
Descriptors<\/h1>\n
When we write something like instance.attr<\/code>, Python performs several steps behind the scenes. Somewhat simplified version of the logic looks as follows:<\/p>\n
    \n
  1. Python looks for attribute named \"attr\"<\/code> in the instance<\/code> object.<\/li>\n
  2. If it is not found, Python looks for the attribute named \"attr\"<\/code> in the instance<\/code>‘s class.<\/li>\n
  3. If this attribute is found and has a __get__()<\/code> method, it is considered a descriptor<\/em>. Python calls the __get__()<\/code> method, and returns the value it produces.<\/li>\n
  4. Otherwise, Python returns the value of the class attribute value as is.<\/li>\n
  5. If the attribute is not found in the class, Python raises an AttributeError<\/code>.\n<\/ol>\n
    Let’s test this in practice:<\/p>\n
    \r\nclass CountDescriptor:\r\n    def __init__(self):\r\n        self.count = 0\r\n        \r\n    def __get__(self, obj, objtype = None):\r\n        self.count += 1\r\n        return self.count\r\n        \r\nclass TestClass2:\r\n    class_attr = 42\r\n    count = CountDescriptor()\r\n    def __init__(self):\r\n        self.instance_attr = 100\r\n        \r\ninstance = TestClass2()\r\nprint(instance.instance_attr) # 100, from the instance attribute\r\nprint(instance.class_attr)    #  42, from TestClass.class_attr\r\nprint(instance.count)         #   1  from TestClass.count.__get__()\r\nprint(instance.count)         #   2  from TestClass.count.__get__()     \r\n<\/pre>\n
    Functions are Descriptors<\/h1>\n
    Going back to our previous example, TestClass1.method<\/code> is a function, and type function<\/code> does have a method named __get__()<\/code>, making every function a descriptor.
    \nWhen invoked, this method produces a “bound method” object. It’s a callable whose job is to prepend instance<\/code> to its list of arguments, and call the original function. The bound method object keeps instance<\/code> as an attribute, and thus is “bound” to it. This code is written in C, but the Python pseudo code looks like this:<\/p>\n
    \r\nclass BoundMethod:\r\n    def __init__(self, func, obj):\r\n        self.__func__ = func\r\n        self.__self__ = obj\r\n\r\n    def __call__(self, *args, **kwargs):\r\n        func = self.__func__\r\n        obj = self.__self__\r\n        return func(obj, *args, **kwargs)\r\n\r\nclass Function:\r\n    ...\r\n\r\n    def __get__(self, obj, objtype=None):\r\n        if obj is None:\r\n            return self\r\n        return MethodType(self, obj)\r\n<\/pre>\n
    Let’s check how it works:<\/p>\n
    \r\nclass TestClass1:\r\n    def method(self, arg):\r\n        print(self)\r\n        print(arg)\r\n    \r\n    def update(self):\r\n        self.method = self.method\r\n        \r\ninstance = TestClass1()\r\nbound_method = TestClass1.method.__get__(instance, TestClass1)\r\n\r\nprint(f"instance is {instance}")  # instance is <__main__.TestClass1 object at 0x784b3023b190>\r\n\r\n# we can now call instance.method in three different ways\r\nTestClass1.method(instance, 42)   # invocation 1\r\nbound_method(42)                  # invocation 2\r\ninstance.method(42)               # invocation 3\r\n\r\n# All three invocations print\r\n# <__main__.TestClass1 object at 0x784b3023b190>\r\n# 42\r\n<\/pre>\n
    Each call to instance.method<\/code> returns a new bound method object. The resulting objects are equal, but not identical.<\/p>\n
    \r\nb1 = instance.method\r\nb2 = instance.method\r\nprint(bound_method)         # <bound method TestClass1.method of <__main__.TestClass1 object at 0x784b3023b190>>\r\nprint(b1)                   # <bound method TestClass1.method of <__main__.TestClass1 object at 0x784b3023b190>>\r\nprint(b2)                   # <bound method TestClass1.method of <__main__.TestClass1 object at 0x784b3023b190>>\r\nprint(b1 is b2)             # False\r\nprint(b1 == b2)             # True\r\nprint(bound_method == b1)   # True\r\nprint(bound_method == b2)   # True\r\n<\/pre>\n
    Bound Method as Instance Attribute<\/h1>\n
    If the instance itself has attribute \"method\"<\/code>, it turns off the descriptor mechanism. The attribute is now part of the instance state (__dict__<\/code>), and the same object is returned each time.<\/p>\n
    \r\nclass TestClass1:\r\n    def method(self, arg):\r\n        print(self)\r\n        print(arg)\r\n\r\n    def update(self):\r\n        self.method = self.method\r\n        \r\ninstance = TestClass1()\r\nprint(instance.__dict__)  # {} -- no instance attributes\r\nb1 = instance.method      \r\nprint(id(b1))             # 134705937658240\r\nb2 = instance.method      \r\nprint(id(b2))             # 134705937658368 - new object\r\n\r\ninstance.update()         # create new instance attribute named "method"\r\nprint(instance.__dict__)  # {'method': <bound method TestClass1.method of <__main__.TestClass1 object at 0x7a83ab902dd0>>}\r\n\r\nb3 = instance.method      # 134705937658432\r\nprint(id(b3))\r\nb4 = instance.method      # 134705937658432 -- same as b3; descriptor code is no longer called\r\nprint(id(b4))\r\nb5 = instance.method      # 134705937658432 -- same as b3 and b4; descriptor code is no longer called\r\nprint(id(b5))\r\n<\/pre>\n
    So, it is now clear that self.method = self.method<\/code> does alter the visible behavior somewhat: the instance now has a new attribute, and its value is fixed, the calls are no longer going through the descriptor logic. This becomes important when we start copying things.<\/p>\n
    References: <\/p>\n
      \n
    1. Descriptor guide: Functions and Methods<\/a>.<\/li>\n
    2. Descriptor Protocol<\/a>.<\/li>\n<\/ol>\n
      How Copy Works<\/h1>\n
      The source code for Python copy()<\/code><\/a> is somewhat confusing. but barring special customizations the workflow is as follows:<\/p>\n
        \n
      1. copy()<\/code> calls magic method __reduce_ex__()<\/code> on whatever is being copied.<\/li>\n
      2. Default implementation of this function found in object<\/code> supplies a tuple with the following members:\n
          \n
        1. func<\/code>: a function that creates new instance of the object without constructor.<\/li>\n
        2. args<\/code>: tuple containing arguments to that function. Typically it has only one member: the object type.<\/li>\n
        3. state<\/code>: the dictionary containing object state.<\/li>\n
        4. listiter<\/code>: usually None<\/code>.<\/li>\n
        5. dictiter<\/code>: usually None<\/code><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n
          The copy logic then calls _reconstruct()<\/code> function, that creates a new uninitialized object via func(*args)<\/code>, and does y.__dict__.update(state)<\/code>, where y<\/code> is the new object. It therefore sets the object attributes directly, bypassing the __init__()<\/code> method. All these details are specific to CPython, but it is a rather widespread implementation of Python.<\/p>\n
          Let’s consider an example:<\/p>\n
          \r\nclass TestClass3:\r\n    def __init__(self):\r\n        self.value = 42\r\n        self.method = self.amethod\r\n        \r\n    def amethod(self):\r\n        print(self.value)\r\n        \r\nsource_instance = TestClass3()\r\ndata = source_instance.__reduce_ex__(4)  # 4 is the current pickle protocol version\r\nprint(data)\r\n<\/pre>\n
          Formatted output:<\/p>\n
          \r\n(\r\n  <function __newobj__ at 0x7bc5100f8ae0>, \r\n  (<class '__main__.TestClass3'>,), \r\n  {'value': 42, 'method': <bound method TestClass3.amethod of <__main__.TestClass3 object at 0x7bc50ff36dd0>>}, \r\n  None, \r\n  None\r\n)\r\n<\/pre>\n
          The state<\/code> dictionary contains the value 42, and the bound method, which is bound to the source instance.<\/p>\n
          When we create a shallow copy of source_instance<\/code>, it is still bound to the original instance:<\/p>\n
          \r\nfrom copy import copy\r\ncopy_instance = copy(source_instance)\r\nprint(dest_instance.method.__self__ is source_instance)  # True\r\ncopy_instance.value = 100                  \r\ncopy_instance.amethod()                                  # 100\r\ncopy_instance.method()                                   # 42 <--- it still uses  source_instance\r\n<\/pre>\n
          This is, of course, not ideal, since calling copy_instance.method()<\/code> will operate on source_instance<\/code>, not on copy_instance<\/code>.<\/p>\n
          How Deep Copy Works<\/h1>\n
          deepcopy()<\/code> generally follows the same steps as copy()<\/code>, but it recursively copies the arguments of the initialization function, and members of the object state, before passing the on. It also keeps track of already copied objects in `memo` dictionary, to prevent copying something twice. It is implemented by these lines in `copy.py`:<\/p>\n
          \r\n  if deep and args:\r\n    args = (deepcopy(arg, memo) for arg in args)\r\n  ...\r\n  if deep:\r\n    state = deepcopy(state, memo)\r\n<\/pre>\n
          Let’s re-examine the state dictionary in our example:
          \n{'value': 42, 'method': <bound method TestClass3.amethod of <__main__.TestClass3object at 0x7bc50ff36dd0>>}<\/code>.<\/p>\n
          Certain types are subject to _deepcopy_atomic<\/code>: the copy process simply returns the original object. Among other things, this applies to integers and functions, but not to bound methods. The reducer for the bound method returns the following data:<\/p>\n
          \r\nprint(source_instance.method.__reduce_ex__(4))\r\n\r\nOutput:\r\n(<built-in function getattr>, (<__main__.TestClass3 object at 0x7e27d788ad90>, 'amethod'))\r\n<\/pre>\n
          Note that this data is somewhat unusual, the “new” function does not create a new object, but retrieves attribute amethod<\/code> of the existing one. Function arguments will be also deep-copied. But when we do deep = deepcopy(source_instance)<\/code>, note that __main__.TestClass3 object at 0x7e27d788ad90<\/code> points to the original object source_instance<\/code>, which already has been deep-copied to deep<\/code>, and therefore deepcopy()<\/code> replaces it with deep<\/code>.<\/p>\n
          As a result it almost accidentally does the right thing: newly created bound method object points to deep<\/code>! Let’s verify it:<\/p>\n
          \r\nfrom copy import deepcopy\r\n\r\ndeep = deepcopy(source_instance)\r\nprint(deep.method.__self__ is source_instance)  # False\r\ndeep.value = 100\r\ndeep.amethod()                      # 100\r\ndeep.method()                       # 100\r\n<\/pre>\n
          Deep Copy Can’t Change Bound Functions<\/h1>\n
          Deep copy will properly copy bound methods, but it won’t deep copy functions that capture bound methods in their code. Example:<\/p>\n
          \r\ndef decorate(method):\r\n    def inner():\r\n        method()\r\n    return inner\r\n\r\nclass TestClass4:\r\n    def __init__(self):\r\n        self.value = 42\r\n        self.method = decorate(self.amethod)\r\n        \r\n    def amethod(self):\r\n        print(self.value)\r\n\r\nsource_instance = TestClass4()\r\ndata = source_instance.__reduce_ex__(4)\r\nprint(data)\r\n<\/pre>\n
          Formatted output:<\/p>\n
          \r\n(\r\n  <function __newobj__ at 0x7e9dae9dcae0>, \r\n  (<class '__main__.TestClass4'>,), \r\n  {'value': 42, 'method': <function decorate.<locals>.inner at 0x7e9dae80c720>}, \r\n  None, \r\n  None\r\n)\r\n<\/pre>\n
          Note that the state now contains a function object instead of a bound method. Functions are one of the special types whose copy is the original value itself. So, both regular copy and deep copy will keep the function bound to the original variable foo<\/code>:<\/p>\n
          \r\nfrom copy import copy, deepcopy\r\ncopy_instance = copy(source_instance)\r\ncopy_instance .value = 100\r\ncopy_instance .amethod()  # 100\r\ncopy_instance .method()   # 42 <-- bound to source_instance\r\n\r\ndeep = deepcopy(source_instance)\r\ndeep.value = 100\r\ndeep.amethod()  # 100\r\ndeep.method()   # 42 <-- bound to source_instance\r\n<\/pre>\n
          Disabling copy and deep copy<\/h1>\n
          It is possible to disable copy()<\/code> and deepcopy()<\/code> by overriding magic methods __copy__()<\/code> and __deepcopy__()<\/code> respectfully and raising an error:<\/p>\n
          \r\nfrom copy import copy, deepcopy\r\n\r\nclass Uncopyable:\r\n   def __copy__(self):\r\n       raise RuntimeError("Uncopyable class cannot be copied")\r\n\r\n   def __deepcopy__(self, memo):\r\n       raise RuntimeError("Uncopyable class cannot be deepcopied")\r\n\r\ninstance = Uncopyable()\r\ncopy(instance)      # RuntimeError: Uncopyable class cannot be copied\r\ndeepcopy(instance)  # RuntimeError: Uncopyable class cannot be deepcopied\r\n<\/pre>\n
          Keep in mind, however that in Python user can always copy things manually, there is no such thing as private attributes.<\/p>\n
          Implementing Custom Copy<\/h1>\n
          We can implement our own copy mechanism, instead of relying on the standard copy()<\/code> that does not always to the right thing.<\/p>\n
          \r\nfrom copy import copy\r\n\r\nclass CopyableIntroduction:\r\n    def __init__(self, name, default_language):\r\n        self.name = name\r\n        self._default_language = default_language  # logically const; changing it won't have effect\r\n        self.introduce = (self.introduce_in_english \r\n            if default_language == "English"\r\n            else self.introduce_in_french)\r\n \r\n    def introduce_in_english(self):\r\n        print(f"I am {self.name}")\r\n \r\n    def introduce_in_french(self):\r\n        print(f"Je m'apelle {self.name}")\r\n\r\n    def __copy__(self):\r\n        return CopyableIntroduction(self.name, self._default_language)\r\n\r\n    def __deepcopy__(self, memo):\r\n        return self.__copy__()\r\n\r\npaul = CopyableIntroduction("Paul", "English")\r\npaul.introduce()    # I am Paul\r\n\r\nharry = copy(paul)\r\nharry.name = "Harry"\r\nharry.introduce()   # I am Harry\r\n<\/pre>\n
          Without the custom copier, harry.introduce()<\/code> would print \"I am Paul\"<\/code>, because mechanically copied introduce<\/code> method would still be bound to paul<\/code>.<\/p>\n
          Summary<\/h1>\n
          Python provides a rather ingenious machinery to implement class methods, but it has some interesting side effects. <\/p>\n