{"id":5146,"date":"2024-11-04T22:56:01","date_gmt":"2024-11-05T03:56:01","guid":{"rendered":"https:\/\/ikriv.com\/blog\/?p=5146"},"modified":"2024-11-04T23:02:01","modified_gmt":"2024-11-05T04:02:01","slug":"python-modifying-imported-value","status":"publish","type":"post","link":"https:\/\/ikriv.com\/blog\/?p=5146","title":{"rendered":"Python: modifying imported value"},"content":{"rendered":"

TL;DR<\/strong>: Python statement from module import obj<\/code>\u00a0creates a copy <\/em>of the object in the importing scope.<\/p>\n

In other words, from module import obj<\/code> is roughly equivalent to obj = __import__('module').obj<\/code>. It creates a new module-level variable, which is independent of\u00a0 the original module.obj<\/code>. Python documentation<\/a> says, somewhat cryptically, that ” it binds the results of that [import] search to a name in the local scope<\/em>“, but as far as I understand it is equivalent to “creates a new variable”.<\/p>\n

This is sort of obvious after the fact, but it took me a while to fully internalize all the consequences. I subconsciously assumed that from module import obj<\/code> is just syntactic sugar that allows to write obj<\/code> instead of module.obj<\/code>, similar to how using<\/code> works in C++ and C#, and import<\/code> in Java. I already knew that this is not the case when patching<\/a> is involved, but I did not think much about it.<\/p>\n

None of the above matters as long as imported variables are not re-assigned. If they point to a modifiable object, such as a list, all modifications to that list will be visible through all variables. However, assigning a new list<\/em> to one variable, will not be reflected in the others.<\/p>\n

\r\n# module_a.py\r\nglobal_list = []                 # declares module_a.global_list\r\n\r\n# module_b.py\r\nfrom module_a import global_list # module_b.global_list = module_a.global_list\r\n\r\n# main.py\r\nfrom module_a import global_list # main.global_list = module_a.global_list\r\nglobal_list.append(42)           # the change is visible in all modules, the list now contains [42]\r\nglobal_list = [1,2,3,4]          # the change only affects main.global_list, other lists still contain [42]\r\n<\/pre>\n
However, if we only import the modules, no new variables are created.<\/p>\n
\r\n# module_a.py\r\nglobal_list = []                 # module_a.global_list\r\n\r\n# module_b.py\r\nimport module_a\r\n...module_a.global_list...\r\n\r\n# main.py\r\nimport module_a \r\nmodule_a.global_list.append(42)   # the change is visible in all modules\r\nmodule_a.global_list = [1,2,3,4]  # the change is visible in modules that use module_a.global_list \r\n                                  # modules that use from module_a import global_list keep the old value\r\n<\/pre>\n
The situation may become rather chaotic if a mix of import module<\/code> and from module import obj<\/code> is used
\nthroughout the code with the same variables.<\/p>\n
Consider this example:<\/p>\n
\r\n#=================== config.py =================== \r\nNUM_THREADS = 10\r\n\r\n#===================  runner_a.py =================== \r\nimport config\r\n\r\ndef run_a():\r\n    print(f"Runner A uses {config.NUM_THREADS} threads")\r\n\r\n#===================  runner_b.py =================== \r\nfrom config import NUM_THREADS\r\n\r\ndef run_b():\r\n    print(f"Runner B uses {NUM_THREADS} threads")\r\n\r\n#===================  main.py =================== \r\nimport config\r\nimport runner_a\r\nimport runner_b\r\n\r\nconfig.NUM_THREADS = 20\r\nrunner_a.run_a()\r\nrunner_b.run_b()\r\n<\/pre>\n
This prints<\/p>\n
Runner A uses 20 threads\r\nRunner B uses 10 threads \r\n<\/pre>\n
I spent about an hour today trying to debug a problem similar to this one.<\/p>\n
The discrepancy can be fixed by making the top config reference immutable, and making the modifiable parts its attributes. This is a little bit more verbose, but this way it is much harder to end up with different config values in different modules.<\/p>\n
\r\n#===================  config.py =================== \r\nclass Config:\r\n    def __init__(self):\r\n        self.NUM_THREADS = 10\r\n\r\nglobal_config = Config()\r\n\r\n#===================  runner_a.py =================== \r\nimport config\r\n\r\ndef run_a():\r\n    print(f"Runner A uses {config.global_config.NUM_THREADS} threads")\r\n\r\n#===================  runner_b.py =================== \r\nfrom config import global_config\r\n\r\ndef run_b():\r\n    print(f"Runner B uses {global_config.NUM_THREADS} threads")\r\n\r\n#===================  main.py =================== \r\nimport config\r\nimport runner_a\r\nimport runner_b\r\n\r\nconfig.global_config.NUM_THREADS = 20\r\nrunner_a.run_a()\r\nrunner_b.run_b()\r\n<\/pre>\n
This version prints<\/p>\n
Runner A uses 20 threads\r\nRunner B uses 20 threads \r\n<\/pre>\n
Better yet, avoid the global config altogether, to prevent any possibility of confusion.<\/p>\n
\r\n#===================  config.py =================== \r\nclass Config:\r\n    def __init__(self):\r\n        self.NUM_THREADS = 10\r\n\r\n#===================  runner_a.py =================== \r\nimport config\r\n\r\ndef run_a(cfg: config.Config) -> None:\r\n    print(f"Runner A uses {cfg.NUM_THREADS} threads")\r\n\r\n#===================  runner_b.py =================== \r\nimport config\r\n\r\ndef run_b(cfg: config.Config) -> None:\r\n    print(f"Runner B uses {cfg.NUM_THREADS} threads")\r\n\r\n#===================  main.py =================== \r\nimport config\r\nimport runner_a\r\nimport runner_b\r\n\r\nmain_config = config.Config()\r\nmain_config.NUM_THREADS = 20\r\nrunner_a.run_a(main_config)\r\nrunner_b.run_b(main_config)\r\n<\/pre>\n
The conclusion is two-fold:<\/p>\n
Avoid mentally mapping similar concepts across programming languages, even when they look similar and do similar things. Java import is not the same as Python import.<\/p>\n
Avoid mixing from module import obj<\/code> and import module ... module.obj<\/code> in the same codebase. Prefer the latter if obj<\/code> can
\nbe potentially reassigned to a new value. This includes both production usage and “patching” for unit tests.<\/p>\n
Happy importing!<\/p>\n","protected":false},"excerpt":{"rendered":"
TL;DR: Python statement from module import obj\u00a0creates a copy of the object in the importing scope. In other words, from module import obj is roughly equivalent to obj = __import__(‘module’).obj. […]<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"Layout":"","footnotes":""},"categories":[4,26],"tags":[],"class_list":["entry","author-ikriv","post-5146","post","type-post","status-publish","format-standard","category-hack","category-python"],"_links":{"self":[{"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5146","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=5146"}],"version-history":[{"count":9,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5146\/revisions"}],"predecessor-version":[{"id":5155,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=\/wp\/v2\/posts\/5146\/revisions\/5155"}],"wp:attachment":[{"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=5146"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=5146"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/ikriv.com\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=5146"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}