Mailing List Archive

[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access
Change by Vinay Sharma <vinay0410sharma@gmail.com>:


----------
pull_requests: +20652
pull_request: https://github.com/python/cpython/pull/21516

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

@Davin, could you merge one or the other of the PRs that fix this? Presumably also backport to 3.9 and 3.8 (but that's up to you and the release manager).

----------
nosy: +gvanrossum
versions: +Python 3.10

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Change by STINNER Victor <vstinner@python.org>:


----------
nosy: -vstinner

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Damian Barabonkov <damianb@mit.edu> added the comment:

As per Guido's comment (https://github.com/python/cpython/pull/21516#issuecomment-668110711), I'm going to use this space to discuss ways to go forward with resource tracking and SharedMemory.

Taking inspiration from Vinay (https://bugs.python.org/issue37754#msg351445), I think the simplest and best way forward is to use a small section of the shared memory at the start as a reference counter.

Every time a process latches onto a shared memory block, it does and atomic increment to the reference counter. And if it detaches, it does an atomic decrement. This atomic operations are available in C via hardware specific instructions. This would require modifying the Python C code posixshmem.c. It should not be a difficult change.

This would then change the SharedMemory API such that a call to `close()` could check the reference count at the end, and aromatically unlink if it reaches 0. Basically, the purpose of the explicit `unlink()` call is dissolved.

I think this would also play nice with the current implementation of the `resource_tracker`. A small change would need to take place such that it calls `close()` instead of `unlink()` as the clean up function. Nonetheless, it would keep track if all attachments of shared memory call `close()` at the end, which they should, and issue a warning if they do not. It would do this with the current code, no need to change anything.

----------
nosy: +damian.barabonkov

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Guido van Rossum <guido@python.org> added the comment:

I recommend bringing this new proposal up on python-dev or python-ideas, to get more eyeballs on the ideas before attempting implementation. One immediate worry I have is that if the reference counter is maintained in the shared memory segment, every process has to participate, and if a process crashes (segfaults) the shared refcount will be forever wrong and the segment will leak. Distributed GC is hard!

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Vinay Sharma <vinay0410sharma@gmail.com> added the comment:

That's a valid point Guido. But, I guess this can be easily handled by resource tracker. At this current moment resource tracker unlinks shared memory if the process which created it dies without unliking it.

Therefore, resource tracker handles cleaning up resources which the respective processes couldn't or didn't do.

So, instead of unlinking the shared memory segment, resource tracker can instead decrement the reference count, if the process failed to do so, and if the reference count becomes 0, then unlink the shared memory segment.

This approach will ensure that even if the respective processes died unexpectedly, there are no leaks.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Damian Barabonkov <damianb@mit.edu> added the comment:

Unless the resource_tracker also dies along with the process. In which case, I'm not sure what there is there to do.

I believe the resource_tracker actually spawns a process alongside the process that uses it. So if the parent process seg-faults, the resource_tracker should still be alive.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Vinay Sharma <vinay0410sharma@gmail.com> added the comment:

Well, the chances of resource tracker dying abruptly are very less because it's thoroughly tested, and there are mechanisms to re-spawn resource_tracker process if you see the code. There is a function called `def ensure_running`.

Resource tracker is still alive even if the process for which it was created dies. It also handles cleaning shared semaphores. So, I guess this is something we can rely on for cleaning up things, because at the end of the day that's what it was made for.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Damian Barabonkov <damianb@mit.edu> added the comment:

Agreed.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue38119] resource tracker destroys shared memory segments when other processes should still have valid access [ In reply to ]
Vinay Sharma <vinay0410sharma@gmail.com> added the comment:

As suggested by Guido I have floated this solution to python-dev mailing list.
Link to archive: https://mail.python.org/archives/list/python-dev@python.org/thread/O67CR7QWEOJ7WDAJEBKSY74NQ2C4W3AI/

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue38119>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com