Mailing List Archive: PyGILState_Release called twice in embedded application

PyGILState_Release called twice in embedded application

Mar 23, 2023, 1:46 AM

Post #1 of 4 (86 views)

Hi all,

I'm running in a crash due to a ResourceWarning (some socket is not
closed in a used module) after calling PyGILState_Release.

I'm running Python in a native thread (so a thread created by C not
Python). I'm acquiring the GIL through PyGILState_Ensure before doing
any CPYthon calls and releasing when finished using PyGILState_Release.

This has worked fine. But now I'm using a python module (openai) which
doesn't close its socket correctly which results in a ResourceWarning
which triggers an assert.

In the backtrace (below) I can see PyGILState_Release is called again.
(7) while I've already called it (126).

I can make the crash go away by adding

import warnings
warnings.simplefilter("ignore", ResourceWarning)

to my python code. But I'd rather prevent this from happening in the
first place.

Any suggestion very welcomed cause I'm puzzled.

Rg,

Arnaud

1 __pthread_kill_implementation pthread_kill.c 44
0x7ffff6fcaccc
2 __pthread_kill_internal pthread_kill.c 78
0x7ffff6fcad2f
3 __GI_raise raise.c 26
0x7ffff6f7bef2
4 __GI_abort abort.c 79
0x7ffff6f66472
5 __assert_fail_base assert.c 92
0x7ffff6f66395
6 __GI___assert_fail assert.c 101
0x7ffff6f74df2
7 PyGILState_Release pystate.c 1742
0x7ffff7abcf9f
8 tracemalloc_raw_alloc _tracemalloc.c 779
0x7ffff7afa803
9 tracemalloc_raw_malloc _tracemalloc.c 789
0x7ffff7afa844
10 PyMem_RawMalloc obmalloc.c 586
0x7ffff79dab8d
11 decode_current_locale fileutils.c 472
0x7ffff7ada4ab
12 _Py_DecodeLocaleEx fileutils.c 598
0x7ffff7adafce
13 unicode_decode_locale unicodeobject.c 3970
0x7ffff7a13b65
14 PyUnicode_DecodeLocale unicodeobject.c 4012
0x7ffff7a18c79
15 PyErr_SetFromErrnoWithFilenameObjects errors.c 772
0x7ffff7a94178
16 PyErr_SetFromErrnoWithFilenameObject errors.c 750
0x7ffff7a942ce
17 posix_path_object_error posixmodule.c 1709
0x7ffff7ae4201
18 path_object_error posixmodule.c 1719
0x7ffff7ae420f
19 path_error posixmodule.c 1737
0x7ffff7ae4221
20 posix_do_stat posixmodule.c 2565
0x7ffff7af42f7
21 os_stat_impl posixmodule.c 2897
0x7ffff7af4491
22 os_stat posixmodule.c.h 71
0x7ffff7af45e5
23 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c 443
0x7ffff79d2ece
24 _PyObject_VectorcallTstate pycore_call.h 92
0x7ffff798b579
25 PyObject_Vectorcall call.c 299
0x7ffff798b65e
26 _PyEval_EvalFrameDefault ceval.c 4772
0x7ffff7a75453
27 _PyEval_EvalFrame pycore_ceval.h 73
0x7ffff7a79887
28 _PyEval_Vector ceval.c 6435
0x7ffff7a799b1
29 _PyFunction_Vectorcall call.c 393
0x7ffff798b215
30 _PyObject_VectorcallTstate pycore_call.h 92
0x7ffff798b579
31 PyObject_CallOneArg call.c 376
0x7ffff798b6cd
32 call_show_warning _warnings.c 609
0x7ffff7a31d24
33 warn_explicit _warnings.c 746
0x7ffff7a3223b
34 do_warn _warnings.c 947
0x7ffff7a3236f
35 warn_unicode _warnings.c 1106
0x7ffff7a32414
36 _PyErr_WarnFormatV _warnings.c 1126
0x7ffff7a32481
37 PyErr_ResourceWarning _warnings.c 1177
0x7ffff7a32d38
38 sock_finalize socketmodule.c 5073
0x7ffff6840cf3
39 PyObject_CallFinalizer object.c 208
0x7ffff79d6449
40 PyObject_CallFinalizerFromDealloc object.c 226
0x7ffff79d75ac
41 subtype_dealloc typeobject.c 1382
0x7ffff79ecf81
42 _Py_Dealloc object.c 2390
0x7ffff79d6c38
43 Py_DECREF object.h 527
0x7ffff79c116e
44 Py_XDECREF object.h 602
0x7ffff79c118d
45 free_keys_object dictobject.c 664
0x7ffff79c120a
46 dictkeys_decref dictobject.c 324
0x7ffff79c130f
47 dict_dealloc dictobject.c 2378
0x7ffff79c2eeb
48 _Py_Dealloc object.c 2390
0x7ffff79d6c38
49 Py_DECREF object.h 527
0x7ffff79eb06b
50 subtype_dealloc typeobject.c 1438
0x7ffff79ed138
51 _Py_Dealloc object.c 2390
0x7ffff79d6c38
52 Py_DECREF object.h 527
0x7ffff7affa77
53 deque_clear _collectionsmodule.c 625
0x7ffff7b01e10
54 deque_dealloc _collectionsmodule.c 1311
0x7ffff7b030d9
55 _Py_Dealloc object.c 2390
0x7ffff79d6c38
56 Py_DECREF object.h 527
0x7ffff79c116e
57 Py_XDECREF object.h 602
0x7ffff79c118d
58 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
59 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
60 _Py_Dealloc object.c 2390
0x7ffff79d6c38
61 Py_DECREF object.h 527
0x7ffff79c116e
62 Py_XDECREF object.h 602
0x7ffff79c118d
63 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
64 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
65 _Py_Dealloc object.c 2390
0x7ffff79d6c38
66 Py_DECREF object.h 527
0x7ffff79c116e
67 Py_XDECREF object.h 602
0x7ffff79c118d
68 free_keys_object dictobject.c 672
0x7ffff79c127f
69 dictkeys_decref dictobject.c 324
0x7ffff79c130f
70 dict_dealloc dictobject.c 2378
0x7ffff79c2eeb
71 odict_dealloc odictobject.c 1364
0x7ffff79ccc8c
72 _Py_Dealloc object.c 2390
0x7ffff79d6c38
73 Py_DECREF object.h 527
0x7ffff79c116e
74 Py_XDECREF object.h 602
0x7ffff79c118d
75 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
76 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
77 _Py_Dealloc object.c 2390
0x7ffff79d6c38
78 Py_DECREF object.h 527
0x7ffff79c116e
79 Py_XDECREF object.h 602
0x7ffff79c118d
80 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
81 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
82 _Py_Dealloc object.c 2390
0x7ffff79d6c38
83 Py_DECREF object.h 527
0x7ffff79c116e
84 Py_XDECREF object.h 602
0x7ffff79c118d
85 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
86 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
87 _Py_Dealloc object.c 2390
0x7ffff79d6c38
88 Py_DECREF object.h 527
0x7ffff79c116e
89 Py_XDECREF object.h 602
0x7ffff79c118d
90 free_keys_object dictobject.c 664
0x7ffff79c120a
91 dictkeys_decref dictobject.c 324
0x7ffff79c130f
92 dict_dealloc dictobject.c 2378
0x7ffff79c2eeb
93 odict_dealloc odictobject.c 1364
0x7ffff79ccc8c
94 _Py_Dealloc object.c 2390
0x7ffff79d6c38
95 Py_DECREF object.h 527
0x7ffff79c116e
96 Py_XDECREF object.h 602
0x7ffff79c118d
97 _PyObject_FreeInstanceAttributes dictobject.c 5583
0x7ffff79ca68a
98 subtype_dealloc typeobject.c 1442
0x7ffff79ed18d
99 _Py_Dealloc object.c 2390
0x7ffff79d6c38
100 Py_DECREF object.h 527
0x7ffff79c116e
101 Py_XDECREF object.h 602
0x7ffff79c118d
102 free_keys_object dictobject.c 664
0x7ffff79c120a
103 dictkeys_decref dictobject.c 324
0x7ffff79c130f
104 dict_dealloc dictobject.c 2378
0x7ffff79c2eeb
105 _Py_Dealloc object.c 2390
0x7ffff79d6c38
106 Py_DECREF object.h 527
0x7ffff79c116e
107 delitem_common dictobject.c 1986
0x7ffff79c4deb
108 _PyDict_DelItem_KnownHash dictobject.c 2028
0x7ffff79c6c8b
109 PyDict_DelItem dictobject.c 2003
0x7ffff79c6d3a
110 _localdummy_destroyed _threadmodule.c 1044
0x7ffff7b31a85
111 cfunction_vectorcall_O methodobject.c 514
0x7ffff79d30f5
112 _PyObject_VectorcallTstate pycore_call.h 92
0x7ffff798b579
113 PyObject_CallOneArg call.c 376
0x7ffff798b6cd
114 handle_callback weakrefobject.c 931
0x7ffff7a300ab
115 PyObject_ClearWeakRefs weakrefobject.c 977
0x7ffff7a305cd
116 localdummy_dealloc _threadmodule.c 673
0x7ffff7b31f18
117 _Py_Dealloc object.c 2390
0x7ffff79d6c38
118 Py_DECREF object.h 527
0x7ffff79c116e
119 Py_XDECREF object.h 602
0x7ffff79c118d
120 free_keys_object dictobject.c 664
0x7ffff79c120a
121 dictkeys_decref dictobject.c 324
0x7ffff79c130f
122 dict_dealloc dictobject.c 2378
0x7ffff79c2eeb
123 _Py_Dealloc object.c 2390
0x7ffff79d6c38
124 Py_DECREF object.h 527
0x7ffff7aba1c9
125 PyThreadState_Clear pystate.c 1029
0x7ffff7abc790
126 PyGILState_Release pystate.c 1743
0x7ffff7abcef3
127 pythonactor_socket pythonactor.c 1021
0x555555709488
128 pythonactor_handle_msg pythonactor.c 1113
0x5555557098a5
129 pythonactor_handler pythonactor.c 1077
0x555555709716
130 sphactor_actor_run_once sphactor_actor.c 965
0x55555599bdfa
131 sphactor_actor_run sphactor_actor.c 1052
0x55555599c1c0
132 s_thread_shim zactor.c 68
0x5555559a97de
133 start_thread pthread_create.c 442
0x7ffff6fc8fd4
134 clone3 clone3.S 81
0x7ffff704966c
--
https://mail.python.org/mailman/listinfo/python-list

Re: PyGILState_Release called twice in embedded application [ In reply to ]

barry at barrys-emacs

Mar 23, 2023, 5:33 AM

Post #2 of 4 (85 views)

Permalink

> On 23 Mar 2023, at 08:46, Arnaud Loonstra <arnaud@sphaero.org> wrote:
>
> Hi all,
>
> I'm running in a crash due to a ResourceWarning (some socket is not closed in a used module) after calling PyGILState_Release.
>
> I'm running Python in a native thread (so a thread created by C not Python). I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon calls and releasing when finished using PyGILState_Release.
>
> This has worked fine. But now I'm using a python module (openai) which doesn't close its socket correctly which results in a ResourceWarning which triggers an assert.
>
> In the backtrace (below) I can see PyGILState_Release is called again. (7) while I've already called it (126).
>
> I can make the crash go away by adding
>
> import warnings
> warnings.simplefilter("ignore", ResourceWarning)
>
> to my python code. But I'd rather prevent this from happening in the first place.
>
> Any suggestion very welcomed cause I'm puzzled.

What 3rd party C extension are you running with?
I'd be surprised if the cpython code was the issue.

Barry

>
> Rg,
>
> Arnaud
>
> 1 __pthread_kill_implementation pthread_kill.c 44 0x7ffff6fcaccc
> 2 __pthread_kill_internal pthread_kill.c 78 0x7ffff6fcad2f
> 3 __GI_raise raise.c 26 0x7ffff6f7bef2
> 4 __GI_abort abort.c 79 0x7ffff6f66472
> 5 __assert_fail_base assert.c 92 0x7ffff6f66395
> 6 __GI___assert_fail assert.c 101 0x7ffff6f74df2
> 7 PyGILState_Release pystate.c 1742 0x7ffff7abcf9f
> 8 tracemalloc_raw_alloc _tracemalloc.c 779 0x7ffff7afa803
> 9 tracemalloc_raw_malloc _tracemalloc.c 789 0x7ffff7afa844
> 10 PyMem_RawMalloc obmalloc.c 586 0x7ffff79dab8d
> 11 decode_current_locale fileutils.c 472 0x7ffff7ada4ab
> 12 _Py_DecodeLocaleEx fileutils.c 598 0x7ffff7adafce
> 13 unicode_decode_locale unicodeobject.c 3970 0x7ffff7a13b65
> 14 PyUnicode_DecodeLocale unicodeobject.c 4012 0x7ffff7a18c79
> 15 PyErr_SetFromErrnoWithFilenameObjects errors.c 772 0x7ffff7a94178
> 16 PyErr_SetFromErrnoWithFilenameObject errors.c 750 0x7ffff7a942ce
> 17 posix_path_object_error posixmodule.c 1709 0x7ffff7ae4201
> 18 path_object_error posixmodule.c 1719 0x7ffff7ae420f
> 19 path_error posixmodule.c 1737 0x7ffff7ae4221
> 20 posix_do_stat posixmodule.c 2565 0x7ffff7af42f7
> 21 os_stat_impl posixmodule.c 2897 0x7ffff7af4491
> 22 os_stat posixmodule.c.h 71 0x7ffff7af45e5
> 23 cfunction_vectorcall_FASTCALL_KEYWORDS methodobject.c 443 0x7ffff79d2ece
> 24 _PyObject_VectorcallTstate pycore_call.h 92 0x7ffff798b579
> 25 PyObject_Vectorcall call.c 299 0x7ffff798b65e
> 26 _PyEval_EvalFrameDefault ceval.c 4772 0x7ffff7a75453
> 27 _PyEval_EvalFrame pycore_ceval.h 73 0x7ffff7a79887
> 28 _PyEval_Vector ceval.c 6435 0x7ffff7a799b1
> 29 _PyFunction_Vectorcall call.c 393 0x7ffff798b215
> 30 _PyObject_VectorcallTstate pycore_call.h 92 0x7ffff798b579
> 31 PyObject_CallOneArg call.c 376 0x7ffff798b6cd
> 32 call_show_warning _warnings.c 609 0x7ffff7a31d24
> 33 warn_explicit _warnings.c 746 0x7ffff7a3223b
> 34 do_warn _warnings.c 947 0x7ffff7a3236f
> 35 warn_unicode _warnings.c 1106 0x7ffff7a32414
> 36 _PyErr_WarnFormatV _warnings.c 1126 0x7ffff7a32481
> 37 PyErr_ResourceWarning _warnings.c 1177 0x7ffff7a32d38
> 38 sock_finalize socketmodule.c 5073 0x7ffff6840cf3
> 39 PyObject_CallFinalizer object.c 208 0x7ffff79d6449
> 40 PyObject_CallFinalizerFromDealloc object.c 226 0x7ffff79d75ac
> 41 subtype_dealloc typeobject.c 1382 0x7ffff79ecf81
> 42 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 43 Py_DECREF object.h 527 0x7ffff79c116e
> 44 Py_XDECREF object.h 602 0x7ffff79c118d
> 45 free_keys_object dictobject.c 664 0x7ffff79c120a
> 46 dictkeys_decref dictobject.c 324 0x7ffff79c130f
> 47 dict_dealloc dictobject.c 2378 0x7ffff79c2eeb
> 48 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 49 Py_DECREF object.h 527 0x7ffff79eb06b
> 50 subtype_dealloc typeobject.c 1438 0x7ffff79ed138
> 51 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 52 Py_DECREF object.h 527 0x7ffff7affa77
> 53 deque_clear _collectionsmodule.c 625 0x7ffff7b01e10
> 54 deque_dealloc _collectionsmodule.c 1311 0x7ffff7b030d9
> 55 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 56 Py_DECREF object.h 527 0x7ffff79c116e
> 57 Py_XDECREF object.h 602 0x7ffff79c118d
> 58 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 59 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 60 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 61 Py_DECREF object.h 527 0x7ffff79c116e
> 62 Py_XDECREF object.h 602 0x7ffff79c118d
> 63 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 64 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 65 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 66 Py_DECREF object.h 527 0x7ffff79c116e
> 67 Py_XDECREF object.h 602 0x7ffff79c118d
> 68 free_keys_object dictobject.c 672 0x7ffff79c127f
> 69 dictkeys_decref dictobject.c 324 0x7ffff79c130f
> 70 dict_dealloc dictobject.c 2378 0x7ffff79c2eeb
> 71 odict_dealloc odictobject.c 1364 0x7ffff79ccc8c
> 72 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 73 Py_DECREF object.h 527 0x7ffff79c116e
> 74 Py_XDECREF object.h 602 0x7ffff79c118d
> 75 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 76 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 77 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 78 Py_DECREF object.h 527 0x7ffff79c116e
> 79 Py_XDECREF object.h 602 0x7ffff79c118d
> 80 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 81 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 82 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 83 Py_DECREF object.h 527 0x7ffff79c116e
> 84 Py_XDECREF object.h 602 0x7ffff79c118d
> 85 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 86 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 87 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 88 Py_DECREF object.h 527 0x7ffff79c116e
> 89 Py_XDECREF object.h 602 0x7ffff79c118d
> 90 free_keys_object dictobject.c 664 0x7ffff79c120a
> 91 dictkeys_decref dictobject.c 324 0x7ffff79c130f
> 92 dict_dealloc dictobject.c 2378 0x7ffff79c2eeb
> 93 odict_dealloc odictobject.c 1364 0x7ffff79ccc8c
> 94 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 95 Py_DECREF object.h 527 0x7ffff79c116e
> 96 Py_XDECREF object.h 602 0x7ffff79c118d
> 97 _PyObject_FreeInstanceAttributes dictobject.c 5583 0x7ffff79ca68a
> 98 subtype_dealloc typeobject.c 1442 0x7ffff79ed18d
> 99 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 100 Py_DECREF object.h 527 0x7ffff79c116e
> 101 Py_XDECREF object.h 602 0x7ffff79c118d
> 102 free_keys_object dictobject.c 664 0x7ffff79c120a
> 103 dictkeys_decref dictobject.c 324 0x7ffff79c130f
> 104 dict_dealloc dictobject.c 2378 0x7ffff79c2eeb
> 105 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 106 Py_DECREF object.h 527 0x7ffff79c116e
> 107 delitem_common dictobject.c 1986 0x7ffff79c4deb
> 108 _PyDict_DelItem_KnownHash dictobject.c 2028 0x7ffff79c6c8b
> 109 PyDict_DelItem dictobject.c 2003 0x7ffff79c6d3a
> 110 _localdummy_destroyed _threadmodule.c 1044 0x7ffff7b31a85
> 111 cfunction_vectorcall_O methodobject.c 514 0x7ffff79d30f5
> 112 _PyObject_VectorcallTstate pycore_call.h 92 0x7ffff798b579
> 113 PyObject_CallOneArg call.c 376 0x7ffff798b6cd
> 114 handle_callback weakrefobject.c 931 0x7ffff7a300ab
> 115 PyObject_ClearWeakRefs weakrefobject.c 977 0x7ffff7a305cd
> 116 localdummy_dealloc _threadmodule.c 673 0x7ffff7b31f18
> 117 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 118 Py_DECREF object.h 527 0x7ffff79c116e
> 119 Py_XDECREF object.h 602 0x7ffff79c118d
> 120 free_keys_object dictobject.c 664 0x7ffff79c120a
> 121 dictkeys_decref dictobject.c 324 0x7ffff79c130f
> 122 dict_dealloc dictobject.c 2378 0x7ffff79c2eeb
> 123 _Py_Dealloc object.c 2390 0x7ffff79d6c38
> 124 Py_DECREF object.h 527 0x7ffff7aba1c9
> 125 PyThreadState_Clear pystate.c 1029 0x7ffff7abc790
> 126 PyGILState_Release pystate.c 1743 0x7ffff7abcef3
> 127 pythonactor_socket pythonactor.c 1021 0x555555709488
> 128 pythonactor_handle_msg pythonactor.c 1113 0x5555557098a5
> 129 pythonactor_handler pythonactor.c 1077 0x555555709716
> 130 sphactor_actor_run_once sphactor_actor.c 965 0x55555599bdfa
> 131 sphactor_actor_run sphactor_actor.c 1052 0x55555599c1c0
> 132 s_thread_shim zactor.c 68 0x5555559a97de
> 133 start_thread pthread_create.c 442 0x7ffff6fc8fd4
> 134 clone3 clone3.S 81 0x7ffff704966c
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
https://mail.python.org/mailman/listinfo/python-list

Re: PyGILState_Release called twice in embedded application [ In reply to ]

arnaud at sphaero

Mar 23, 2023, 7:34 AM

Post #3 of 4 (85 views)

Permalink

On 23-03-2023 13:33, Barry Scott wrote:
>
>
>> On 23 Mar 2023, at 08:46, Arnaud Loonstra <arnaud@sphaero.org> wrote:
>>
>> Hi all,
>>
>> I'm running in a crash due to a ResourceWarning (some socket is not closed in a used module) after calling PyGILState_Release.
>>
>> I'm running Python in a native thread (so a thread created by C not Python). I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon calls and releasing when finished using PyGILState_Release.
>>
>> This has worked fine. But now I'm using a python module (openai) which doesn't close its socket correctly which results in a ResourceWarning which triggers an assert.
>>
>> In the backtrace (below) I can see PyGILState_Release is called again. (7) while I've already called it (126).
>>
>> I can make the crash go away by adding
>>
>> import warnings
>> warnings.simplefilter("ignore", ResourceWarning)
>>
>> to my python code. But I'd rather prevent this from happening in the first place.
>>
>> Any suggestion very welcomed cause I'm puzzled.
>
> What 3rd party C extension are you running with?
> I'd be surprised if the cpython code was the issue.
>
> Barry

I'm not using any 3rd party extension myself. But the issue is caused by
the openai module using the requests module which is not closing sockets:

https://github.com/openai/openai-python/issues/140

I'm not aware what C extensions they might use.

Btw, I've tested this with python 3.8 and 3.11.2.

Rg,

Arnaud
--
https://mail.python.org/mailman/listinfo/python-list

Re: PyGILState_Release called twice in embedded application [ In reply to ]

barry at barrys-emacs

Mar 23, 2023, 10:28 AM

Post #4 of 4 (85 views)

Permalink

> On 23 Mar 2023, at 14:34, Arnaud Loonstra <arnaud@sphaero.org> wrote:
>
> ?On 23-03-2023 13:33, Barry Scott wrote:
>>>> On 23 Mar 2023, at 08:46, Arnaud Loonstra <arnaud@sphaero.org> wrote:
>>>
>>> Hi all,
>>>
>>> I'm running in a crash due to a ResourceWarning (some socket is not closed in a used module) after calling PyGILState_Release.
>>>
>>> I'm running Python in a native thread (so a thread created by C not Python). I'm acquiring the GIL through PyGILState_Ensure before doing any CPYthon calls and releasing when finished using PyGILState_Release.
>>>
>>> This has worked fine. But now I'm using a python module (openai) which doesn't close its socket correctly which results in a ResourceWarning which triggers an assert.
>>>
>>> In the backtrace (below) I can see PyGILState_Release is called again. (7) while I've already called it (126).
>>>
>>> I can make the crash go away by adding
>>>
>>> import warnings
>>> warnings.simplefilter("ignore", ResourceWarning)
>>>
>>> to my python code. But I'd rather prevent this from happening in the first place.
>>>
>>> Any suggestion very welcomed cause I'm puzzled.
>> What 3rd party C extension are you running with?
>> I'd be surprised if the cpython code was the issue.
>> Barry
>
> I'm not using any 3rd party extension myself. But the issue is caused by the openai module using the requests module which is not closing sockets:
>
> https://github.com/openai/openai-python/issues/140
>
> I'm not aware what C extensions they might use.

In gdb use info shared to see what is loaded.

Leaving a socket open will not cause a logic error as you are encountering.
I expect a bug in C/C++ cod3 that is not part of python itself.
You are using 3rd party code, requests and openai that you name.

Barry

>
> Btw, I've tested this with python 3.8 and 3.11.2.
>
> Rg,
>
> Arnaud
>

--
https://mail.python.org/mailman/listinfo/python-list