Mailing List Archive

On 2022-09-29 16:54, Jen Kris via Python-list wrote:
> Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as before, but I am getting a segfault with random.seed.
>
> At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
Both PyUnicode_FromString and PyImport_Import return new references or
null pointers.

> if (pMod_random == 0x0){
> PyErr_Print();

You're leaking a reference here (pName_random).

> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>

You're leaking 2 references here (pName_random and pMod_random).

> return 0;
> }
>
> Later in the same program I call a C API program to call random.seed:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);

It's expecting all of the arguments to be PyObject*, but value_1 is
Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).

The argument list must end with a null pointer.

It returns a new reference or a null pointer.

>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
You're leaking a reference here (p_seed_calc).

> return return_val;
> }
>
> The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this segfault:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
> 459     ../Include/object.h: No such file or directory.
>
> So I tried Py_INCREF in the first program:
>
> Py_INCREF(pMod_random);
> Py_INCREF(pAttr_seed);
>
> Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault.
>
> Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.
>
> The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.
>
> I’m using Python 3.8 on Ubuntu.
>
> Thanks for any ideas on how to solve this.
>
> Jen
>

--
https://mail.python.org/mailman/listinfo/python-list

Thanks very much to @MRAB for taking time to answer.  I changed my code to conform to your answer (as best I understand your comments on references), but I still get the same error.  My comments continue below the new code immediately below. 

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = (PyObject * )value_1;
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_seed_calc);

return return_val;
}

So I incremented the reference to all objects in Get_LibModules, but I still get the same segfault at PyObject_CallFunctionObjArgs.  Unfortunately, reference counting is not well documented so I’m not clear what’s wrong. 




Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:

> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>
>> Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as before, but I am getting a segfault with random.seed.
>>
>> At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:
>>
>> int64_t Get_LibModules(int64_t * return_array)
>> {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> PyObject * pMod_random = PyImport_Import(pName_random);
>>
> Both PyUnicode_FromString and PyImport_Import return new references or null pointers.
>
>> if (pMod_random == 0x0){
>> PyErr_Print();
>>
>
> You're leaking a reference here (pName_random).
>
>> return 1;}
>>
>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
>>
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>
> You're leaking 2 references here (pName_random and pMod_random).
>
>> return 0;
>> }
>>
>> Later in the same program I call a C API program to call random.seed:
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>> {
>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>>
>
> It's expecting all of the arguments to be PyObject*, but value_1 is Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).
>
> The argument list must end with a null pointer.
>
> It returns a new reference or a null pointer.
>
>>
>> if (p_seed_calc == 0x0){
>>     PyErr_Print();
>>     return 1;}
>>
>> //Prepare return values
>> long return_val = PyLong_AsLong(p_seed_calc);
>>
> You're leaking a reference here (p_seed_calc).
>
>> return return_val;
>> }
>>
>> The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this segfault:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
>> 459     ../Include/object.h: No such file or directory.
>>
>> So I tried Py_INCREF in the first program:
>>
>> Py_INCREF(pMod_random);
>> Py_INCREF(pAttr_seed);
>>
>> Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault.
>>
>> Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.
>>
>> The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.
>>
>> I’m using Python 3.8 on Ubuntu.
>>
>> Thanks for any ideas on how to solve this.
>>
>> Jen
>>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
https://mail.python.org/mailman/listinfo/python-list

To update my previous email, I found the problem, but I have a new problem. 

Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct.  Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works.  HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is not the right answer for random.seed.  I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long. 

So my question is why do I get -1 as return value?  When I query p_seed calc : get:

(gdb) p p_seed_calc
$2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>

Thanks again.

Jen




Sep 29, 2022, 13:02 by python-list@python.org:

> Thanks very much to @MRAB for taking time to answer.  I changed my code to conform to your answer (as best I understand your comments on references), but I still get the same error.  My comments continue below the new code immediately below. 
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
> if (pMod_random == 0x0){
> PyErr_Print();
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
>
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = (PyObject * )value_1;
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> return return_val;
> }
>
> So I incremented the reference to all objects in Get_LibModules, but I still get the same segfault at PyObject_CallFunctionObjArgs.  Unfortunately, reference counting is not well documented so I’m not clear what’s wrong. 
>
>
>
>
> Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:
>
>> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>>
>>> Recently I completed a project where I used PyObject_CallFunctionObjArgs extensively with the NLTK library from a program written in NASM, with no problems.  Now I am on a new project where I call the Python random library.  I use the same setup as before, but I am getting a segfault with random.seed.
>>>
>>> At the start of the NASM program I call a C API program that gets PyObject pointers to “seed” and “randrange” in the same way as I did before:
>>>
>>> int64_t Get_LibModules(int64_t * return_array)
>>> {
>>> PyObject * pName_random = PyUnicode_FromString("random");
>>> PyObject * pMod_random = PyImport_Import(pName_random);
>>>
>> Both PyUnicode_FromString and PyImport_Import return new references or null pointers.
>>
>>> if (pMod_random == 0x0){
>>> PyErr_Print();
>>>
>>
>> You're leaking a reference here (pName_random).
>>
>>> return 1;}
>>>
>>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
>>>
>>> return_array[0] = (int64_t)pAttr_seed;
>>> return_array[1] = (int64_t)pAttr_randrange;
>>>
>>
>> You're leaking 2 references here (pName_random and pMod_random).
>>
>>> return 0;
>>> }
>>>
>>> Later in the same program I call a C API program to call random.seed:
>>>
>>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>>> {
>>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>>>
>>
>> It's expecting all of the arguments to be PyObject*, but value_1 is Py_ssize_t instead of PyObject* (a pointer to a _Python_ int).
>>
>> The argument list must end with a null pointer.
>>
>> It returns a new reference or a null pointer.
>>
>>>
>>> if (p_seed_calc == 0x0){
>>>     PyErr_Print();
>>>     return 1;}
>>>
>>> //Prepare return values
>>> long return_val = PyLong_AsLong(p_seed_calc);
>>>
>> You're leaking a reference here (p_seed_calc).
>>
>>> return return_val;
>>> }
>>>
>>> The first program correctly imports “random” and gets pointers to “seed” and “randrange.”  I verified that the same pointer is correctly passed into C_API_2, and the seed value (1234) is passed as  Py_ssize_t value_1.  But I get this segfault:
>>>
>>> Program received signal SIGSEGV, Segmentation fault.
>>> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at ../Include/object.h:459
>>> 459     ../Include/object.h: No such file or directory.
>>>
>>> So I tried Py_INCREF in the first program:
>>>
>>> Py_INCREF(pMod_random);
>>> Py_INCREF(pAttr_seed);
>>>
>>> Then I moved Py_INCREF(pAttr_seed) to the second program.  Same segfault.
>>>
>>> Finally, I initialized “random” and “seed” in the second program, where they are used.  Same segfault.
>>>
>>> The segfault refers to Py_INCREF, so this seems to do with reference counting, but Py_INCREF didn’t solve it.
>>>
>>> I’m using Python 3.8 on Ubuntu.
>>>
>>> Thanks for any ideas on how to solve this.
>>>
>>> Jen
>>>
>>
>> --
>> https://mail.python.org/mailman/listinfo/python-list
>>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
https://mail.python.org/mailman/listinfo/python-list

On 2022-09-29 21:47, Jen Kris wrote:
> To update my previous email, I found the problem, but I have a new
> problem.
>
> Previously I cast PyObject * value_ptr = (PyObject * )value_1 but
> that's not correct.  Instead I used PyObject * value_ptr =
> PyLong_FromLong(value_1) and that works.  HOWEVER, while
> PyObject_CallFunctionObjArgs does work now, it returns -1, which is
> not the right answer for random.seed.  I use "long return_val =
> PyLong_AsLong(p_seed_calc);" to convert it to a long.
>
random.seed returns None, so when you call PyObject_CallFunctionObjArgs
it returns a new reference to Py_None.

If you then pass to PyLong_AsLong a reference to something that's not a
PyLong, it'll set an error and return -1.

> So my question is why do I get -1 as return value?  When I query
> p_seed calc : get:
>
> (gdb) p p_seed_calc
> $2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>
>
Exactly. It's Py_None, not a PyLong.
> Thanks again.
>
> Jen
>
>
>
>
> Sep 29, 2022, 13:02 by python-list@python.org:
>
> Thanks very much to @MRAB for taking time to answer.  I changed my
> code to conform to your answer (as best I understand your comments
> on references), but I still get the same error.  My comments
> continue below the new code immediately below.
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
> if (pMod_random == 0x0){
> PyErr_Print();
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
>
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = (PyObject * )value_1;
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> return return_val;
> }
>
> So I incremented the reference to all objects in Get_LibModules,
> but I still get the same segfault at
> PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
> is not well documented so I’m not clear what’s wrong.
>
>
>
>
> Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:
>
> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>
> Recently I completed a project where I used
> PyObject_CallFunctionObjArgs extensively with the NLTK
> library from a program written in NASM, with no problems.
> Now I am on a new project where I call the Python random
> library.  I use the same setup as before, but I am getting
> a segfault with random.seed.
>
> At the start of the NASM program I call a C API program
> that gets PyObject pointers to “seed” and “randrange” in
> the same way as I did before:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Both PyUnicode_FromString and PyImport_Import return new
> references or null pointers.
>
> if (pMod_random == 0x0){
> PyErr_Print();
>
>
> You're leaking a reference here (pName_random).
>
> return 1;}
>
> PyObject * pAttr_seed =
> PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange =
> PyObject_GetAttrString(pMod_random, "randrange");
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
>
> You're leaking 2 references here (pName_random and pMod_random).
>
> return 0;
> }
>
> Later in the same program I call a C API program to call
> random.seed:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * p_seed_calc =
> PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>
>
> It's expecting all of the arguments to be PyObject*, but
> value_1 is Py_ssize_t instead of PyObject* (a pointer to a
> _Python_ int).
>
> The argument list must end with a null pointer.
>
> It returns a new reference or a null pointer.
>
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> You're leaking a reference here (p_seed_calc).
>
> return return_val;
> }
>
> The first program correctly imports “random” and gets
> pointers to “seed” and “randrange.”  I verified that the
> same pointer is correctly passed into C_API_2, and the
> seed value (1234) is passed as  Py_ssize_t value_1.  But I
> get this segfault:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
> ../Include/object.h:459
> 459     ../Include/object.h: No such file or directory.
>
> So I tried Py_INCREF in the first program:
>
> Py_INCREF(pMod_random);
> Py_INCREF(pAttr_seed);
>
> Then I moved Py_INCREF(pAttr_seed) to the second program. 
> Same segfault.
>
> Finally, I initialized “random” and “seed” in the second
> program, where they are used.  Same segfault.
>
> The segfault refers to Py_INCREF, so this seems to do with
> reference counting, but Py_INCREF didn’t solve it.
>
> I’m using Python 3.8 on Ubuntu.
>
> Thanks for any ideas on how to solve this.
>
> Jen
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
--
https://mail.python.org/mailman/listinfo/python-list

I just solved this C API problem, and I’m posting the answer to help anyone else who might need it. 

The errors were:

(1) we must call Py_INCREF on each object when it’s created.

(2) in C_API_2 (see below) we don’t cast value_1 as I did before with PyObject * value_ptr = (PyObject * )value_1.  Instead we use PyObject * value_ptr = PyLong_FromLong(value_1);

(3) The command string to PyObject_CallFunctionObjArgs must be null terminated.

Here’s the revised code:

First we load the modules, and increment the reference to each object: 

int64_t Get_LibModules(int64_t * return_array)
{
PyObject * pName_random = PyUnicode_FromString("random");
PyObject * pMod_random = PyImport_Import(pName_random);

Py_INCREF(pName_random);
Py_INCREF(pMod_random);

if (pMod_random == 0x0){
PyErr_Print();
return 1;}

PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");

Py_INCREF(pAttr_seed);
Py_INCREF(pAttr_randrange);

return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

Next we call a program to initialize the random number generator with random.seed(), and increment the reference to its return value p_seed_calc:

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);

// _________

if (p_seed_calc == 0x0){
    PyErr_Print();
    return 1;}

Py_INCREF(p_seed_calc);

return 0;
}

Now we call another program to get a random number:

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
{
PyObject * value_ptr = PyLong_FromLong(value_1);
PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);

if (p_randrange_calc == 0x0){
    PyErr_Print();
    return 1;}

//Prepare return values
long return_val = PyLong_AsLong(p_randrange_calc);

return return_val;
}

That returns 28, which is what I get from the Python command line. 

Thanks again to MRAB for helpful comments. 

Jen


Sep 29, 2022, 15:31 by python@mrabarnett.plus.com:

> On 2022-09-29 21:47, Jen Kris wrote:
>
>> To update my previous email, I found the problem, but I have a new problem.
>>
>> Previously I cast PyObject * value_ptr = (PyObject * )value_1 but that's not correct.  Instead I used PyObject * value_ptr = PyLong_FromLong(value_1) and that works.  HOWEVER, while PyObject_CallFunctionObjArgs does work now, it returns -1, which is not the right answer for random.seed.  I use "long return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
>>
> random.seed returns None, so when you call PyObject_CallFunctionObjArgs it returns a new reference to Py_None.
>
> If you then pass to PyLong_AsLong a reference to something that's not a PyLong, it'll set an error and return -1.
>
>> So my question is why do I get -1 as return value?  When I query p_seed calc : get:
>>
>> (gdb) p p_seed_calc
>> $2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>
>>
> Exactly. It's Py_None, not a PyLong.
>
>> Thanks again.
>>
>> Jen
>>
>>
>>
>>
>> Sep 29, 2022, 13:02 by python-list@python.org:
>>
>> Thanks very much to @MRAB for taking time to answer.  I changed my
>> code to conform to your answer (as best I understand your comments
>> on references), but I still get the same error.  My comments
>> continue below the new code immediately below.
>>
>> int64_t Get_LibModules(int64_t * return_array)
>> {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> PyObject * pMod_random = PyImport_Import(pName_random);
>>
>> Py_INCREF(pName_random);
>> Py_INCREF(pMod_random);
>>
>> if (pMod_random == 0x0){
>> PyErr_Print();
>> return 1;}
>>
>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>> "randrange");
>>
>> Py_INCREF(pAttr_seed);
>> Py_INCREF(pAttr_randrange);
>>
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>> return 0;
>> }
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>> {
>> PyObject * value_ptr = (PyObject * )value_1;
>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
>> value_ptr, NULL);
>>
>> if (p_seed_calc == 0x0){
>>     PyErr_Print();
>>     return 1;}
>>
>> //Prepare return values
>> long return_val = PyLong_AsLong(p_seed_calc);
>>
>> return return_val;
>> }
>>
>> So I incremented the reference to all objects in Get_LibModules,
>> but I still get the same segfault at
>> PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
>> is not well documented so I’m not clear what’s wrong.
>>
>>
>>
>>
>> Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:
>>
>> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>>
>> Recently I completed a project where I used
>> PyObject_CallFunctionObjArgs extensively with the NLTK
>> library from a program written in NASM, with no problems.
>> Now I am on a new project where I call the Python random
>> library.  I use the same setup as before, but I am getting
>> a segfault with random.seed.
>>
>> At the start of the NASM program I call a C API program
>> that gets PyObject pointers to “seed” and “randrange” in
>> the same way as I did before:
>>
>> int64_t Get_LibModules(int64_t * return_array)
>> {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> PyObject * pMod_random = PyImport_Import(pName_random);
>>
>> Both PyUnicode_FromString and PyImport_Import return new
>> references or null pointers.
>>
>> if (pMod_random == 0x0){
>> PyErr_Print();
>>
>>
>> You're leaking a reference here (pName_random).
>>
>> return 1;}
>>
>> PyObject * pAttr_seed =
>> PyObject_GetAttrString(pMod_random, "seed");
>> PyObject * pAttr_randrange =
>> PyObject_GetAttrString(pMod_random, "randrange");
>>
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>>
>> You're leaking 2 references here (pName_random and pMod_random).
>>
>> return 0;
>> }
>>
>> Later in the same program I call a C API program to call
>> random.seed:
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
>> {
>> PyObject * p_seed_calc =
>> PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>>
>>
>> It's expecting all of the arguments to be PyObject*, but
>> value_1 is Py_ssize_t instead of PyObject* (a pointer to a
>> _Python_ int).
>>
>> The argument list must end with a null pointer.
>>
>> It returns a new reference or a null pointer.
>>
>>
>> if (p_seed_calc == 0x0){
>>     PyErr_Print();
>>     return 1;}
>>
>> //Prepare return values
>> long return_val = PyLong_AsLong(p_seed_calc);
>>
>> You're leaking a reference here (p_seed_calc).
>>
>> return return_val;
>> }
>>
>> The first program correctly imports “random” and gets
>> pointers to “seed” and “randrange.”  I verified that the
>> same pointer is correctly passed into C_API_2, and the
>> seed value (1234) is passed as  Py_ssize_t value_1.  But I
>> get this segfault:
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
>> ../Include/object.h:459
>> 459     ../Include/object.h: No such file or directory.
>>
>> So I tried Py_INCREF in the first program:
>>
>> Py_INCREF(pMod_random);
>> Py_INCREF(pAttr_seed);
>>
>> Then I moved Py_INCREF(pAttr_seed) to the second program. 
>> Same segfault.
>>
>> Finally, I initialized “random” and “seed” in the second
>> program, where they are used.  Same segfault.
>>
>> The segfault refers to Py_INCREF, so this seems to do with
>> reference counting, but Py_INCREF didn’t solve it.
>>
>> I’m using Python 3.8 on Ubuntu.
>>
>> Thanks for any ideas on how to solve this.
>>
>> Jen
>>
>>
>> -- https://mail.python.org/mailman/listinfo/python-list
>>
>>
>> -- https://mail.python.org/mailman/listinfo/python-list
>>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
https://mail.python.org/mailman/listinfo/python-list

On 2022-09-29 23:41, Jen Kris wrote:
>
> I just solved this C API problem, and I’m posting the answer to help
> anyone else who might need it.
>
> The errors were:
>
> (1) we must call Py_INCREF on each object when it’s created.
>
Some functions return an object that has already been incref'ed ("new
reference"). This occurs when it has either created a new object (the
refcount will be 1) or has returned a pointer to an existing object (the
refcount will be > 1 because it has been incref'ed).

Other functions return an object that hasn't been incref'ed. This occurs
when you're looking up something, for example, looking at a member of a
list or the value of an attribute.

> (2) in C_API_2 (see below) we don’t cast value_1 as I did before with
> PyObject * value_ptr = (PyObject * )value_1.  Instead we use PyObject
> * value_ptr = PyLong_FromLong(value_1);
>
> (3) The command string to PyObject_CallFunctionObjArgs must be null
> terminated.
>
Always read the docs carefully!
> Here’s the revised code:
>
> First we load the modules, and increment the reference to each object:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
Possible null pointers here.
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
Py_INCREF will fail on a null pointer, which you haven't check for yet.

If they aren't null, then they're new references (incref'ed) that you've
incref'ed.

> if (pMod_random == 0x0){
> PyErr_Print();
Leaks here because of the refcount.
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
>
Possible null pointers here too.
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
Same issue as above.
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> Next we call a program to initialize the random number generator with
> random.seed(), and increment the reference to its return value
> p_seed_calc:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = PyLong_FromLong(value_1);
New reference.
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
>
> // _________
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
Leak.
>     return 1;}
>
> Py_INCREF(p_seed_calc);
>
p_seed_calc will be a new reference to Py_None, incref'ed again, so leak.
> return 0;
> }
>
> Now we call another program to get a random number:
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1)
> {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> PyObject * p_randrange_calc =
> PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
>
> if (p_randrange_calc == 0x0){
Leak.
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_randrange_calc);
>
Leak.
> return return_val;
> }
>
> That returns 28, which is what I get from the Python command line.
>
> Thanks again to MRAB for helpful comments.
>
> Jen
>
Getting refcounting right can be difficult!
>
> Sep 29, 2022, 15:31 by python@mrabarnett.plus.com:
>
> On 2022-09-29 21:47, Jen Kris wrote:
>
> To update my previous email, I found the problem, but I have a
> new problem.
>
> Previously I cast PyObject * value_ptr = (PyObject * )value_1
> but that's not correct.  Instead I used PyObject * value_ptr =
> PyLong_FromLong(value_1) and that works. HOWEVER, while
> PyObject_CallFunctionObjArgs does work now, it returns -1,
> which is not the right answer for random.seed.  I use "long
> return_val = PyLong_AsLong(p_seed_calc);" to convert it to a long.
>
> random.seed returns None, so when you call
> PyObject_CallFunctionObjArgs it returns a new reference to Py_None.
>
> If you then pass to PyLong_AsLong a reference to something that's
> not a PyLong, it'll set an error and return -1.
>
> So my question is why do I get -1 as return value?  When I
> query p_seed calc : get:
>
> (gdb) p p_seed_calc
> $2 = (PyObject *) 0x7ffff69be120 <_Py_NoneStruct>
>
> Exactly. It's Py_None, not a PyLong.
>
> Thanks again.
>
> Jen
>
>
>
>
> Sep 29, 2022, 13:02 by python-list@python.org:
>
> Thanks very much to @MRAB for taking time to answer.  I changed my
> code to conform to your answer (as best I understand your comments
> on references), but I still get the same error.  My comments
> continue below the new code immediately below.
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Py_INCREF(pName_random);
> Py_INCREF(pMod_random);
>
> if (pMod_random == 0x0){
> PyErr_Print();
> return 1;}
>
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random,
> "seed");
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
>
> Py_INCREF(pAttr_seed);
> Py_INCREF(pAttr_randrange);
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * value_ptr = (PyObject * )value_1;
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> return return_val;
> }
>
> So I incremented the reference to all objects in Get_LibModules,
> but I still get the same segfault at
> PyObject_CallFunctionObjArgs.  Unfortunately, reference counting
> is not well documented so I’m not clear what’s wrong.
>
>
>
>
> Sep 29, 2022, 10:06 by python@mrabarnett.plus.com:
>
> On 2022-09-29 16:54, Jen Kris via Python-list wrote:
>
> Recently I completed a project where I used
> PyObject_CallFunctionObjArgs extensively with the NLTK
> library from a program written in NASM, with no problems.
> Now I am on a new project where I call the Python random
> library.  I use the same setup as before, but I am getting
> a segfault with random.seed.
>
> At the start of the NASM program I call a C API program
> that gets PyObject pointers to “seed” and “randrange” in
> the same way as I did before:
>
> int64_t Get_LibModules(int64_t * return_array)
> {
> PyObject * pName_random = PyUnicode_FromString("random");
> PyObject * pMod_random = PyImport_Import(pName_random);
>
> Both PyUnicode_FromString and PyImport_Import return new
> references or null pointers.
>
> if (pMod_random == 0x0){
> PyErr_Print();
>
>
> You're leaking a reference here (pName_random).
>
> return 1;}
>
> PyObject * pAttr_seed =
> PyObject_GetAttrString(pMod_random, "seed");
> PyObject * pAttr_randrange =
> PyObject_GetAttrString(pMod_random, "randrange");
>
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
>
> You're leaking 2 references here (pName_random and pMod_random).
>
> return 0;
> }
>
> Later in the same program I call a C API program to call
> random.seed:
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1)
> {
> PyObject * p_seed_calc =
> PyObject_CallFunctionObjArgs(pAttr_seed, value_1);
>
>
> It's expecting all of the arguments to be PyObject*, but
> value_1 is Py_ssize_t instead of PyObject* (a pointer to a
> _Python_ int).
>
> The argument list must end with a null pointer.
>
> It returns a new reference or a null pointer.
>
>
> if (p_seed_calc == 0x0){
>     PyErr_Print();
>     return 1;}
>
> //Prepare return values
> long return_val = PyLong_AsLong(p_seed_calc);
>
> You're leaking a reference here (p_seed_calc).
>
> return return_val;
> }
>
> The first program correctly imports “random” and gets
> pointers to “seed” and “randrange.”  I verified that the
> same pointer is correctly passed into C_API_2, and the
> seed value (1234) is passed as  Py_ssize_t value_1.  But I
> get this segfault:
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff64858d5 in _Py_INCREF (op=0x4d2) at
> ../Include/object.h:459
> 459     ../Include/object.h: No such file or directory.
>
> So I tried Py_INCREF in the first program:
>
> Py_INCREF(pMod_random);
> Py_INCREF(pAttr_seed);
>
> Then I moved Py_INCREF(pAttr_seed) to the second program.
> Same segfault.
>
> Finally, I initialized “random” and “seed” in the second
> program, where they are used.  Same segfault.
>
> The segfault refers to Py_INCREF, so this seems to do with
> reference counting, but Py_INCREF didn’t solve it.
>
> I’m using Python 3.8 on Ubuntu.
>
> Thanks for any ideas on how to solve this.
>
> Jen
>
>
> -- https://mail.python.org/mailman/listinfo/python-list
>
>
> -- https://mail.python.org/mailman/listinfo/python-list
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
--
https://mail.python.org/mailman/listinfo/python-list

On 2022-09-30 01:02, MRAB wrote:
> On 2022-09-29 23:41, Jen Kris wrote:
>>
>> I just solved this C API problem, and I’m posting the answer to help
>> anyone else who might need it.
>>
[snip]

What I like to do is write comments that state which variables hold a
reference, followed by '+' if it's a new reference (incref'ed) and '?'
if it could be null. '+?' means that it's probably a new reference but
could be null. Once I know that it's not null, I can remove the '?', and
once I've decref'ed it (if required) and no longer need it, I remobe it
from the comment.

Clearing up references, as soon as they're not needed, helps to keep the
number of current references more manageable.


int64_t Get_LibModules(int64_t * return_array) {
PyObject * pName_random = PyUnicode_FromString("random");
//> pName_random+?
if (!pName_random) {
PyErr_Print();
return 1;
}

//> pName_random+
PyObject * pMod_random = PyImport_Import(pName_random);
//> pName_random+ pMod_random+?
Py_DECREF(pName_random);
//> pMod_random+?
if (!pMod_random) {
PyErr_Print();
return 1;
}

//> pMod_random+
PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
//> pMod_random+ pAttr_seed?
if (!pAttr_seed) {
Py_DECREF(pMod_random);
PyErr_Print();
return 1;
}

//> pMod_random+ pAttr_seed
PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
"randrange");
//> pMod_random+ pAttr_seed pAttr_randrange?
Py_DECREF(pMod_random);
//> pAttr_seed pAttr_randrange?
if (!pAttr_randrange) {
PyErr_Print();
return 1;
}

//> pAttr_seed pAttr_randrange
return_array[0] = (int64_t)pAttr_seed;
return_array[1] = (int64_t)pAttr_randrange;

return 0;
}

int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
value_ptr, NULL);
//> value_ptr+ p_seed_calc+?
Py_DECREF(value_ptr);
//> p_seed_calc+?
if (!p_seed_calc) {
PyErr_Print();
return 1;
}

//> p_seed_calc+
Py_DECREF(p_seed_calc);
return 0;
}

int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
PyObject * value_ptr = PyLong_FromLong(value_1);
//> value_ptr+?
if (!value_ptr) {
PyErr_Print();
return 1;
}

//> value_ptr+
PyObject * p_randrange_calc =
PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
//> value_ptr+ p_randrange_calc+?
Py_DECREF(value_ptr);
//> p_randrange_calc+?
if (!p_randrange_calc) {
PyErr_Print();
return 1;
}

//Prepare return values
//> p_randrange_calc+
return_val = PyLong_AsLong(p_randrange_calc);
Py_DECREF(p_randrange_calc);

return return_val;
}

--
https://mail.python.org/mailman/listinfo/python-list

Thanks very much for your detailed reply.  I have a few followup questions. 

You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because it has been incref'ed).  Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.” 

In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished with it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is no reason to leave them out.”  Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help. 

Also: 

What is a borrowed reference, and how does it effect reference counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count on the object, and obligates you to dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.” 

What does it mean to steal a reference?  If a function steals a reference does it have to decref it without incref (because it’s stolen)?

Finally, you said:

if (pMod_random == 0x0){
    PyErr_Print();
Leaks here because of the refcount

Assuming pMod_random is not null, why would this leak? 

Thanks again for your input on this question. 

Jen



Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:

> On 2022-09-30 01:02, MRAB wrote:
>
>> On 2022-09-29 23:41, Jen Kris wrote:
>>
>>>
>>> I just solved this C API problem, and I’m posting the answer to help anyone else who might need it.
>>>
> [snip]
>
> What I like to do is write comments that state which variables hold a reference, followed by '+' if it's a new reference (incref'ed) and '?' if it could be null. '+?' means that it's probably a new reference but could be null. Once I know that it's not null, I can remove the '?', and once I've decref'ed it (if required) and no longer need it, I remobe it from the comment.
>
> Clearing up references, as soon as they're not needed, helps to keep the number of current references more manageable.
>
>
> int64_t Get_LibModules(int64_t * return_array) {
> PyObject * pName_random = PyUnicode_FromString("random");
> //> pName_random+?
> if (!pName_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pName_random+
> PyObject * pMod_random = PyImport_Import(pName_random);
> //> pName_random+ pMod_random+?
> Py_DECREF(pName_random);
> //> pMod_random+?
> if (!pMod_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> //> pMod_random+ pAttr_seed?
> if (!pAttr_seed) {
> Py_DECREF(pMod_random);
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+ pAttr_seed
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random, "randrange");
> //> pMod_random+ pAttr_seed pAttr_randrange?
> Py_DECREF(pMod_random);
> //> pAttr_seed pAttr_randrange?
> if (!pAttr_randrange) {
> PyErr_Print();
> return 1;
> }
>
> //> pAttr_seed pAttr_randrange
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed, value_ptr, NULL);
> //> value_ptr+ p_seed_calc+?
> Py_DECREF(value_ptr);
> //> p_seed_calc+?
> if (!p_seed_calc) {
> PyErr_Print();
> return 1;
> }
>
> //> p_seed_calc+
> Py_DECREF(p_seed_calc);
> return 0;
> }
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_randrange_calc = PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
> //> value_ptr+ p_randrange_calc+?
> Py_DECREF(value_ptr);
> //> p_randrange_calc+?
> if (!p_randrange_calc) {
> PyErr_Print();
> return 1;
> }
>
> //Prepare return values
> //> p_randrange_calc+
> return_val = PyLong_AsLong(p_randrange_calc);
> Py_DECREF(p_randrange_calc);
>
> return return_val;
> }
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>

--
https://mail.python.org/mailman/listinfo/python-list

On 2022-09-30 17:02, Jen Kris wrote:
>
> Thanks very much for your detailed reply.  I have a few followup
> questions.
>
> You said, “Some functions return an object that has already been
> incref'ed ("new reference"). This occurs when it has either created a
> new object (the refcount will be 1) or has returned a pointer to an
> existing object (the refcount will be > 1 because it has been
> incref'ed).  Other functions return an object that hasn't been
> incref'ed. This occurs when you're looking up something, for example,
> looking at a member of a list or the value of an attribute.”
>
> In the official docs some functions show “Return value: New reference”
> and others do not.  Is there any reason why I should not just INCREF
> on every new object, regardless of whether it’s a new reference or
> not, and DECREF when I am finished with it?  The answer at
> https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to
> says “With out-of-order execution, the INCREF/DECREF are basically
> free operations, so performance is no reason to leave them out.” 
> Doing so means I don’t have to check each object to see if it needs to
> be INCREF’d or not, and that is a big help.
>
It's OK to INCREF them, provided that you DECREF them when you no longer
need them, and remember that if it's a "new reference" you'd need to
DECREF it twice.
> Also:
>
> What is a borrowed reference, and how does it effect reference
> counting?  According to
> https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use
> Py_INCREF on a borrowed PyObject pointer you already have. This
> increments the reference count on the object, and obligates you to
> dispose of it properly.”  So I guess it’s yes, but I’m confused by
> “pointer you already have.”

A borrowed reference is when it hasn't been INCREFed.

You can think of INCREFing as a way of indicating ownership, which is
often shared ownership (refcount > 1). When you're borrowing a
reference, you're using it temporarily, but not claiming ownership. When
the last owner releases its ownership (DECREF reduces the refcount to
0), the object can be garbage collected.

When, say, you lookup an attribute, or get an object from a list with
PyList_GetItem, it won't have been INCREFed. You're using it
temporarily, just borrowing a reference.
>
> What does it mean to steal a reference?  If a function steals a
> reference does it have to decref it without incref (because it’s stolen)?
When function steals a reference, it's claiming ownership but not
INCREFing it.
>
> Finally, you said:
>
> if (pMod_random == 0x0){
>     PyErr_Print();
> Leaks here because of the refcount
>
> Assuming pMod_random is not null, why would this leak?
>
It's pName_random that's the leak.

PyUnicode_FromString("random") will either create and return a new
object for the string "random" (refcount == 1) or return a reference to
an existing object (refcount > 1). You need to DECREF it before
returning from the function.

Suppose it created a new object. You call the function, it creates an
object, you use it, then return from the function. The object still
exists, but there's no reference to it. Now call the function again. It
creates another object, you use it, then return from the function. You
now have 2 objects with no reference to them.

> Thanks again for your input on this question.
>
> Jen
>
>
>
> Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:
>
> On 2022-09-30 01:02, MRAB wrote:
>
> On 2022-09-29 23:41, Jen Kris wrote:
>
>
> I just solved this C API problem, and I’m posting the
> answer to help anyone else who might need it.
>
> [snip]
>
> What I like to do is write comments that state which variables
> hold a reference, followed by '+' if it's a new reference
> (incref'ed) and '?' if it could be null. '+?' means that it's
> probably a new reference but could be null. Once I know that it's
> not null, I can remove the '?', and once I've decref'ed it (if
> required) and no longer need it, I remobe it from the comment.
>
> Clearing up references, as soon as they're not needed, helps to
> keep the number of current references more manageable.
>
>
> int64_t Get_LibModules(int64_t * return_array) {
> PyObject * pName_random = PyUnicode_FromString("random");
> //> pName_random+?
> if (!pName_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pName_random+
> PyObject * pMod_random = PyImport_Import(pName_random);
> //> pName_random+ pMod_random+?
> Py_DECREF(pName_random);
> //> pMod_random+?
> if (!pMod_random) {
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+
> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
> //> pMod_random+ pAttr_seed?
> if (!pAttr_seed) {
> Py_DECREF(pMod_random);
> PyErr_Print();
> return 1;
> }
>
> //> pMod_random+ pAttr_seed
> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
> "randrange");
> //> pMod_random+ pAttr_seed pAttr_randrange?
> Py_DECREF(pMod_random);
> //> pAttr_seed pAttr_randrange?
> if (!pAttr_randrange) {
> PyErr_Print();
> return 1;
> }
>
> //> pAttr_seed pAttr_randrange
> return_array[0] = (int64_t)pAttr_seed;
> return_array[1] = (int64_t)pAttr_randrange;
>
> return 0;
> }
>
> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
> value_ptr, NULL);
> //> value_ptr+ p_seed_calc+?
> Py_DECREF(value_ptr);
> //> p_seed_calc+?
> if (!p_seed_calc) {
> PyErr_Print();
> return 1;
> }
>
> //> p_seed_calc+
> Py_DECREF(p_seed_calc);
> return 0;
> }
>
> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
> PyObject * value_ptr = PyLong_FromLong(value_1);
> //> value_ptr+?
> if (!value_ptr) {
> PyErr_Print();
> return 1;
> }
>
> //> value_ptr+
> PyObject * p_randrange_calc =
> PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
> //> value_ptr+ p_randrange_calc+?
> Py_DECREF(value_ptr);
> //> p_randrange_calc+?
> if (!p_randrange_calc) {
> PyErr_Print();
> return 1;
> }
>
> //Prepare return values
> //> p_randrange_calc+
> return_val = PyLong_AsLong(p_randrange_calc);
> Py_DECREF(p_randrange_calc);
>
> return return_val;
> }
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
>
--
https://mail.python.org/mailman/listinfo/python-list

That's great.  It clarifies things a lot for me, particularly re ref count for new references.  I would have had trouble if I didn't decref it twice. 

Thanks very much once again. 


Sep 30, 2022, 12:18 by python@mrabarnett.plus.com:

> On 2022-09-30 17:02, Jen Kris wrote:
>
>>
>> Thanks very much for your detailed reply.  I have a few followup questions.
>>
>> You said, “Some functions return an object that has already been incref'ed ("new reference"). This occurs when it has either created a new object (the refcount will be 1) or has returned a pointer to an existing object (the refcount will be > 1 because it has been incref'ed).  Other functions return an object that hasn't been incref'ed. This occurs when you're looking up something, for example, looking at a member of a list or the value of an attribute.”
>>
>> In the official docs some functions show “Return value: New reference” and others do not.  Is there any reason why I should not just INCREF on every new object, regardless of whether it’s a new reference or not, and DECREF when I am finished with it?  The answer at https://stackoverflow.com/questions/59870703/python-c-extension-need-to-py-incref-a-borrowed-reference-if-not-returning-it-to says “With out-of-order execution, the INCREF/DECREF are basically free operations, so performance is no reason to leave them out.”  Doing so means I don’t have to check each object to see if it needs to be INCREF’d or not, and that is a big help.
>>
> It's OK to INCREF them, provided that you DECREF them when you no longer need them, and remember that if it's a "new reference" you'd need to DECREF it twice.
>
>> Also:
>>
>> What is a borrowed reference, and how does it effect reference counting?  According to https://jayrambhia.com/blog/pythonc-api-reference-counting, “Use Py_INCREF on a borrowed PyObject pointer you already have. This increments the reference count on the object, and obligates you to dispose of it properly.”  So I guess it’s yes, but I’m confused by “pointer you already have.”
>>
>
> A borrowed reference is when it hasn't been INCREFed.
>
> You can think of INCREFing as a way of indicating ownership, which is often shared ownership (refcount > 1). When you're borrowing a reference, you're using it temporarily, but not claiming ownership. When the last owner releases its ownership (DECREF reduces the refcount to 0), the object can be garbage collected.
>
> When, say, you lookup an attribute, or get an object from a list with PyList_GetItem, it won't have been INCREFed. You're using it temporarily, just borrowing a reference.
>
>>
>> What does it mean to steal a reference?  If a function steals a reference does it have to decref it without incref (because it’s stolen)?
>>
> When function steals a reference, it's claiming ownership but not INCREFing it.
>
>>
>> Finally, you said:
>>
>> if (pMod_random == 0x0){
>>     PyErr_Print();
>> Leaks here because of the refcount
>>
>> Assuming pMod_random is not null, why would this leak?
>>
> It's pName_random that's the leak.
>
> PyUnicode_FromString("random") will either create and return a new object for the string "random" (refcount == 1) or return a reference to an existing object (refcount > 1). You need to DECREF it before returning from the function.
>
> Suppose it created a new object. You call the function, it creates an object, you use it, then return from the function. The object still exists, but there's no reference to it. Now call the function again. It creates another object, you use it, then return from the function. You now have 2 objects with no reference to them.
>
>> Thanks again for your input on this question.
>>
>> Jen
>>
>>
>>
>> Sep 29, 2022, 17:33 by python@mrabarnett.plus.com:
>>
>> On 2022-09-30 01:02, MRAB wrote:
>>
>> On 2022-09-29 23:41, Jen Kris wrote:
>>
>>
>> I just solved this C API problem, and I’m posting the
>> answer to help anyone else who might need it.
>>
>> [snip]
>>
>> What I like to do is write comments that state which variables
>> hold a reference, followed by '+' if it's a new reference
>> (incref'ed) and '?' if it could be null. '+?' means that it's
>> probably a new reference but could be null. Once I know that it's
>> not null, I can remove the '?', and once I've decref'ed it (if
>> required) and no longer need it, I remobe it from the comment.
>>
>> Clearing up references, as soon as they're not needed, helps to
>> keep the number of current references more manageable.
>>
>>
>> int64_t Get_LibModules(int64_t * return_array) {
>> PyObject * pName_random = PyUnicode_FromString("random");
>> //> pName_random+?
>> if (!pName_random) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> pName_random+
>> PyObject * pMod_random = PyImport_Import(pName_random);
>> //> pName_random+ pMod_random+?
>> Py_DECREF(pName_random);
>> //> pMod_random+?
>> if (!pMod_random) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> pMod_random+
>> PyObject * pAttr_seed = PyObject_GetAttrString(pMod_random, "seed");
>> //> pMod_random+ pAttr_seed?
>> if (!pAttr_seed) {
>> Py_DECREF(pMod_random);
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> pMod_random+ pAttr_seed
>> PyObject * pAttr_randrange = PyObject_GetAttrString(pMod_random,
>> "randrange");
>> //> pMod_random+ pAttr_seed pAttr_randrange?
>> Py_DECREF(pMod_random);
>> //> pAttr_seed pAttr_randrange?
>> if (!pAttr_randrange) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> pAttr_seed pAttr_randrange
>> return_array[0] = (int64_t)pAttr_seed;
>> return_array[1] = (int64_t)pAttr_randrange;
>>
>> return 0;
>> }
>>
>> int64_t C_API_2(PyObject * pAttr_seed, Py_ssize_t value_1) {
>> PyObject * value_ptr = PyLong_FromLong(value_1);
>> //> value_ptr+?
>> if (!!value_ptr) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> value_ptr+
>> PyObject * p_seed_calc = PyObject_CallFunctionObjArgs(pAttr_seed,
>> value_ptr, NULL);
>> //> value_ptr+ p_seed_calc+?
>> Py_DECREF(value_ptr);
>> //> p_seed_calc+?
>> if (!p_seed_calc) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> p_seed_calc+
>> Py_DECREF(p_seed_calc);
>> return 0;
>> }
>>
>> int64_t C_API_12(PyObject * pAttr_randrange, Py_ssize_t value_1) {
>> PyObject * value_ptr = PyLong_FromLong(value_1);
>> //> value_ptr+?
>> if (!value_ptr) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //> value_ptr+
>> PyObject * p_randrange_calc =
>> PyObject_CallFunctionObjArgs(pAttr_randrange, value_ptr, NULL);
>> //> value_ptr+ p_randrange_calc+?
>> Py_DECREF(value_ptr);
>> //> p_randrange_calc+?
>> if (!p_randrange_calc) {
>> PyErr_Print();
>> return 1;
>> }
>>
>> //Prepare return values
>> //> p_randrange_calc+
>> return_val = PyLong_AsLong(p_randrange_calc);
>> Py_DECREF(p_randrange_calc);
>>
>> return return_val;
>> }
>>
>> -- https://mail.python.org/mailman/listinfo/python-list
>>

--
https://mail.python.org/mailman/listinfo/python-list

On 1/10/22 8:18 am, MRAB wrote:
> It's OK to INCREF them, provided that you DECREF them when you no longer
> need them, and remember that if it's a "new reference" you'd need to
> DECREF it twice.

Which means there would usually be no point in doing the extra
INCREF/DECREF. You still need to know whether it's a new reference
or not and treat it accordingly.

--
Greg

--
https://mail.python.org/mailman/listinfo/python-list