Mailing List Archive

Too many open files caused by repeated decryption operations via a GPGME API
Hi,

On modern Linux, we can change the maximum number of file descriptors per process in some ways. This feature is a well-known way for long time operation without reboot in cases such as server machines.

When I tried many iteration test of decryption via gpgme_op_decrypt() with the maximum number of file descriptors increased, "Too many open files" was caught from its result. At that time, the number of used file descriptors in the process did not reached to the limit I set.

The following error log was observed with the debug level of 5 enabled.

GPGME 2021-10-21 16:18:26 <0x047e>  gpgme_op_decrypt:493: error: Too many open files <GPGME>

I investigated where this error was caused in the source code, then found the following if statement to see if a value of file descriptor is bigger than FD_SETSIZE assumed the passed file descriptor is invalid and set the error code as EMFILE.

https://github.com/gpg/gpgme/blob/gpgme-1.10.0/src/posix-io.c#L673

The file descriptor seemed to be counted up to 1024 by repeating decryption operations. This value is derived from the upper limit of select(), but poll() is currently recommended on Linux and no limiting values are determined. This code using select() seems incompatible with the current file descriptor management on Linux.

My failure case occurred in v1.10.0, but the logic in master has the same if statement. For this reason, this problem will occur in the latest version.

https://github.com/gpg/gpgme/blob/master/src/posix-io.c#L725

One of the solution to avoid this issue would be to export the I/O callback functions to the caller side. I think we can avoid using the internal select() code and do appropriate synchronization I hope outside the gpgme library.

I referred to the following document.

https://www.gnupg.org/documentation/manuals/gpgme/I_002fO-Callback-Example.html#I_002fO-Callback-Example

However I feel that way is different from the original purpose of exporting the I/O callbacks, so it's just a workaround. If there is a more appropriate way to handle file descriptors more than the system default in GPGME, please let me know.

Regards,

--
Kazunori Kobayashi
EDGEMATRIX Inc.
https://edgematrix.com


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users
Re: Too many open files caused by repeated decryption operations via a GPGME API [ In reply to ]
On Fri, 29 Oct 2021 16:24, Kazunori Kobayashi said:

> On modern Linux, we can change the maximum number of file descriptors
> per process in some ways. This feature is a well-known way for long
> time operation without reboot in cases such as server machines.

That is a known problem we recently raise the priority of the feature
request https://dev.gnupg.org/T2385 to high. Thus there is a good
chance that this problem will be solved in the not too distant future.

> https://github.com/gpg/gpgme/blob/gpgme-1.10.0/src/posix-io.c#L673

Please don't refer to github for gnupg; their copy is out of date. Use
our https://dev.gnupg.org instead.


Salam-Shalom,

Werner

--
Die Gedanken sind frei. Ausnahmen regelt ein Bundesgesetz.
Re: Too many open files caused by repeated decryption operations via a GPGME API [ In reply to ]
On 2021/11/10 1:30, Werner Koch wrote:
> On Fri, 29 Oct 2021 16:24, Kazunori Kobayashi said:
>
>> On modern Linux, we can change the maximum number of file descriptors
>> per process in some ways. This feature is a well-known way for long
>> time operation without reboot in cases such as server machines.
> That is a known problem we recently raise the priority of the feature
> request https://dev.gnupg.org/T2385 to high. Thus there is a good
> chance that this problem will be solved in the not too distant future.
>
>> https://github.com/gpg/gpgme/blob/gpgme-1.10.0/src/posix-io.c#L673
> Please don't refer to github for gnupg; their copy is out of date. Use
> our https://dev.gnupg.org instead.
>
>
> Salam-Shalom,
>
> Werner
>
I appreciate your reply and I'll watch the issue tracker you told me.
I hope that this issue will be improved soon.

Kazunori

--
Kazunori Kobayashi
EDGEMATRIX Inc.
https://edgematrix.com


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users