Mailing List Archive

Fatal error: out of core in secure memory - during decrypt function call
Greetings all!

We are trying to develop a sample program in ANSI C running on Linux to understand how we can use libgcrypt functions to encrypt data via a web interface for secure storage. I have spent days searching through the libgcrypt documentation and the web, trying to find a reason/solution for this 'out of core' problem, but came up empty.

I was having the same problems as Spencer Ogden
(http://lists.gnupg.org/pipermail/gcrypt-devel/2003-May/000276.html)
with the sexp_sscan delivering an error 201. To get around it, I've emulated the testsexp.c example and used the gcry_sexp_build function to generate an rsa key pair, encapsulate a short string into an sexp, and encrypt the sexp. So far all is good.

The problem is when I try to decrypt the encrypted sexp, I get an 'out of core in secure memory' fatal error. This occurs whether running as root or not. Everyone in our shop is quite novice at using encryption software (why the senior engineers gave me the project, I suppose) but it doesn't make sense to me why the gcry_pki_decrypt function should fail if I give it a freshly created sKey, an encrypted sexp and a valid sexp* -- I'm stumped.

*****************************
We're running on:
*****************************
Red Hat Linux 7.2
Kernel 2.4.9-13smp on an i686
(Not sure if you need any other info)




*****************************
Source code follows...
*****************************


/* Sample libgcryp Program - v0.1 */
/* 08 MAY 03 */

#include <stdio.h>
#include "/usr/local/include/gcrypt.h"


int main (int argc, char *argv[])
{

GcrySexp data2;
char encryptMe[] = "";
const char *decrypted;
char name[20];
char fkey[3000];
FILE *fp;
char pubkey[]="public-key";
char secKey[]="private-key";
GcrySexp PARMS, result, data;
GCRY_MPI mData;
GcrySexp pKey, sKey, Key;
size_t n;
int i, rcode, nbits = 1024;

memset( &data2, 0, sizeof( data2 ) );
memset( &data, 0, sizeof( data ) );
memset( &PARMS, 0, sizeof( PARMS ) );
memset( &result, 0, sizeof( result ) );

printf( "Enter data to encode:" );
scanf( "%s", encryptMe );
printf( "data entered: %s", encryptMe);
n=strlen(encryptMe);

gcry_mpi_scan( &mData, GCRYMPI_FMT_HEX, encryptMe, NULL);
printf( "\n\n" );

gcry_control( GCRYCTL_INIT_SECMEM, 32768, 0 );

rcode = gcry_sexp_build (&data, NULL, "(data(flags raw)(value %d))", mData);
printf("\nreturn value for data sexp generation is [%d]", rcode);
printf( "\n" );


rcode = gcry_sexp_build (&PARMS, NULL, "(genkey(rsa(nbits %d)))", nbits);
printf("return code for sexp_new(PARMS) is [%d]", rcode);
printf("\n\n");

rcode = gcry_pk_genkey (&Key, PARMS);
printf("return code for genkey is [%d]", rcode);


printf("\n");
if (!rcode)
{gcry_sexp_release( PARMS );}

pKey = gcry_sexp_find_token(Key, pubkey, strlen(pubkey));
sKey = gcry_sexp_find_token(Key, secKey, strlen(secKey));
printf( "\n\n" );
if (sKey&&pKey)
{gcry_sexp_release( Key );}
printf( "\n\n" );

printf( "\n\n" );
rcode = gcry_pk_testkey (sKey);
printf("return code for testkey is [%d]", rcode);

rcode = gcry_pk_encrypt (&result, data, pKey);
printf("Return value for encryption with pKey is [%d]", rcode);
printf( "\n\n" );

rcode = gcry_pk_decrypt (&data2, result, sKey); <-----Fatal error occurs here
printf("Return value for decryption: [%d]", rcode); out of core in secure memory
if (!rcode)
{gcry_sexp_dump (data2);}
printf( "\n\n" );

decrypted = gcry_sexp_nth_data(data, 2, &n);
printf( "\n\nDecrypted data: %s", decrypted );
printf( "\n\n" );
gcry_sexp_dump (data);

printf( "\n\n" );

/*************************************************
// Cleaning House - memory-wise...
*/
if (Key) { gcry_sexp_release( Key ); }
if (result) { gcry_sexp_release( result ); }
if (data2) { gcry_sexp_release( data2 ); }
if (sKey) { gcry_sexp_release( sKey ); }
if (pKey) { gcry_sexp_release( pKey ); }
if (data) { gcry_sexp_release( data ); }

} // end main


********************
Results:
********************

Enter data to encode:12345678909876543210
data entered: 12345678909876543210


return value for data sexp generation is [0]
return code for sexp_new(PARMS) is [0]

return code for genkey is [0]

return code for testkey is [0]

Return value for encryption with pKey is [0]


Fatal error: out of core in secure memory
Aborted (core dumped)

********************
/Results
********************


Thanks for any assistance or information you can direct this way.


--
Tony Warren
Prairie Systems, Inc.
Omaha, NE USA

<}-:
Re: Fatal error: out of core in secure memory - during decrypt function call [ In reply to ]
"Warren, Tony" <tonyw@prairiesys.com> writes:

Hi,

One comment on your code:

> char encryptMe[] = "";
[...]
> scanf( "%s", encryptMe );

That is not safe.

And now, to the actuall problem: it seems as if your code would
trigger a bug with in libgcrypt. As far as I can tell right now [i
have to investigate this more deeply], some MPI related arithmetic is
wrong. The consequence is that gcry_xmalloc_secure is asked to
allocate zero bytes; of course the return value is NULL. This is then
interpretated as an out-of-memory situation.

It seems that somehow `m1' and `m2' in rsa.c:secret() end up being the
same. The difference of these numbers is then used as one factor
passed to mpi_mulm. I am still trying to find out where the problem
is.

moritz
--
moritz@duesseldorf.ccc.de - http://duesseldorf.ccc.de/~moritz/
GPG fingerprint = 3A14 3923 15BE FD57 FC06 B501 0841 2D7B 6F98 4199
Re: Fatal error: out of core in secure memory - during decrypt function call [ In reply to ]
Hi Warren,

sorry for the late reply.

> Does the problem with the library code extend into the creation of
> the mpi / sexp of the source data, or is it only the creation of the
> key that is corrupted?

I am pretty sure that the problem is the input data. In short: it is
too short. In rsa.c::secret() we use an optimized RSA algorithm that
obviously fails for `small' numbers. I cannot tell right now, where
exactly the algorithm fails, I am still trying to find out.

> If the key is the problem

As I said, I think your problem is caused by your input data, not by
your key.

> I also see the testKey() result of the key pair is sane, but the
> testKey of only pubKey returns [65], invalid object... Is that part
> of the same mpi math error?

Oh, seems that the documentation is a bit bogus: as the comment in
pubkey.c::gcry_pk_testkey says, at the moment `only secret key
checking' is supported. I will include this comment in the manual.

> I've included the more current / modular version of the program. It
> now ends with a different error, but I'm not sure if it's due to the
> same problem, or something else...

Oh. I will have a look at it; what I have written above applies to
the first program and the triggered bug.

moritz
--
moritz@duesseldorf.ccc.de - http://duesseldorf.ccc.de/~moritz/
GPG fingerprint = 3A14 3923 15BE FD57 FC06 B501 0841 2D7B 6F98 4199
Re: Fatal error: out of core in secure memory - during decrypt function call [ In reply to ]
Moritz Schulte <moritz@duesseldorf.ccc.de> writes:

> I cannot tell right now, where exactly the algorithm fails, I am
> still trying to find out.

Ok. Seems that the math behind it fine; the problem is simply that
mpi_mul did not expect that U or V would be zero. The case that `m2 -
m1' is zero seemed bogus to me, although the mathematic algorithm
deals with that and will calculate the correct result.

I will commit a fix into CVS.

moritz
--
moritz@duesseldorf.ccc.de - http://duesseldorf.ccc.de/~moritz/
GPG fingerprint = 3A14 3923 15BE FD57 FC06 B501 0841 2D7B 6F98 4199