Mailing List Archive: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode

Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode

Jul 25, 2019, 6:46 AM

Post #1 of 6 (522 views)

Hi everyone,

First, please allow me to define a bit of ad-hoc
nomenclature. I will use the uppercase tems "ENCRYPT",
"ENCRYPTION", etc. as shorthands for "compress and
AES256-encrypt", "compression and AES256 encryption", etc.
Likewise, I will use "DECRYPT", etc. as shorthands for
"[AES256] decrypt and decompress", etc.

I need to ENCRYPT ~20 million files (~150TB) for long-term
(>15y) storage. This ENCRYPTION will be done in several
batches, and will take place over many months (due to CPU and
bandwidth limitations).

The ideal solution would produce ENCRYPTED files that can be
decrypted using standard off-the-shelf gpg/gpg2. [1]

In my search for a library I could use to do this, I found
gpgme and libgcrypt. I tried the former, and found it not
suitable, due to frequent gpg-agent-related failures.

libgcrypt, on the other hand, is a bit too low-level for
someone who is not acquainted with the fine details of gpg's
ENCRYPTION to replicate it. (AFAICT, using straight-up
gcry_cipher_encrypt would not necessarily produce an
encrypted file (let alone an ENCRYPTED file) that could be
decrypted/DECRYPTED with standard gpg/gpg2.)

Is there something in-between gpgme and libgcrypt that would
allow me implement the required tool?

*Alternatively*, can someone tell me of a more efficient way
than reading the gpg2 source code for me to learn how to
implement gpg-compatible ENCRYPTION/DECRYPTION using
libgcrypt?

Thank you all in advance!

kj

[1] This gpg/gpg2 compatibility requirement is important, as
an insurance that the files will be DECRYPTABLE in the
"distant" future (10-15y), even the my tool is not properly
enough maintained to be operational then. This, of course,
assumes that gpg will have greater longevity than a privately
implemented, single-user tool like mine.

Re: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode [ In reply to ]

rjh at sixdemonbag

Jul 25, 2019, 12:22 PM

Post #2 of 6 (522 views)

Permalink

> First, please allow me to define a bit of ad-hoc
> nomenclature. I will use the uppercase tems "ENCRYPT",
> "ENCRYPTION", etc. as shorthands for "compress and
> AES256-encrypt", "compression and AES256 encryption", etc.
> Likewise, I will use "DECRYPT", etc. as shorthands for
> "[AES256] decrypt and decompress", etc.

For a straightforward purely-symmetric use case, consider using OpenSSL.
Just use it wisely: specify your own key and initialization vector,
rather than trust OpenSSL's weak password derivation function.

$ tar -cJf - mydir | openssl aes-128-cbc -out foo.txz -K
DEADBEEFDEADBEEFDEADBEEFDEADBEEF -iv DEADBEEFDEADBEEFDEADBEEFDEADBEEF
-out mydir.txz.aes

Don't get me wrong: GnuPG can definitely do the job you want. But
consider whether you really need RFC4880's baroque packet format, weird
CFB mode, and everything else. Sometimes there's a lot to be said for
simplicity.

> [1] This gpg/gpg2 compatibility requirement is important, as
> an insurance that the files will be DECRYPTABLE in the
> "distant" future (10-15y), even the my tool is not properly
> enough maintained to be operational then. This, of course,
> assumes that gpg will have greater longevity than a privately
> implemented, single-user tool like mine.

OpenSSL meets your requirement.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode [ In reply to ]

gnupg-users at gnupg

Jul 25, 2019, 1:59 PM

Post #3 of 6 (522 views)

Permalink

On Thu, Jul 25, 2019 at 3:26 PM Robert J. Hansen <rjh@sixdemonbag.org>
wrote:

> > First, please allow me to define a bit of ad-hoc
> > nomenclature. I will use the uppercase tems "ENCRYPT",
> > "ENCRYPTION", etc. as shorthands for "compress and
> > AES256-encrypt", "compression and AES256 encryption", etc.
> > Likewise, I will use "DECRYPT", etc. as shorthands for
> > "[AES256] decrypt and decompress", etc.
>
> For a straightforward purely-symmetric use case, consider using OpenSSL.
> Just use it wisely: specify your own key and initialization vector,
> rather than trust OpenSSL's weak password derivation function.
>
> $ tar -cJf - mydir | openssl aes-128-cbc -out foo.txz -K
> DEADBEEFDEADBEEFDEADBEEFDEADBEEF -iv DEADBEEFDEADBEEFDEADBEEFDEADBEEF
> -out mydir.txz.aes
>

Pardon my ignorance, but I gather from this example that I would have
to manage not only passphrases but also iv's as well? (That would add
to my work's complexity.)

Don't get me wrong: GnuPG can definitely do the job you want. But
> consider whether you really need RFC4880's baroque packet format, weird
> CFB mode, and everything else. Sometimes there's a lot to be said for
> simplicity.
>

Thank you. This is a valuable perspective. I'm all for simplicity!

In fact, if I could have it my way, I would use a library that does
nothing more than AES256-encrypt/decrypt (as long as I had any
confidence that it would still be maintained 5 years from now).

In other words, I would love to use a single-purpose tool that is to
AES256-encryption/decryption what, for example, gzip is to
compression/decompression.

Unfortunately, I have not been able to hit upon such a tool, which I
find a bit mystifying. Instead, all the tools I have found fall into
two distinct categories: (1) ephemeral projects; (2) large systems
(like GnuPG or OpenSSL) in which the AES256-encryption/decryption
capability is such tiny subcomponent that it is very difficult for me
to find out what I need to do for my purposes.

(Who knows, maybe in the cryptography domain, complex
multifunctionality is what give systems their longevity.)

> [1] This gpg/gpg2 compatibility requirement is important, as
> > an insurance that the files will be DECRYPTABLE in the
> > "distant" future (10-15y), even the my tool is not properly
> > enough maintained to be operational then. This, of course,
> > assumes that gpg will have greater longevity than a privately
> > implemented, single-user tool like mine.
>
> OpenSSL meets your requirement.
>

I assume that by this you mean that OpenSSL will still be around 10-15
years from now, and *not* that one can use gpg to decrypt an
openssl-encrypted file. (Please correct me if I'm wrong!)

Thank you again!

kj

Re: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode [ In reply to ]

rjh at sixdemonbag

Jul 25, 2019, 2:26 PM

Post #4 of 6 (522 views)

Permalink

> Pardon my ignorance, but I gather from this example that I would have
> to manage not only passphrases but also iv's as well? (That would add
> to my work's complexity.)

An AES256 key is only 32 bytes long; an IV, only 16. Keeping track of
48 bytes to decrypt your files isn't exactly a lot. You can fit this on
the back of a business card with room left over. I've done it before.

Passphrase-based crypto works by converting a passphrase into a
(seemingly) random series of bytes. The problem is OpenSSL's
passphrase-to-bytes routine is pretty badly substandard. Specify your
own key and IV.

> In fact, if I could have it my way, I would use a library that does
> nothing more than AES256-encrypt/decrypt (as long as I had any
> confidence that it would still be maintained 5 years from now).

Which language are you looking to use? C#, Java, and Python all include
AES256 in the standard library and have excellent long-term support.
Many other languages offer it as well. Python has some excellent PyPI
packages like passlib and Crypto which can make your task much simpler.

> In other words, I would love to use a single-purpose tool that is to
> AES256-encryption/decryption what, for example, gzip is to
> compression/decompression.

OpenSSL. Look at the command line I gave you: it's used as part of a
pipeline that creates a tar archive and encrypted output all in one go.

> I assume that by this you mean that OpenSSL will still be around 10-15
> years from now

Yes.

We're getting pretty far afield from GnuPG here: please feel free to
follow up off-list. Thank you! :)

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode [ In reply to ]

gnupg-users at gnupg

Jul 27, 2019, 7:50 AM

Post #5 of 6 (521 views)

Permalink

Hi.

Am 25.07.2019 um 15:46 schrieb Kynn Jones via Gnupg-users:
> Hi everyone,
>
> First, please allow me to define a bit of ad-hoc
> nomenclature. I will use the uppercase tems "ENCRYPT",
> "ENCRYPTION", etc. as shorthands for "compress and
> AES256-encrypt", "compression and AES256 encryption", etc.
> Likewise, I will use "DECRYPT", etc. as shorthands for
> "[AES256] decrypt and decompress", etc.
>
> I need to ENCRYPT ~20 million files (~150TB) for long-term
> (>15y) storage. This ENCRYPTION will be done in several
> batches, and will take place over many months (due to CPU and
> bandwidth limitations).
>
> The ideal solution would produce ENCRYPTED files that can be
> decrypted using standard off-the-shelf gpg/gpg2. [1]
>
> In my search for a library I could use to do this, I found
> gpgme and libgcrypt. I tried the former, and found it not
> suitable, due to frequent gpg-agent-related failures.
>
> libgcrypt, on the other hand, is a bit too low-level for
> someone who is not acquainted with the fine details of gpg's
> ENCRYPTION to replicate it. (AFAICT, using straight-up
> gcry_cipher_encrypt would not necessarily produce an
> encrypted file (let alone an ENCRYPTED file) that could be
> decrypted/DECRYPTED with standard gpg/gpg2.)
>
> Is there something in-between gpgme and libgcrypt that would
> allow me implement the required tool?
>
> *Alternatively*, can someone tell me of a more efficient way
> than reading the gpg2 source code for me to learn how to
> implement gpg-compatible ENCRYPTION/DECRYPTION using
> libgcrypt?
>
> Thank you all in advance!

Have you take a look into libsodium based tools?
https://download.libsodium.org/doc/libsodium_users

For example https://github.com/TLINDEN/pcp

> kj

Best regards
Aleks

> [1] This gpg/gpg2 compatibility requirement is important, as
> an insurance that the files will be DECRYPTABLE in the
> "distant" future (10-15y), even the my tool is not properly
> enough maintained to be operational then. This, of course,
> assumes that gpg will have greater longevity than a privately
> implemented, single-user tool like mine.
>
> _______________________________________________
> Gnupg-users mailing list
> Gnupg-users@gnupg.org
> http://lists.gnupg.org/mailman/listinfo/gnupg-users
>

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
http://lists.gnupg.org/mailman/listinfo/gnupg-users

Re: Need to implement a gpg/gpg2-compatible tool to encrypt millions of files in unsupervised mode [ In reply to ]

gnupg-users at gnupg

Jul 27, 2019, 10:06 AM

Post #6 of 6 (521 views)

Permalink

On Sat, Jul 27, 2019 at 10:50 AM Aleksandar Lazic <al-gnupg_users@none.at>
wrote:

> Have you take a look into libsodium based tools?
> https://download.libsodium.org/doc/libsodium_users
>

No I hadn't. Thank you, it looks amazing. (Still need to wrap my head
around it.)

kj