Mailing List Archive

On 24 Oct 2023, at 04:38, Felix E. Klee <felix.klee@inka.de> wrote:
>
> For the purpose of re-encryption with a new key, I’d like to find all
> files that are encrypted with my key BEF6EFD38FE8DCA0. All encrypted
> files, independent of key, have the extension `.gpg`.
>
> How do I do that for a massive directory tree?

Hi, Felix.

GNU `file` will print the encryption key ID:

```
andrewg@fum:~$ file hidden_service/private_key.gpg
hidden_service/private_key.gpg: PGP RSA encrypted session key - keyid: 6B090693 14549D4B RSA (Encrypt or Sign) 4096b .
```

That keyid is the encryption subkey, so you can grep file’s batch output for its short ID, e.g.:

```
file *.gpg | grep $SHORT_ENC_SUBKEY_ID
```

Note that due to file’s use of whitespace, you can’t grep for the long ID unless you mangle it accordingly.

If you don’t have GNU file, you can try `gpg —list-packets` instead, but this will be slower as gpg will parse the entire file. Also, it only parses one file at a time, and the encryption key ID is output on STDERR. You can invoke it in a bash loop like this:

```
find . -name '*.gpg' -print0 | while read -r -d '' file; do
echo -n "$file: "
gpg --list-packets "$file" 2>&1 >/dev/null
done | grep $SHORT_ENC_SUBKEY_ID
```

A

Apologies to the `file` authors, it’s a BSD utility, not GNU.

A

On 24 Oct 2023, at 10:11, Andrew Gallagher via Gnupg-users <gnupg-users@gnupg.org> wrote:
>
> Signed PGP part
> On 24 Oct 2023, at 04:38, Felix E. Klee <felix.klee@inka.de> wrote:
>>
>> For the purpose of re-encryption with a new key, I’d like to find all
>> files that are encrypted with my key BEF6EFD38FE8DCA0. All encrypted
>> files, independent of key, have the extension `.gpg`.
>>
>> How do I do that for a massive directory tree?
>
> Hi, Felix.
>
> GNU `file` will print the encryption key ID:
>
> ```
> andrewg@fum:~$ file hidden_service/private_key.gpg
> hidden_service/private_key.gpg: PGP RSA encrypted session key - keyid: 6B090693 14549D4B RSA (Encrypt or Sign) 4096b .
> ```
>
> That keyid is the encryption subkey, so you can grep file’s batch output for its short ID, e.g.:
>
> ```
> file *.gpg | grep $SHORT_ENC_SUBKEY_ID
> ```
>
> Note that due to file’s use of whitespace, you can’t grep for the long ID unless you mangle it accordingly.
>
> If you don’t have GNU file, you can try `gpg —list-packets` instead, but this will be slower as gpg will parse the entire file. Also, it only parses one file at a time, and the encryption key ID is output on STDERR. You can invoke it in a bash loop like this:
>
> ```
> find . -name '*.gpg' -print0 | while read -r -d '' file; do
> echo -n "$file: "
> gpg --list-packets "$file" 2>&1 >/dev/null
> done | grep $SHORT_ENC_SUBKEY_ID
> ```
>
> A
>
>
>

On Tue, 24 Oct 2023 11:38, Felix E. Klee said:
> For the purpose of re-encryption with a new key, I’d like to find all
> files that are encrypted with my key BEF6EFD38FE8DCA0. All encrypted
> files, independent of key, have the extension `.gpg`.
>
> How do I do that for a massive directory tree?

AMybe something like this

encrypted-to-me-p.sh
--8<---------------cut here---------------start------------->8---
#/bin/sh
gpg -d --status-fd 1 -o /dev/null 2>/dev/null "$1" | awk '
$1=="[GNUPG:]" && $2=="ENC_TO" && $3=="BEF6EFD38FE8DCA0" {print $1; exit 0}'
--8<---------------cut here---------------end--------------->8---


find /foo -type f -name '*.gpg' -print0 | xargs -0 -n1 encrypted-to-me-p.sh

Best done with a keyring which does not hold any keys. Does not catch
files which have hidden recipients. Note that you need to test for the
subkey because that is the only information available in the encrypted
files.

Using --list-packets or pgpdump might be better but those have no stable
API.


Salam-Shalom,

Werner

--
The pioneers of a warless world are the youth that
refuse military service. - A. Einstein

On Tue, Oct 24, 2023 at 11:38:52AM +0800, "Felix E. Klee" <felix.klee@inka.de> wrote:

> For the purpose of re-encryption with a new key, I’d like to find all
> files that are encrypted with my key BEF6EFD38FE8DCA0. All encrypted
> files, independent of key, have the extension `.gpg`.
>
> How do I do that for a massive directory tree?

With my rawhide (rh) program (github.com/raforg/rawhide) you can do
it with something like this:

rh /path '"*.gpg" && "*PGP*encrypted*BEF6EFD3 8FE8DCA0*".what'

That looks under /path for files whose names end in .gpg and
whose file(1) output would contain the given glob pattern,
but no file(1) processes are created. The output of file(1)
for an encrypted file looks something like:

file.gpg: PGP RSA encrypted session key - keyid: 49C40F3A BA227C81 RSA (Encrypt or Sign) 4096b .

It can also be done with find(1) of course, but it's a
little slower because it needs additional processes for
each encrypted file:

find /path -name '*.gpg' \
-execdir /bin/sh -c 'file {} | grep -q "PGP.*encrypted.*BEF6EFD3 8FE8DCA0"' \; \
-print

But the extra time is probably immaterial when followed
by re-encryption.

While testing these, I just noticed that /usr/bin/file
on my macOS-10.14 laptop shows a different keyid to
what libmagic shows. That's bizarre.

For some encrypted files of mine, /usr/bin/file (v5.33)
shows 3A0FC449 817C22BA but libmagic/rh shows 49C40F3A
BA227C81 for the same files. A more recent version of
file (v5.45) installed via macports shows the same as
libmagic/rh. So choose your version of file(1) wisely. :-)

Also, in case you need to re-encrypt regularly, I
recommend assigning some label to the key and putting
it in the filename (e.g. blah.gpg.key23). Then you
don't need to look inside the file, and if it takes a
long time to re-encrypt lots of files, you can easily
see how it's progressing.

cheers,
raf


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Tue, Oct 24, 2023 at 5:21?PM Werner Koch <wk@gnupg.org> wrote:
> encrypted-to-me-p.sh
> --8<---------------cut here---------------start------------->8---
> #/bin/sh
> gpg -d --status-fd 1 -o /dev/null 2>/dev/null "$1" | awk '
> $1=="[GNUPG:]" && $2=="ENC_TO" && $3=="BEF6EFD38FE8DCA0" {print $1; exit 0}'
> --8<---------------cut here---------------end--------------->8---

Thank you! I modified that a bit, to make it more readable to me and fix
a little bug: The second `$1` doesn’t expand to the file name. Also, I
had to pass `--pinentry-mode cancel`. Otherwise it would ask me for the
PIN of my smartcard. See below for my version.

What I don’t like is the `2>/dev/null` because that may mask actual
error messages. I specified `--quiet`. That works to some extend, but I
still get:

gpg: decryption failed: No secret key

I wonder how to get rid of that.

My version:

#/bin/sh

filename=$1
enc_sub_key=04FDF78D1679DD94

gpg --decrypt \
--pinentry-mode cancel \
--status-fd 1 \
--quiet \
--output /dev/null "$1" |
awk -v filename="$filename" \
-v enc_sub_key="$enc_sub_key" \
'
$1=="[GNUPG:]" &&
$2=="ENC_TO" &&
$3==enc_sub_key {
print filename
exit 0
}'

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Wed, Oct 25, 2023 at 10:08?AM raf via Gnupg-users
<gnupg-users@gnupg.org> wrote:
> > How do I do that for a massive directory tree?
>
> With my rawhide (rh) program (github.com/raforg/rawhide) you can do it
> with something like this:
>
> rh /path '"*.gpg" && "*PGP*encrypted*BEF6EFD3 8FE8DCA0*".what'

Very interesting, may look into that. But first working with Werner’s
solution.

> Also, in case you need to re-encrypt regularly, I recommend assigning
> some label to the key and putting it in the filename (e.g.
> blah.gpg.key23).

I may do that.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Tue, Oct 24, 2023 at 5:12?PM Andrew Gallagher <andrewg@andrewg.com>
wrote:
> GNU `file` will print the encryption key ID:

Interesting. I wonder if there is any disadvantage of using `file` over
Werner’s proposal.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Wed, 25 Oct 2023 10:54, Felix E. Klee said:

> Thank you! I modified that a bit, to make it more readable to me and fix
> a little bug: The second `$1` doesn’t expand to the file name. Also, I

Upoi caught me, I didn't test the posted version.

> gpg: decryption failed: No secret key
>
> I wonder how to get rid of that.

grep -v on stderr ;-).

I think it is time to make things like this easier. Actually re-encrypt
support has been on our feature list for many years. Every time I want
to tackle this I stop at the descsion on whether to to also handle the
large file on server shall be re-encrypted ot on teh simple re-encrypt a copy.

>
> My version:
>
> #/bin/sh
>
> filename=$1
> enc_sub_key=04FDF78D1679DD94
>
> gpg --decrypt \
> --pinentry-mode cancel \
> --status-fd 1 \
> --quiet \
> --output /dev/null "$1" |
> awk -v filename="$filename" \
> -v enc_sub_key="$enc_sub_key" \
> '
> $1=="[GNUPG:]" &&
> $2=="ENC_TO" &&
> $3==enc_sub_key {
> print filename
> exit 0
> }'
>
> _______________________________________________
> Gnupg-users mailing list
> Gnupg-users@gnupg.org
> https://lists.gnupg.org/mailman/listinfo/gnupg-users

--
The pioneers of a warless world are the youth that
refuse military service. - A. Einstein

On Wed, Oct 25, 2023 at 11:01:30AM +0800, "Felix E. Klee" <felix.kleei@inka.de> wrote:

> On Tue, Oct 24, 2023 at 5:12?PM Andrew Gallagher <andrewg at andrewg.com>
> wrote:
> > GNU `file` will print the encryption key ID:
>
> Interesting. I wonder if there is any disadvantage of using `file` over
> Werner's proposal.

I don't think so. The version you are quoting didn't
use find(1) as well, so it doesn't search recursively,
but apart from that, using file would be simpler (no
decryption, and no need to think about pinentry, and no
need to access keys). If the files you are searching
for are gigabytes in size, and there are many of them,
decrypting them to find the key would be too slow,
unless I've misunderstood something. file would read
much less of each file. No, gpg would die with SIGPIPE
as soon as awk terminates, so they probably read a
similar amount. And they both use multiple additional
processes for each candidate file (either sh+file+grep
or sh+gpg+awk).

cheers,
raf


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

Hi,

hit the sent key combination by accident. The last para should have
read:

I think it is time to make things like this easier. Actually re-encrypt
support has been on our feature list for many years.

Every time I want to tackle this I stop at the decision on whether to
also handle the large-file-on-server case or just do the simple
re-encrypt-a-copy.

Feature I'd like to have are:

- Select whether anything should be done at all for a message.
- Add new public keys (to encrypt the session key)
- Add new symmetric keys
- Remove existing public keys
- Remove unknown keys (hidden keys)
- Remove symmetric keys



Shalom-Salam,

Werner


--
The pioneers of a warless world are the youth that
refuse military service. - A. Einstein

raf via Gnupg-users wrote:
> [...]
> While testing these, I just noticed that /usr/bin/file
> on my macOS-10.14 laptop shows a different keyid to
> what libmagic shows. That's bizarre.
>
> For some encrypted files of mine, /usr/bin/file (v5.33)
> shows 3A0FC449 817C22BA but libmagic/rh shows 49C40F3A
> BA227C81 for the same files. A more recent version of
> file (v5.45) installed via macports shows the same as
> libmagic/rh. So choose your version of file(1) wisely. :-)
>

You have an endianness-mismatch issue somewhere. The octets are
reversed in each 32-bit group between the samples.


-- Jacob


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Wed, Oct 25, 2023 at 9:23?PM Werner Koch <wk@gnupg.org> wrote:
> > gpg: decryption failed: No secret key
> >
> > I wonder how to get rid of that.
>
> grep -v on stderr ;-).

Thanks, I was thinking about that. But I think simply using find, as
suggested by Andrew and raf, is sufficient and simple.

> I think it is time to make things like this easier. Actually
> re-encrypt support has been on our feature list for many years.

That would be fancy. Personally, I’m happy with a bit of shell
scripting. My use case is rather simple, and I don’t need to do
re-encryption very often.

_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Thu, Oct 26, 2023 at 11:29:38AM +0800, "Felix E. Klee" <felix.klee@inka.de> wrote:

> On Wed, Oct 25, 2023 at 9:23?PM Werner Koch <wk@gnupg.org> wrote:
> > > gpg: decryption failed: No secret key
> > >
> > > I wonder how to get rid of that.
> >
> > grep -v on stderr ;-).
>
> Thanks, I was thinking about that. But I think simply using find, as
> suggested by Andrew and raf, is sufficient and simple.

rh intead of find+n*(sh+file+grep) is even simpler (but I'm biased). :-)

> > I think it is time to make things like this easier. Actually
> > re-encrypt support has been on our feature list for many years.
>
> That would be fancy. Personally, I’m happy with a bit of shell
> scripting. My use case is rather simple, and I don’t need to do
> re-encryption very often.

Yeah. I think gpg -d ... | gpg -e -r ... is simple enough. I use that
(in a "recrypt" python script) for annual key rollover re-encryptions
of many database backups.

cheers,
raf


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users

On Wed, Oct 25, 2023 at 09:38:55PM -0500, Jacob Bachmeyer via Gnupg-users <gnupg-users@gnupg.org> wrote:

> raf via Gnupg-users wrote:
> > [...]
> > While testing these, I just noticed that /usr/bin/file
> > on my macOS-10.14 laptop shows a different keyid to
> > what libmagic shows. That's bizarre.
> >
> > For some encrypted files of mine, /usr/bin/file (v5.33)
> > shows 3A0FC449 817C22BA but libmagic/rh shows 49C40F3A
> > BA227C81 for the same files. A more recent version of
> > file (v5.45) installed via macports shows the same as
> > libmagic/rh. So choose your version of file(1) wisely. :-)
>
> You have an endianness-mismatch issue somewhere. The octets are reversed in
> each 32-bit group between the samples.
>
> -- Jacob

Well spotted! Thanks. The actual endianness wouldn't have changed,
but file's presentation of it much have been fixed (to match
gpg output) between those versions.

cheers,
raf


_______________________________________________
Gnupg-users mailing list
Gnupg-users@gnupg.org
https://lists.gnupg.org/mailman/listinfo/gnupg-users