Mailing List Archive: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation.

TAKE NOTE 2: Future Let's Encrypt CA choice randomisation.

Nov 15, 2023, 3:14 PM

Post #1 of 7 (154 views)

On Wed, Nov 15, 2023 at 12:17:50AM -0500, Viktor Dukhovni wrote:

> It must be that Let's Encrypt finally stopped by default including that
> cross certificate in their chains.

As pointed out helpfully by Geert Hendrickx on the postfix-users list:

> They plan to stop providing the cross-signed "long chain" by default
> in February 2024, and completely in June, as the cross-sign expires
> in September. Dropping it last week was unintended.

The ensuing conversation on the LE forum uncovered a second potential
future incompatibility to plan for:

https://community.letsencrypt.org/t/short-chain-and-dane/208422/8?u=ietf-dane

Let's Encrypt are apparently also planning to *randomise* the choice of
intermediate issuer CA used with each renewal. Instead of consistently
using say "R3", they'll randomly choose one of R3/R4/E1/E2.

Therefore, anyone who publishes TLSA records for just one of the 4
issuers, will eventually also be "disappointed".

If you're using Let's Encrypt as your CA and prefer to publish
DANE-TA(2), rather than DANE-EE(3) TLSA records, please look over:

<http://dnssec-stats.ant.isi.edu/~viktor/x3hosts.html>

carefully, and publish all four of the **required** TLSA records, for
each MX host:

_25._tcp.mx1.org.example. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d ; R3
_25._tcp.mx1.org.example. IN TLSA 2 1 1 e5545e211347241891c554a03934cde9b749664a59d26d615fe58f77990f2d03 ; R4
_25._tcp.mx1.org.example. IN TLSA 2 1 1 276fe8a8c4ec7611565bf9fce6dcace9be320c1b5bea27596b2204071ed04f10 ; E1
_25._tcp.mx1.org.example. IN TLSA 2 1 1 bd936e72b212ef6f773102c6b77d38f94297322efc25396bc3279422e0c89270 ; E2
...

or if you prefer:

_25._tcp.mx1.org.example. IN CNAME _25._tlsa.org.example.
_25._tcp.mx2.org.example. IN CNAME _25._tlsa.org.example.
...
_25._tcp.mxN.org.example. IN CNAME _25._tlsa.org.example.
;
_25._tlsa.org.example. IN TLSA 2 1 1 8d02536c887482bc34ff54e41d2ba659bf85b341a0a20afadb5813dcfbcf286d ; R3
_25._tlsa.org.example. IN TLSA 2 1 1 e5545e211347241891c554a03934cde9b749664a59d26d615fe58f77990f2d03 ; R4
_25._tlsa.org.example. IN TLSA 2 1 1 276fe8a8c4ec7611565bf9fce6dcace9be320c1b5bea27596b2204071ed04f10 ; E1
_25._tlsa.org.example. IN TLSA 2 1 1 bd936e72b212ef6f773102c6b77d38f94297322efc25396bc3279422e0c89270 ; E2

--
Viktor.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 16, 2023, 11:41 AM

Post #2 of 7 (152 views)

Permalink

D?a 15. novembra 2023 23:14:39 UTC používate? Viktor Dukhovni via Exim-users <exim-users@lists.exim.org> napísal:

>If you're using Let's Encrypt as your CA and prefer to publish
>DANE-TA(2), rather than DANE-EE(3) TLSA records, please look over:

Just curious. Enough recent certbot provides --reuse-key and --new-key
(or so) options, thus one can stay with DANE-EE records. Please, beside
using ACME client which doesn't support that, is it still useful to use
DANE-TA with LE?

regards

--
Slavko
https://www.slavino.sk/

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 16, 2023, 12:12 PM

Post #3 of 7 (152 views)

Permalink

On Thu, Nov 16, 2023 at 07:41:46PM +0000, Slavko via Exim-users wrote:

> >If you're using Let's Encrypt as your CA and prefer to publish
> >DANE-TA(2), rather than DANE-EE(3) TLSA records, please look over:
>
> Just curious. Enough recent certbot provides --reuse-key and --new-key
> (or so) options, thus one can stay with DANE-EE records. Please, beside
> using ACME client which doesn't support that, is it still useful to use
> DANE-TA with LE?

I don't recommend DANE-TA(2), and encourage use of DANE-EE(3) instead.

This avoids unnecessary trust in the CA's usually weak "DV" certificate
issuance, and gives you more control over the timing of changes.

You do however need to be more sophisticated about any key rollovers
that you do perform from time to time.

I have a partial (usabel work-in-progress) solution to that workflow for
"certbot" in the form of:

https://github.com/tlsaware/danebot

and there are other tools that take a similar approach. I would love to
see code contributions that flesh out "danebot" to make it more "feature
compleet", adding support for robust (retried to ensure "at least" one
successful activation) hooks and for changes in the list of requested
domains without forcing a key rollover.

Any motivated and suitably skilled volunteers?

--
Viktor.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 19, 2023, 4:30 AM

Post #4 of 7 (151 views)

Permalink

Ahoj,

D?a Thu, 16 Nov 2023 15:12:15 -0500 Viktor Dukhovni via Exim-users
<exim-users@lists.exim.org> napísal:

> I don't recommend DANE-TA(2), and encourage use of DANE-EE(3) instead.

I am far from DANE expert, but my understanding is, that DANE-TA is
good for own CAs, where one have full control on (intermediate) CA's
certs and its renews.

If one use that for foreign CA, soon or latter can meet unexpected CA
certificate replace, and monitoring can only avoid to problem persist
for long time, but not avoid to happen. Right?

> You do however need to be more sophisticated about any key rollovers
> that you do perform from time to time.

IMO not as sophisticated is needed. I still don't use DANE yet, but i am
in stage of preparation for it.

For now i have SMTP's cert with persistent key already. I have deploy
(shell) script on MX, which detects certificate change (systemd's path
unit), and on change it compares old and new cert's keys and if they
match, it copies new certificate to right place (and exim auto-reloads
it). This part works for some time (months) already.

If keys doesn't match, it has to reject cert update/replace and
notifies me (as i need manually modify DNS), but this part is not
tested yet. The notification contains new required TLSA-EE value(s),
thus can be simply switched to automate TLSA change, when my provider
will start to support that.

> I have a partial (usabel work-in-progress) solution to that workflow
> for "certbot" in the form of:
>
> https://github.com/tlsaware/danebot
>
> Any motivated and suitably skilled volunteers?

I take quick look on it. I am not very open to "wrapper" solution. Does
it something, what is not possible from certbot's deploy hook?

regards

--
Slavko
https://www.slavino.sk

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 19, 2023, 11:33 AM

Post #5 of 7 (150 views)

Permalink

On Sun, Nov 19, 2023 at 01:30:29PM +0100, Slavko via Exim-users wrote:

> > I don't recommend DANE-TA(2), and encourage use of DANE-EE(3) instead.
>
> I am far from DANE expert, but my understanding is, that DANE-TA is
> good for own CAs, where one have full control on (intermediate) CA's
> certs and its renews.

Ah, yes, most users of DANE-TA(2) use it with a public CA, but indeed
the story is very different with a self-managed CA, where it does make
more sense, because you're then not subject to the whims and lax (DV)
issuance practices of the public CA.

> If one use that for foreign CA, soon or latter can meet unexpected CA
> certificate replace, and monitoring can only avoid to problem persist
> for long time, but not avoid to happen. Right?

With careful monitoring and "gating" (as you also point out), it can
be managed, but no longer the "simple" set up and neglect that the users
opting for the public CA imagined.

https://community.letsencrypt.org/t/short-chain-and-dane/208422/22?u=ietf-dane

> > You do however need to be more sophisticated about any key rollovers
> > that you do perform from time to time.
>
> IMO not as sophisticated is needed. I still don't use DANE yet, but i am
> in stage of preparation for it.

Preparing, is exactly what's missing from some naïve users' practice.

> For now i have SMTP's cert with persistent key already. I have deploy
> (shell) script on MX, which detects certificate change (systemd's path
> unit), and on change it compares old and new cert's keys and if they
> match, it copies new certificate to right place (and exim auto-reloads
> it). This part works for some time (months) already.

It is possible for the path unit to fail to run, but the ACME client
believes it is done. Does systemd's path unit guarantee "at least once"
execution.

> If keys doesn't match, it has to reject cert update/replace and
> notifies me (as i need manually modify DNS), but this part is not
> tested yet.

I called that "gating" in the linked thread. You're (at least compared
to most :-) a sophisticated user.

> > I have a partial (usabel work-in-progress) solution to that workflow
> > for "certbot" in the form of:
> >
> > https://github.com/tlsaware/danebot
> >
> > Any motivated and suitably skilled volunteers?
>
> I take quick look on it. I am not very open to "wrapper" solution. Does
> it something, what is not possible from certbot's deploy hook?

Yes, but the wrapper addresses two issues that are difficult without it.

* Staging a future key, that the ACME client will conditionally
switch to, once the TLSA record is live.

* Avoiding reliance on "certbot" hooks, which (last I checked) don't
guarantee "at least once" execution.

--
Viktor.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 19, 2023, 1:33 PM

Post #6 of 7 (149 views)

Permalink

D?a 19. novembra 2023 19:33:12 UTC používate? Viktor Dukhovni via Exim-users <exim-users@lists.exim.org> napísal:

>It is possible for the path unit to fail to run, but the ACME client
>believes it is done. Does systemd's path unit guarantee "at least once"
>execution.

ACME client doesn't need (nor is) to know about that. The cert is
renewed on another host (container), it is placed to one dir on
target host after renew and that is all for certbot. Then systemd
path's unit is activated locally and cert will be copied to final place.

Sure, it's execution can fail. Systemd has ability to restart its services
in case of non-success return code, but now i am not sure if that will
happen with units triggered by path (or similar) unit, i add that to my
ToDo (to try/verify it). Anyway, if unit fails:

a) monitoring will alert me about failed systemd unit
b) the old certificate will stay in place
c) soon or latter monitoring will alert me about expiring cert
d) if my script fails, its unit fails and a) will happen

Alerts are repeated (hourly for systemd units and daily for expiration)
until solved. As LE certs are renewed 30 days before expiration,
enough time to solve the problem. Various problems happened
already (over years), including cert renew, that monitoring/alerting
works for me.

>I called that "gating" in the linked thread. You're (at least compared
>to most :-) a sophisticated user.

Or more simple, i am aware of "Murphy law": "If something can
fail, it will fail, and if something cannot fail, it will fail too" (raw
translation) :-)

> * Staging a future key, that the ACME client will conditionally
> switch to, once the TLSA record is live.

Do you mean opposite of usual certbot logic: first generate key, then
setup TLSA for it, and after that request certificate for/with that key?

> * Avoiding reliance on "certbot" hooks, which (last I checked) don't
> guarantee "at least once" execution.

Do you mean ability to rerun hook if it fails? Or do you mean, that
certbot can skip/fail to run hook after renew at all?

regards

--
Slavko
https://www.slavino.sk/

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/

Re: TAKE NOTE 2: Future Let's Encrypt CA choice randomisation. [ In reply to ]

exim-users at lists

Nov 19, 2023, 2:11 PM

Post #7 of 7 (149 views)

Permalink

On Sun, Nov 19, 2023 at 09:33:37PM +0000, Slavko via Exim-users wrote:

> > * Staging a future key, that the ACME client will conditionally
> > switch to, once the TLSA record is live.
>
> Do you mean opposite of usual certbot logic: first generate key, then
> setup TLSA for it, and after that request certificate for/with that key?

Generate a staged key, that sits there, unused, waiting for TLSA records
to appear, and once that has been in place long enough, obtain a cert
for the new key, otherwise, in the mean time keep using the old key.

> > * Avoiding reliance on "certbot" hooks, which (last I checked) don't
> > guarantee "at least once" execution.
>
> Do you mean ability to rerun hook if it fails? Or do you mean, that
> certbot can skip/fail to run hook after renew at all?

I mean "at least once" successful execution, so yes, rerun on failure,
or even if the hook never got to run at all, because the system rebooted
before it got a chance to run, but the new cert was alread obtained.

Of course with alerts any time the hook fails. When I looked, I found
no code in certbot that ensured at least once execution of hooks, so
(per Murphy's Law) they're unreliable, and a better approach is needed.

--
Viktor.

--
## subscription configuration (requires account):
## https://lists.exim.org/mailman3/postorius/lists/exim-users.lists.exim.org/
## unsubscribe (doesn't require an account):
## exim-users-unsubscribe@lists.exim.org
## Exim details at http://www.exim.org/
## Please use the Wiki with this list - http://wiki.exim.org/