Mailing List Archive

Probably confusion with the tomcat cert, versus the tomcat-trust.

Remember this defect?
https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy13916

On Sat, May 30, 2020 at 7:11 PM Hunter Fuller <hf0002@uah.edu> wrote:

> I was wondering the same thing. You may be able to skip that one, but it
> advises you to restart it when updating the callmanager-trust store, so we
> did it.
>
> On Sat, May 30, 2020 at 19:10 Anthony Holloway <
> avholloway+cisco-voip@gmail.com> wrote:
>
>> MVP
>>
>> But why restart TFTP?
>>
>> On Sat, May 30, 2020 at 7:02 PM Hunter Fuller <hf0002@uah.edu> wrote:
>>
>>> All,
>>>
>>> If you use certs whose trust is derived from the Sectigo root that
>>> expired today, and your MRA isn’t working, I’ll try to save you a call to
>>> TAC.
>>>
>>> Do all of these things:
>>>
>>> - Load the new intermediates and root into callmanager-trust and
>>> tomcat-trust on all your UCMs
>>> - restart tomcat, tftp, and callmanager on those boxes
>>> - load the new intermediates and root into the CA trust store on all
>>> expressways
>>> - reboot the Expressway-Es
>>>
>>> If you need more detail or help, let me know, we just got off the phone
>>> with TAC. Hope it helps.
>>>
>>> --
>>>
>>> --
>>> Hunter Fuller (they)
>>> Router Jockey
>>> VBH Annex B-5
>>> +1 256 824 5331
>>>
>>> Office of Information Technology
>>> The University of Alabama in Huntsville
>>> Network Engineering
>>>
>> _______________________________________________
>>> cisco-voip mailing list
>>> cisco-voip@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>
>> --
>
> --
> Hunter Fuller (they)
> Router Jockey
> VBH Annex B-5
> +1 256 824 5331
>
> Office of Information Technology
> The University of Alabama in Huntsville
> Network Engineering
>

Oh, I didn't know about that issue, but it explains why you have to restart
TFTP. Anything that takes a restart for tomcat will also take a restart for
tomcat-trust.

Why? Because Tomcat (and most apps) ship the server cert *and
intermediates* with the TLS transaction. Cisco Unified stuff seems to
generate the intermediate chain from the tomcat-trust chain. So any time
you alter tomcat-trust, you have to restart everything that serves the
tomcat cert, so that it will start serving the correct intermediates. (This
is also why you have to reboot Expressway-E when changing the root and
intermediates. It doesn't tell you this when you change them, but it will
not start shipping the new root and intermediates with its transactions
until it is rebooted - and MRA phones will not work in that state!)

For example, here I am connecting to our TFTP port using the OpenSSL CLI
and grepping for the cert CNs, and you can see it is actually sending FOUR
certs - not just the server cert. The other three are from tomcat-trust.

hf0002@burrito ~$ openssl s_client -connect vbhucmpub.voip.uah.edu:6972
-showcerts -prexit 2>/dev/null | grep -E '[0-9] s:'
0 s:/C=US/ST=Alabama/L=Huntsville/O=The University of Alabama in
Huntsville/OU=Office of Information Technology (OIT)/CN=
libimsub-ms.voip.uah.edu
1 s:/C=US/ST=MI/L=Ann Arbor/O=Internet2/OU=InCommon/CN=InCommon RSA Server
CA
2 s:/C=US/ST=New Jersey/L=Jersey City/O=The USERTRUST Network/CN=USERTrust
RSA Certification Authority
3 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA
Certificate Services


--
Hunter Fuller (they)
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering


--
Hunter Fuller (they)
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering


On Sun, May 31, 2020 at 2:16 AM Anthony Holloway <
avholloway+cisco-voip@gmail.com> wrote:

> Probably confusion with the tomcat cert, versus the tomcat-trust.
>
> Remember this defect?
> https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuy13916
>
> On Sat, May 30, 2020 at 7:11 PM Hunter Fuller <hf0002@uah.edu> wrote:
>
>> I was wondering the same thing. You may be able to skip that one, but it
>> advises you to restart it when updating the callmanager-trust store, so we
>> did it.
>>
>> On Sat, May 30, 2020 at 19:10 Anthony Holloway <
>> avholloway+cisco-voip@gmail.com> wrote:
>>
>>> MVP
>>>
>>> But why restart TFTP?
>>>
>>> On Sat, May 30, 2020 at 7:02 PM Hunter Fuller <hf0002@uah.edu> wrote:
>>>
>>>> All,
>>>>
>>>> If you use certs whose trust is derived from the Sectigo root that
>>>> expired today, and your MRA isn’t working, I’ll try to save you a call to
>>>> TAC.
>>>>
>>>> Do all of these things:
>>>>
>>>> - Load the new intermediates and root into callmanager-trust and
>>>> tomcat-trust on all your UCMs
>>>> - restart tomcat, tftp, and callmanager on those boxes
>>>> - load the new intermediates and root into the CA trust store on all
>>>> expressways
>>>> - reboot the Expressway-Es
>>>>
>>>> If you need more detail or help, let me know, we just got off the phone
>>>> with TAC. Hope it helps.
>>>>
>>>> --
>>>>
>>>> --
>>>> Hunter Fuller (they)
>>>> Router Jockey
>>>> VBH Annex B-5
>>>> +1 256 824 5331
>>>>
>>>> Office of Information Technology
>>>> The University of Alabama in Huntsville
>>>> Network Engineering
>>>>
>>> _______________________________________________
>>>> cisco-voip mailing list
>>>> cisco-voip@puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>
>>> --
>>
>> --
>> Hunter Fuller (they)
>> Router Jockey
>> VBH Annex B-5
>> +1 256 824 5331
>>
>> Office of Information Technology
>> The University of Alabama in Huntsville
>> Network Engineering
>>
>

We have a handful of reasons for the certs in CUCM, some to do with MRA,
some not.

- Users use the Self Care Portal, and some pickier browsers don't like
being sent the wrong root.
- Something to do with SSO? I was never super clear on this. Either our
SSO IdP didn't trust the cert from UCM or the other way around. We fixed
both at the same time, so I guess I will never know.
- We are, in fact, checking crypto for all the Expressway tunnels
(ExpE-ExpC tunnel as well as ExpC-CUCM).

Honorable mention:
- Because it's a bad idea to leave expired certs laying around in there,
and I never know what one of my colleagues may have configured that relies
on the TLS verify working. :)

--
Hunter Fuller (they)
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering


On Wed, Jun 3, 2020 at 8:28 AM Anthony Holloway <
avholloway+cisco-voip@gmail.com> wrote:

> Hunter,
>
> I might be exposing a gap in my knowledge here, but why did you need these
> certs on CUCM?
>
> Cisco has now published a troubleshooting guide for this issue, and the
> article does not mention modifying CUCM cert store.
>
>
> https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/215561-troubleshooting-expressway-mra-login-and.html
>
> On Sat, May 30, 2020 at 7:02 PM Hunter Fuller <hf0002@uah.edu> wrote:
>
>> All,
>>
>> If you use certs whose trust is derived from the Sectigo root that
>> expired today, and your MRA isn’t working, I’ll try to save you a call to
>> TAC.
>>
>> Do all of these things:
>>
>> - Load the new intermediates and root into callmanager-trust and
>> tomcat-trust on all your UCMs
>> - restart tomcat, tftp, and callmanager on those boxes
>> - load the new intermediates and root into the CA trust store on all
>> expressways
>> - reboot the Expressway-Es
>>
>> If you need more detail or help, let me know, we just got off the phone
>> with TAC. Hope it helps.
>>
>> --
>>
>> --
>> Hunter Fuller (they)
>> Router Jockey
>> VBH Annex B-5
>> +1 256 824 5331
>>
>> Office of Information Technology
>> The University of Alabama in Huntsville
>> Network Engineering
>> _______________________________________________
>> cisco-voip mailing list
>> cisco-voip@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>
>

Ah ok, in your original email you only mentioned MRA, and so I was very
focused on how CUCM might need certs in the store for MRA. You are in fact
doing more than just MRA. Got it.

On Wed, Jun 3, 2020 at 11:35 AM Hunter Fuller <hf0002@uah.edu> wrote:

> We have a handful of reasons for the certs in CUCM, some to do with MRA,
> some not.
>
> - Users use the Self Care Portal, and some pickier browsers don't like
> being sent the wrong root.
> - Something to do with SSO? I was never super clear on this. Either our
> SSO IdP didn't trust the cert from UCM or the other way around. We fixed
> both at the same time, so I guess I will never know.
> - We are, in fact, checking crypto for all the Expressway tunnels
> (ExpE-ExpC tunnel as well as ExpC-CUCM).
>
> Honorable mention:
> - Because it's a bad idea to leave expired certs laying around in there,
> and I never know what one of my colleagues may have configured that relies
> on the TLS verify working. :)
>
> --
> Hunter Fuller (they)
> Router Jockey
> VBH Annex B-5
> +1 256 824 5331
>
> Office of Information Technology
> The University of Alabama in Huntsville
> Network Engineering
>
>
> On Wed, Jun 3, 2020 at 8:28 AM Anthony Holloway <
> avholloway+cisco-voip@gmail.com> wrote:
>
>> Hunter,
>>
>> I might be exposing a gap in my knowledge here, but why did you need
>> these certs on CUCM?
>>
>> Cisco has now published a troubleshooting guide for this issue, and the
>> article does not mention modifying CUCM cert store.
>>
>>
>> https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/215561-troubleshooting-expressway-mra-login-and.html
>>
>> On Sat, May 30, 2020 at 7:02 PM Hunter Fuller <hf0002@uah.edu> wrote:
>>
>>> All,
>>>
>>> If you use certs whose trust is derived from the Sectigo root that
>>> expired today, and your MRA isn’t working, I’ll try to save you a call to
>>> TAC.
>>>
>>> Do all of these things:
>>>
>>> - Load the new intermediates and root into callmanager-trust and
>>> tomcat-trust on all your UCMs
>>> - restart tomcat, tftp, and callmanager on those boxes
>>> - load the new intermediates and root into the CA trust store on all
>>> expressways
>>> - reboot the Expressway-Es
>>>
>>> If you need more detail or help, let me know, we just got off the phone
>>> with TAC. Hope it helps.
>>>
>>> --
>>>
>>> --
>>> Hunter Fuller (they)
>>> Router Jockey
>>> VBH Annex B-5
>>> +1 256 824 5331
>>>
>>> Office of Information Technology
>>> The University of Alabama in Huntsville
>>> Network Engineering
>>> _______________________________________________
>>> cisco-voip mailing list
>>> cisco-voip@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>
>>

We are doing a lot more than MRA, but it all kinda got wrapped up together.
The initial report being "Jabber no worky," and of course I know it's cert
related, but in my head I'm like, "well that's either an Expressway-E issue
or a CUCM issue or some issue with the traversal zone or..."

I guess if you don't use SSO, the certs probably matter a lot less. But if
you do use SSO, a lot of things happen that look like MRA issues but are
actually SSO sign-in or ticket issues.

Regarding monitoring cert expiration, emails from the CA are our go-to, but
it doesn't help when it's their damn root that's expiring. They were
shockingly silent on that, and when they did say something, it was
basically "this won't affect anything!" Wow, thanks.

--
Hunter Fuller (they)
Router Jockey
VBH Annex B-5
+1 256 824 5331

Office of Information Technology
The University of Alabama in Huntsville
Network Engineering


On Wed, Jun 3, 2020 at 12:26 PM Anthony Holloway <
avholloway+cisco-voip@gmail.com> wrote:

> Ah ok, in your original email you only mentioned MRA, and so I was very
> focused on how CUCM might need certs in the store for MRA. You are in fact
> doing more than just MRA. Got it.
>
> On Wed, Jun 3, 2020 at 11:35 AM Hunter Fuller <hf0002@uah.edu> wrote:
>
>> We have a handful of reasons for the certs in CUCM, some to do with MRA,
>> some not.
>>
>> - Users use the Self Care Portal, and some pickier browsers don't like
>> being sent the wrong root.
>> - Something to do with SSO? I was never super clear on this. Either our
>> SSO IdP didn't trust the cert from UCM or the other way around. We fixed
>> both at the same time, so I guess I will never know.
>> - We are, in fact, checking crypto for all the Expressway tunnels
>> (ExpE-ExpC tunnel as well as ExpC-CUCM).
>>
>> Honorable mention:
>> - Because it's a bad idea to leave expired certs laying around in there,
>> and I never know what one of my colleagues may have configured that relies
>> on the TLS verify working. :)
>>
>> --
>> Hunter Fuller (they)
>> Router Jockey
>> VBH Annex B-5
>> +1 256 824 5331
>>
>> Office of Information Technology
>> The University of Alabama in Huntsville
>> Network Engineering
>>
>>
>> On Wed, Jun 3, 2020 at 8:28 AM Anthony Holloway <
>> avholloway+cisco-voip@gmail.com> wrote:
>>
>>> Hunter,
>>>
>>> I might be exposing a gap in my knowledge here, but why did you need
>>> these certs on CUCM?
>>>
>>> Cisco has now published a troubleshooting guide for this issue, and the
>>> article does not mention modifying CUCM cert store.
>>>
>>>
>>> https://www.cisco.com/c/en/us/support/docs/unified-communications/expressway/215561-troubleshooting-expressway-mra-login-and.html
>>>
>>> On Sat, May 30, 2020 at 7:02 PM Hunter Fuller <hf0002@uah.edu> wrote:
>>>
>>>> All,
>>>>
>>>> If you use certs whose trust is derived from the Sectigo root that
>>>> expired today, and your MRA isn’t working, I’ll try to save you a call to
>>>> TAC.
>>>>
>>>> Do all of these things:
>>>>
>>>> - Load the new intermediates and root into callmanager-trust and
>>>> tomcat-trust on all your UCMs
>>>> - restart tomcat, tftp, and callmanager on those boxes
>>>> - load the new intermediates and root into the CA trust store on all
>>>> expressways
>>>> - reboot the Expressway-Es
>>>>
>>>> If you need more detail or help, let me know, we just got off the phone
>>>> with TAC. Hope it helps.
>>>>
>>>> --
>>>>
>>>> --
>>>> Hunter Fuller (they)
>>>> Router Jockey
>>>> VBH Annex B-5
>>>> +1 256 824 5331
>>>>
>>>> Office of Information Technology
>>>> The University of Alabama in Huntsville
>>>> Network Engineering
>>>> _______________________________________________
>>>> cisco-voip mailing list
>>>> cisco-voip@puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-voip
>>>>
>>>