Mailing List Archive

Ah split brains. Gotta love it. Did you restart the ones that did not move? I have seen this when things go stupid. The sync usually isn’t the problem it’s the real-time communication between the nodes that’s all messed up. Usually can fix with a node reboot or restarting the cucm and cti services. Course their maybe more to it but if things are in sync should not be hard to fix


Kent

> On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com> wrote:
> ?
> This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.
>
> Node configuration:
>
> DC A
> CM1: Pub which was re-ip’d
> CM2: Sub which was re-ip’d
>
> DC B:
> CM3: Sub at remote site that was not changed
>
> Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.
>
> The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.
>
> If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.
>
> I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.
>
> Anyone able to point me in the right direction to figure this out?
>
> Thanks.
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip

Thanks for the quick reply. I have a change request to restart the node that did not move tonight. I will follow up to let you know if this helped.

From: Kent Roberts <kent@fredf.org>
Sent: Tuesday, January 12, 2021 4:42 PM
To: Riley, Sean <SRiley@robinsonbradshaw.com>
Cc: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] CUCM call set up issue after migration

Ah split brains. Gotta love it. Did you restart the ones that did not move? I have seen this when things go stupid. The sync usually isn’t the problem it’s the real-time communication between the nodes that’s all messed up. Usually can fix with a node reboot or restarting the cucm and cti services. Course their maybe more to it but if things are in sync should not be hard to fix

Kent


On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:
?
This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.

Node configuration:

DC A
CM1: Pub which was re-ip’d
CM2: Sub which was re-ip’d

DC B:
CM3: Sub at remote site that was not changed

Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.

The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.

If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.

I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.

Anyone able to point me in the right direction to figure this out?

Thanks.
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

On UCM cli ‘utils diagnose test’ to ensure things look mostly okay from OS and low level perspective.

After that in UCMadmin ensure system->server shows the correct IP/name for each server.

After that ensure ccm process restarted (or whole server rebooted). Based on your description SDL links on tcp:8002 are not coming up because remote name/ip is not recognized against what is the the database and hosts file for name resolution.

-w

On Jan 12, 2021, at 4:05 PM, Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:

This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.

Node configuration:

DC A
CM1: Pub which was re-ip’d
CM2: Sub which was re-ip’d

DC B:
CM3: Sub at remote site that was not changed

Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.

The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.

If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.

I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.

Anyone able to point me in the right direction to figure this out?

Thanks.
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

This would be discovered in the “utils diagnose test” however, I have seen on more than one occasion, in a “node-move-IP-change” situation, where either the reverse DNS entry didn’t get changed, or more scrupulously, the incorrect one exists along with the correct one.

-Ryan

On Jan 12, 2021, at 16:55, Wes Sisk (wsisk) via cisco-voip <cisco-voip@puck.nether.net> wrote:

? On UCM cli ‘utils diagnose test’ to ensure things look mostly okay from OS and low level perspective.

After that in UCMadmin ensure system->server shows the correct IP/name for each server.

After that ensure ccm process restarted (or whole server rebooted). Based on your description SDL links on tcp:8002 are not coming up because remote name/ip is not recognized against what is the the database and hosts file for name resolution.

-w

On Jan 12, 2021, at 4:05 PM, Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:

This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.

Node configuration:

DC A
CM1: Pub which was re-ip’d
CM2: Sub which was re-ip’d

DC B:
CM3: Sub at remote site that was not changed

Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.

The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.

If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.

I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.

Anyone able to point me in the right direction to figure this out?

Thanks.
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Fcisco-voip&data=04%7C01%7C%7Ce25162930d9d4256897408d8b744bdc8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637460853350257444%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=I4pZOp2ABHuclYBN%2Fh4VJ0SCf16cGesrSQg8212xOqI%3D&reserved=0>

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip

After your reboot login into that box, do your standard silly checkes….. ping by name, db replication, reverse lookup etc….


Once you shut down CM3. The phones, (I assume….. you have other cm’s in the devicepool before SRST if your using it..)) should go over to your other cm’s. At that point you should be able to call them and everything should work as expected. (Never hurts to check this while your box is down, easier then doing another outage window)

My place has about 15 nodes, and thousands of end points. This happens at times with upgrade etc, TAC says its not supposed to happen, but…

Hopefully that will fix it for you.

If it doesn’t fix it, after 20 mins or so, you might need to get tac involved if you are not good with reading traces to figure out why.




> On Jan 12, 2021, at 2:47 PM, Riley, Sean <SRiley@robinsonbradshaw.com> wrote:
>
> Thanks for the quick reply. I have a change request to restart the node that did not move tonight. I will follow up to let you know if this helped.
>
> From: Kent Roberts <kent@fredf.org>
> Sent: Tuesday, January 12, 2021 4:42 PM
> To: Riley, Sean <SRiley@robinsonbradshaw.com>
> Cc: cisco-voip@puck.nether.net
> Subject: Re: [cisco-voip] CUCM call set up issue after migration
>
> Ah split brains. Gotta love it. Did you restart the ones that did not move? I have seen this when things go stupid. The sync usually isn’t the problem it’s the real-time communication between the nodes that’s all messed up. Usually can fix with a node reboot or restarting the cucm and cti services. Course their maybe more to it but if things are in sync should not be hard to fix
>
>
> Kent
>
>
> On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com <mailto:SRiley@robinsonbradshaw.com>> wrote:
>
> ?
> This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.
>
> Node configuration:
>
> DC A
> CM1: Pub which was re-ip’d
> CM2: Sub which was re-ip’d
>
> DC B:
> CM3: Sub at remote site that was not changed
>
> Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.
>
> The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.
>
> If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.
>
> I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.
>
> Anyone able to point me in the right direction to figure this out?
>
> Thanks.
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net <mailto:cisco-voip@puck.nether.net>
> https://puck.nether.net/mailman/listinfo/cisco-voip <https://puck.nether.net/mailman/listinfo/cisco-voip>

So far it seems the reboot resolved this problem. Thanks for all the replies with guidance and help.

From: Kent Roberts <kent@fredf.org>
Sent: Tuesday, January 12, 2021 4:42 PM
To: Riley, Sean <SRiley@robinsonbradshaw.com>
Cc: cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] CUCM call set up issue after migration

Ah split brains. Gotta love it. Did you restart the ones that did not move? I have seen this when things go stupid. The sync usually isn’t the problem it’s the real-time communication between the nodes that’s all messed up. Usually can fix with a node reboot or restarting the cucm and cti services. Course their maybe more to it but if things are in sync should not be hard to fix

Kent


On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:
?
This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.

Node configuration:

DC A
CM1: Pub which was re-ip’d
CM2: Sub which was re-ip’d

DC B:
CM3: Sub at remote site that was not changed

Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.

The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.

If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.

I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.

Anyone able to point me in the right direction to figure this out?

Thanks.
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip

Nice! That was easy. This email chain goes into my folder called: When
Someone Says Only Windows Servers Need Reboots.

On Tue, Jan 12, 2021 at 7:40 PM Riley, Sean <SRiley@robinsonbradshaw.com>
wrote:

> So far it seems the reboot resolved this problem. Thanks for all the
> replies with guidance and help.
>
>
>
> *From:* Kent Roberts <kent@fredf.org>
> *Sent:* Tuesday, January 12, 2021 4:42 PM
> *To:* Riley, Sean <SRiley@robinsonbradshaw.com>
> *Cc:* cisco-voip@puck.nether.net
> *Subject:* Re: [cisco-voip] CUCM call set up issue after migration
>
>
>
> Ah split brains. Gotta love it. Did you restart the ones that did not
> move? I have seen this when things go stupid. The sync usually isn’t the
> problem it’s the real-time communication between the nodes that’s all
> messed up. Usually can fix with a node reboot or restarting the cucm
> and cti services. Course their maybe more to it but if things are in sync
> should not be hard to fix
>
>
>
> Kent
>
>
>
> On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com>
> wrote:
>
> ?
>
> This past weekend we migrated 2 CUCM servers to a new datacenter. This
> involved changing the IP address on these 2 CUCM nodes. These 2 nodes
> consist of the Publisher and 1 Subscriber. We have another Sub at a remote
> datacenter that was not touched this past weekend.
>
>
>
> Node configuration:
>
>
>
> DC A
>
> CM1: Pub which was re-ip’d
>
> CM2: Sub which was re-ip’d
>
>
>
> DC B:
>
> CM3: Sub at remote site that was not changed
>
>
>
> Phones are at many sites, but issue is independent of the phone type,
> phone location or subnet. Also, Expressway phones have the same issue.
>
>
>
> The issue is any phone that is registered to CM3 cannot call phones
> registered to CM1 or CM2 and vice versa. The phones do not see the call
> coming in. If SNR is configured, the call will ring to the remote
> destination. Phones registered to CM3 can make outbound PSTN calls without
> issue, but not receive inbound from PSTN (probably because the gateway is
> handing off to CM1 or CM2). While the gateways are not unique to the
> issue, they are running H323.
>
>
>
> If the phones are both registered to CM3, they can call each other, but
> not phones registered to CM1 or CM2.
>
>
>
> I have had my network team verify there is not anything they can see in
> the network causing this behavior. Database replication checks out OK and I
> can ping from/to each node.
>
>
>
> Anyone able to point me in the right direction to figure this out?
>
>
>
> Thanks.
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>

Yes, I like easy. I wanted to reboot the node earlier, but was told that should not be necessary. Should have trusted my gut. Now onto the other issues left over from this migration.

Jabber line presence is extremely delayed. Have an IM&P cluster reboot scheduled for tonight.

From: Anthony Holloway <avholloway+cisco-voip@gmail.com>
Sent: Wednesday, January 13, 2021 12:24 AM
To: Riley, Sean <SRiley@robinsonbradshaw.com>
Cc: Kent Roberts <kent@fredf.org>; cisco-voip@puck.nether.net
Subject: Re: [cisco-voip] CUCM call set up issue after migration

Nice! That was easy. This email chain goes into my folder called: When Someone Says Only Windows Servers Need Reboots.

On Tue, Jan 12, 2021 at 7:40 PM Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:
So far it seems the reboot resolved this problem. Thanks for all the replies with guidance and help.

From: Kent Roberts <kent@fredf.org<mailto:kent@fredf.org>>
Sent: Tuesday, January 12, 2021 4:42 PM
To: Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>>
Cc: cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
Subject: Re: [cisco-voip] CUCM call set up issue after migration

Ah split brains. Gotta love it. Did you restart the ones that did not move? I have seen this when things go stupid. The sync usually isn’t the problem it’s the real-time communication between the nodes that’s all messed up. Usually can fix with a node reboot or restarting the cucm and cti services. Course their maybe more to it but if things are in sync should not be hard to fix

Kent

On Jan 12, 2021, at 14:06, Riley, Sean <SRiley@robinsonbradshaw.com<mailto:SRiley@robinsonbradshaw.com>> wrote:
?
This past weekend we migrated 2 CUCM servers to a new datacenter. This involved changing the IP address on these 2 CUCM nodes. These 2 nodes consist of the Publisher and 1 Subscriber. We have another Sub at a remote datacenter that was not touched this past weekend.

Node configuration:

DC A
CM1: Pub which was re-ip’d
CM2: Sub which was re-ip’d

DC B:
CM3: Sub at remote site that was not changed

Phones are at many sites, but issue is independent of the phone type, phone location or subnet. Also, Expressway phones have the same issue.

The issue is any phone that is registered to CM3 cannot call phones registered to CM1 or CM2 and vice versa. The phones do not see the call coming in. If SNR is configured, the call will ring to the remote destination. Phones registered to CM3 can make outbound PSTN calls without issue, but not receive inbound from PSTN (probably because the gateway is handing off to CM1 or CM2). While the gateways are not unique to the issue, they are running H323.

If the phones are both registered to CM3, they can call each other, but not phones registered to CM1 or CM2.

I have had my network team verify there is not anything they can see in the network causing this behavior. Database replication checks out OK and I can ping from/to each node.

Anyone able to point me in the right direction to figure this out?

Thanks.
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net<mailto:cisco-voip@puck.nether.net>
https://puck.nether.net/mailman/listinfo/cisco-voip