Mailing List Archive

Of Expressways and max-forwards...
Afternoon all, my team and I just got to the bottom of a particularly gnarly problem with a pair of new SIP trunks which I’ll explain in case it’s of use to others, but I have a question at the end regarding SIP configuration on Expressways, particularly in traversal/MRA zones.

In summary, a small but reproducible volume of calls were silently failing when routed over our new SIP trunks rather than our legacy ISDN30 circuits. We were getting a "483 Too Many Hops" error back from the TSP indication we’d reached the hop limit specified for connecting the call. Most calls were being set up with max-forwards=70 (the default) but certain calls were exiting our network through our CUBES with it set to 12 or 13. Calls from both physical and Jabber softphones where affected, although notably only newer 8800 series SIP handsets.

I tried forcing max-forwards on the CUBEs to 70 but this didn’t change the outgoing calls that were already having problems.

Eventually we narrowed it down to calls from MRA registered devices on our expressways (mostly Jabber but with a small number of 8845s in staff home offices), as all failed calls had the same source IP address when we looked at the corresponding CDRs; although this wasn’t visible in the SIP traces which made diagnosis harder (source IP address is the subscriber when looking at the CUBE’s SIP ). A quick look at edge and core expressways showed that “max hops” was to set to 15 in the relevant zones. Cisco documentation says this is the default, but suggests to set it higher if calls are failing with a 483 code. So we set the max hops to 70 and calls are now connecting as expected.



So…question: why is the max hops set so low (15) on expressway zones by default when it’s set to 70 on CUBEs, and is there anything this is likely to break/that I should look out for now I’ve made the change?


---
/-Gary Parker----------------------------------f--\
| Unified Communications Service Manager |
n Loughborough University, IT Services |
| tel:+441509635635 sip:gary@lboro.ac.uk o
| https://www.osx.ninja/pubkey.txt |
\r----------------------------------------------d-/

_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Of Expressways and max-forwards... [ In reply to ]
Thanks for sharing. My expressways were set up by professional services with almost little to no “learning” involved.

Also, I think we should follow in the footsteps of our scientific brothers and sisters and name these anomalies after the person that first finds them.

;)

Sent from my iPhone

> On Nov 15, 2021, at 10:41 AM, Gary Parker <G.J.Parker@lboro.ac.uk> wrote:
>
> ?CAUTION: This email originated from outside of the University of Guelph. Do not click links or open attachments unless you recognize the sender and know the content is safe. If in doubt, forward suspicious emails to IThelp@uoguelph.ca
>
>
> Afternoon all, my team and I just got to the bottom of a particularly gnarly problem with a pair of new SIP trunks which I’ll explain in case it’s of use to others, but I have a question at the end regarding SIP configuration on Expressways, particularly in traversal/MRA zones.
>
> In summary, a small but reproducible volume of calls were silently failing when routed over our new SIP trunks rather than our legacy ISDN30 circuits. We were getting a "483 Too Many Hops" error back from the TSP indication we’d reached the hop limit specified for connecting the call. Most calls were being set up with max-forwards=70 (the default) but certain calls were exiting our network through our CUBES with it set to 12 or 13. Calls from both physical and Jabber softphones where affected, although notably only newer 8800 series SIP handsets.
>
> I tried forcing max-forwards on the CUBEs to 70 but this didn’t change the outgoing calls that were already having problems.
>
> Eventually we narrowed it down to calls from MRA registered devices on our expressways (mostly Jabber but with a small number of 8845s in staff home offices), as all failed calls had the same source IP address when we looked at the corresponding CDRs; although this wasn’t visible in the SIP traces which made diagnosis harder (source IP address is the subscriber when looking at the CUBE’s SIP ). A quick look at edge and core expressways showed that “max hops” was to set to 15 in the relevant zones. Cisco documentation says this is the default, but suggests to set it higher if calls are failing with a 483 code. So we set the max hops to 70 and calls are now connecting as expected.
>
>
>
> So…question: why is the max hops set so low (15) on expressway zones by default when it’s set to 70 on CUBEs, and is there anything this is likely to break/that I should look out for now I’ve made the change?
>
>
> ---
> /-Gary Parker----------------------------------f--\
> | Unified Communications Service Manager |
> n Loughborough University, IT Services |
> | tel:+441509635635 sip:gary@lboro.ac.uk o
> | https://www.osx.ninja/pubkey.txt |
> \r----------------------------------------------d-/
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip
Re: Of Expressways and max-forwards... [ In reply to ]
Isn't Calmanager Service Parameter for max-forwards 12? Says if QSIG set to
15. Nothing about if SIP set to 70.

is the value reset at CUBE to PSTN to 70 on outgoing? that is what logs
seems to show. Makes sense if CUBE is IP-IP gateway.

On Mon, Nov 15, 2021, 10:48 AM Gary Parker <G.J.Parker@lboro.ac.uk> wrote:

> Afternoon all, my team and I just got to the bottom of a particularly
> gnarly problem with a pair of new SIP trunks which I’ll explain in case
> it’s of use to others, but I have a question at the end regarding SIP
> configuration on Expressways, particularly in traversal/MRA zones.
>
> In summary, a small but reproducible volume of calls were silently failing
> when routed over our new SIP trunks rather than our legacy ISDN30 circuits.
> We were getting a "483 Too Many Hops" error back from the TSP indication
> we’d reached the hop limit specified for connecting the call. Most calls
> were being set up with max-forwards=70 (the default) but certain calls were
> exiting our network through our CUBES with it set to 12 or 13. Calls from
> both physical and Jabber softphones where affected, although notably only
> newer 8800 series SIP handsets.
>
> I tried forcing max-forwards on the CUBEs to 70 but this didn’t change the
> outgoing calls that were already having problems.
>
> Eventually we narrowed it down to calls from MRA registered devices on our
> expressways (mostly Jabber but with a small number of 8845s in staff home
> offices), as all failed calls had the same source IP address when we looked
> at the corresponding CDRs; although this wasn’t visible in the SIP traces
> which made diagnosis harder (source IP address is the subscriber when
> looking at the CUBE’s SIP ). A quick look at edge and core expressways
> showed that “max hops” was to set to 15 in the relevant zones. Cisco
> documentation says this is the default, but suggests to set it higher if
> calls are failing with a 483 code. So we set the max hops to 70 and calls
> are now connecting as expected.
>
>
>
> So…question: why is the max hops set so low (15) on expressway zones by
> default when it’s set to 70 on CUBEs, and is there anything this is likely
> to break/that I should look out for now I’ve made the change?
>
>
> ---
> /-Gary Parker----------------------------------f--\
> | Unified Communications Service Manager |
> n Loughborough University, IT Services |
> | tel:+441509635635 sip:gary@lboro.ac.uk o
> | https://www.osx.ninja/pubkey.txt |
> \r----------------------------------------------d-/
>
> _______________________________________________
> cisco-voip mailing list
> cisco-voip@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-voip
>
Re: Of Expressways and max-forwards... [ In reply to ]
I think maybe you're referring to "forward maximum hop count". That parameter is used to guard against call routing loops caused by numbers being forwarded.

There is also a "max hop" parameter for multicast music on hold. That might be more related to high cpu on boundary devices or failing to receive MMoH.

The risk of increasing the forward hop loop detection is that call routing loops caused by forwarded numbers will consume more circuits. For IP circuits this may not be a big deal. For your pots/t1/e1 circuits you are more likely to consume all available channels and experience fully used circuits. When this happens you're like to see spikes in CPU usage and may experience call throttling like code yellow depending on the rate and extent of the forward loop.

Forward max hop count was put in place to address the situation where 2 users forward their calls to each other. Example: Market and Sales both leave for different appointments and forward their call to the other "for coverage". A call to marketing will reach that marketing line and be forwarded to sales. It reaches sales and is forwarded to marketing. Forwards can be internal or external, and can use number translations and different call routing paths. The forwards are end user initiated and the end user stations/lines may existing on separate systems where there is no single administrative, monitoring, or operational domain.

IP trunks do change this picture some. Call flows that involve this many forwards may be worth a design review.

Expected impact: In the event of a call forwarding loop you are more likely to consume all available traditional trunks. You may experience high cpu and potentially call throttling or code yellow.
Unexpected impacts: Introducing that many touch points in a call flow creates many opportunities for unexpected impacts. Pretty cool that that many call processing events can happen in close enough to real time for the call to be delivered end to end.

Good persistence and great find Gary! Even more kudos for taking time to share your experience with the community.

-Wes

On Nov 15, 2021, at 10:34 AM, Gary Parker <G.J.Parker@lboro.ac.uk<mailto:G.J.Parker@lboro.ac.uk>> wrote:

Eventually we narrowed it down to calls from MRA registered devices on our expressways (mostly Jabber but with a small number of 8845s in staff home offices), as all failed calls had the same source IP address when we looked at the corresponding CDRs; although this wasn’t visible in the SIP traces which made diagnosis harder (source IP address is the subscriber when looking at the CUBE’s SIP ). A quick look at edge and core expressways showed that “max hops” was to set to 15 in the relevant zones. Cisco documentation says this is the default, but suggests to set it higher if calls are failing with a 483 code. So we set the max hops to 70 and calls are now connecting as expected.



So…question: why is the max hops set so low (15) on expressway zones by default when it’s set to 70 on CUBEs, and is there anything this is likely to break/that I should look out for now I’ve made the change?
Re: Of Expressways and max-forwards... [ In reply to ]
> On 15 Nov 2021, at 15:34, Gary Parker <g.j.parker@lboro.ac.uk> wrote:
>
> ...
>
> So…question: why is the max hops set so low (15) on expressway zones by default when it’s set to 70 on CUBEs, and is there anything this is likely to break/that I should look out for now I’ve made the change?

Thanks for all the feedback and taking the time to reply, folks. A few follow-ups below:

> My expressways were set up by professional services with almost little to no “learning” involved.

Same here. It’s one of those things that’s “always worked” since it was put in so I’ve never had the impetus to learn about it, sadly :-/

> Isn't Calmanager Service Parameter for max-forwards 12? Says if QSIG set to 15. Nothing about if SIP set to 70.

Having just checked it, yes it is.

Cisco seems to use “maximum hops” and “maximum forwards” interchangeably on different systems, which is less than helpful.

In the CallManager Service Parameters we have “Forward Maximum Hop Count”, which controls the number of times a call can be *forwarded* within the cluster, ie. from one DN to another. I don’t believe this has an impact on SIP “max-forwards” when passing call from router to router when routing calls to PSTN.

Damnit Cisco, pick a word and stick with it :-)

(I know, I know…there’s history…)

> is the value reset at CUBE to PSTN to 70 on outgoing? that is what logs seems to show. Makes sense if CUBE is IP-IP gateway.


Yes. I can see that the initial INVITE of a SIP call passed from CUCM to CUBE has Max-Forwards <70 (as it passes through my campus network), but the corresponding INVITE sent to my TSP has it reset to 70.

> All the things Wes said

Thanks, that all makes sense wrt to causing internal loops. I think the problem here, as alluded to earlier, is that Cisco mixes use of maximum “hops" and “forwards" in different contexts (no doubt IETF SIP standards are also partly to blame), and that the defaults on Expressways weren’t set up with SIP PSTN access in mind.

I should also apologise at this point for an error in my previous post: it was not the “max hops” parameter that I had to change in the Zones on the core and edge expressways, but “Hop count”, which is somewhat unintuitive imho.

It’s interesting that CUBE seems to respect and preserve the max-forwards field that’s set on calls via the expressways, but not on those from directly registered CUCM clients. FWIW I’ve not looked at the behaviour wrt to SCCP devices; that may be different again.

Anyways, thanks again folks.

Gary
_______________________________________________
cisco-voip mailing list
cisco-voip@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-voip