Mailing List Archive

Modem recovery can automatically busy out modems, if it should see
excessive failures on certain modems and decide that it wants to
"recover" the module.

There are probably some other reasons why the router will decide
automatically to busy out modems, ... oh, I just though of one, that
little featurette where where we proactively busy out modems to signal
to the switch that the trunks are busy to force the switch to hunt to a
different router.

Maybe if you look at the modem logs, they'll provide some explanation,
unless they have rolled over past the busyout event.

Aaron

------------------------------------------------------------------------


> Router1 is AS5300 with 12.3(18.8) - 2.9.5.0
>
> router1>sh mode sum
> Avg Hold Incoming calls Outgoing calls Busied Failed No Succ
> Time Succ Fail Avail Succ Fail Avail Out Dial Ans Pct.
> 00:32:21 65667 4330 91 0 0 0 54 0 0 94%
>
> Router2 is AS5300 with 12.2(15)T16 - 2.9.5.0
>
> router2>sh modem sum
> Avg Hold Incoming calls Outgoing calls Busied Failed No Succ
> Time Succ Fail Avail Succ Fail Avail Out Dial Ans Pct.
> 00:31:52 435989 25314 76 0 0 0 0 0 4 95%
>
>
> Router1 is the only AS5300 running 12.3(18.8) and is the only one showing busied out calls during
> the last 2 months. Previously, when it was running 12.2(15)T16 too, no busied calls were there for
> over a year.
>
> So, is there a known bug with "busied out" calls in 12.3.18 IOS? I couldn't find anything in bugtool.
>
>
> --
> Tassos
> _______________________________________________
> cisco-nas mailing list
> cisco-nas@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nas
>

_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas

Hi Aaron,

Aaron Leonard wrote on 26/5/2006 2:54 ðì:
> Modem recovery can automatically busy out modems, if it should see
> excessive failures on certain modems and decide that it wants to
> "recover" the module.
>

I have neven seen on 12.2(15)T16 busied out calls (we're talking about 20 routers with this IOS),
although i get messages like:

May 25 04:00:00: %MODEM-1-MODEMOK: Modem (2/96) Running old firmware
May 25 04:38:00: %MICA-5-MODEM_RECOVERY: Modem (2/96) is being recovered by asap-redownload
May 25 04:38:06: %MODEM-5-DL_START: Modem (2/96) started firmware download
May 25 04:38:19: %MODEM-5-DL_GOOD: Modem (2/96) completed firmware download:

On the other hand, in 12.3.18 (only one router and in the same hunt group with the other 20), i
never got such messages, but i had quite a lot of busied out calls.

> There are probably some other reasons why the router will decide
> automatically to busy out modems, ... oh, I just though of one, that
> little featurette where where we proactively busy out modems to signal
> to the switch that the trunks are busy to force the switch to hunt to a
> different router.
>

Can you explain it a little bit more?
Why do you proactively busy out the modems in this case (signal to the switch), when the isdn
channels are the ones that should be busied out?

> Maybe if you look at the modem logs, they'll provide some explanation,
> unless they have rolled over past the busyout event.
>

I just reloaded the box :( and loaded 12.3.19....i'm waiting for busied out calls to appear again.
Then i'll try to grab the modem logs.

--
Tassos

> Aaron
>
> ------------------------------------------------------------------------
>
>
>> Router1 is AS5300 with 12.3(18.8) - 2.9.5.0
>>
>> router1>sh mode sum
>> Avg Hold Incoming calls Outgoing calls Busied Failed
>> No Succ
>> Time Succ Fail Avail Succ Fail Avail Out
>> Dial Ans Pct.
>> 00:32:21 65667 4330 91 0 0 0 54
>> 0 0 94%
>>
>> Router2 is AS5300 with 12.2(15)T16 - 2.9.5.0
>>
>> router2>sh modem sum
>> Avg Hold Incoming calls Outgoing calls Busied Failed
>> No Succ
>> Time Succ Fail Avail Succ Fail Avail Out
>> Dial Ans Pct.
>> 00:31:52 435989 25314 76 0 0 0 0
>> 0 4 95%
>>
>>
>> Router1 is the only AS5300 running 12.3(18.8) and is the only one
>> showing busied out calls during the last 2 months. Previously, when it
>> was running 12.2(15)T16 too, no busied calls were there for over a year.
>>
>> So, is there a known bug with "busied out" calls in 12.3.18 IOS? I
>> couldn't find anything in bugtool.
>>
>>
>> --
>> Tassos
>> _______________________________________________
>> cisco-nas mailing list
>> cisco-nas@puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nas
>>
>
_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas

Hi Tassos,

>> Modem recovery can automatically busy out modems, if it should see
>> excessive failures on certain modems and decide that it wants to
>> "recover" the module.
>>
>
> I have neven seen on 12.2(15)T16 busied out calls (we're talking about
> 20 routers with this IOS), although i get messages like:
>
> May 25 04:00:00: %MODEM-1-MODEMOK: Modem (2/96) Running old firmware
> May 25 04:38:00: %MICA-5-MODEM_RECOVERY: Modem (2/96) is being
> recovered by asap-redownload
> May 25 04:38:06: %MODEM-5-DL_START: Modem (2/96) started firmware
> download
> May 25 04:38:19: %MODEM-5-DL_GOOD: Modem (2/96) completed firmware
> download:
>
> On the other hand, in 12.3.18 (only one router and in the same hunt
> group with the other 20), i never got such messages, but i had quite a
> lot of busied out calls.

Doesn't the modem log show *anything* when the busied counter increments
for a modem?

>> There are probably some other reasons why the router will decide
>> automatically to busy out modems, ... oh, I just though of one, that
>> little featurette where where we proactively busy out modems to
>> signal to the switch that the trunks are busy to force the switch to
>> hunt to a different router.
>>
>
> Can you explain it a little bit more?
> Why do you proactively busy out the modems in this case (signal to the
> switch), when the isdn channels are the ones that should be busied out?

Um, you're right, it was my foggy memory that was faulty.

>> Maybe if you look at the modem logs, they'll provide some
>> explanation, unless they have rolled over past the busyout event.
>>
>
> I just reloaded the box :( and loaded 12.3.19....i'm waiting for
> busied out calls to appear again.
> Then i'll try to grab the modem logs.
>
> --
> Tassos

OK.

I do have one concern that I should raise here, about using current 12.3
mainline code on MICA platforms - (such as the 5300, or the 3600/3700
with NM-DM modems) - see below. I think if I were running such a
platform, I would be tempted to run latest 12.3(6) throttle (12.3(6f))
rather than a subsequent 12.3M release.

Regards,

Aaron

---

CSCei63851
mica modems randomly marked as bad in any version afer 12.3(6)
Release-note: Modified 050919

Click to see the enclosure with line wrapping enabled


In 3640 and 5300 platform running any version after 12.3(6), mica modems
may randomly
marked as BAD

The symptoms observed only in situations where call success rate(CSR) is
very
low and there are good amount of calls failing continously in modem to modem
trainup

Modem log of modem in bad state, showed that IOS put modem out of service
because modem failed to go onhook/offhook

Workaround
-Use 12.3(6)d or any version before it

-Manually download the modem portware to spe that has bad modem using
"firmware location.." command in spe mode

-Reboot the router

-Make sure many calls are not failing in trainup that is CSR is good

---

This is marked Unreproducible, although it probably should be reopened,
as this problem is seen on 5300s and 36/3700s on all 12.3 mainline after
the 12.3(6) throttle.

------------------------------------------------------------------------

>
>> Aaron
>>
>> ------------------------------------------------------------------------
>>
>>
>>> Router1 is AS5300 with 12.3(18.8) - 2.9.5.0
>>>
>>> router1>sh mode sum
>>> Avg Hold Incoming calls Outgoing calls Busied
>>> Failed No Succ
>>> Time Succ Fail Avail Succ Fail Avail Out
>>> Dial Ans Pct.
>>> 00:32:21 65667 4330 91 0 0 0 54
>>> 0 0 94%
>>>
>>> Router2 is AS5300 with 12.2(15)T16 - 2.9.5.0
>>>
>>> router2>sh modem sum
>>> Avg Hold Incoming calls Outgoing calls Busied
>>> Failed No Succ
>>> Time Succ Fail Avail Succ Fail Avail Out
>>> Dial Ans Pct.
>>> 00:31:52 435989 25314 76 0 0 0 0
>>> 0 4 95%
>>>
>>>
>>> Router1 is the only AS5300 running 12.3(18.8) and is the only one
>>> showing busied out calls during the last 2 months. Previously, when
>>> it was running 12.2(15)T16 too, no busied calls were there for over
>>> a year.
>>>
>>> So, is there a known bug with "busied out" calls in 12.3.18 IOS? I
>>> couldn't find anything in bugtool.
>>>
>>>
>>> --
>>> Tassos
>>> _______________________________________________
>>> cisco-nas mailing list
>>> cisco-nas@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-nas
>>>
>>

After trying 12.3.19 for 40 days, it showed the same behavior.

Avg Hold Inc calls Out calls Busied Failed No Succ
Mdm Time Succ Fail Succ Fail Out Dial Answer Pct.
.........
* 2/30 00:29:40 344 47 0 0 1 0 0 88%
* 2/31 00:30:08 370 21 0 0 1 0 0 95%
2/32 00:30:54 374 24 0 0 1 0 0 94%
2/33 00:27:56 370 24 0 0 1 0 0 94%
* 2/34 00:29:04 368 25 0 0 1 0 0 94%
* 2/35 00:32:53 357 18 0 0 1 0 0 95%
.........
Total: 00:30:12 43876 2605 0 0 6 0 1 94%

Unfortunately the modem logs didn't show anything, because their log buffer (which was the default)
was overwritten with the latest entries. I increased the modem log buffer (modem buffer-size 500),
so i'm waiting for it to appear again.


About the "on AS5300 better stick with <12.3(6)d" issue, i think if that was an option i would
prefer to move back to 12.2(15)T since it has proven quite stable all these years.

I just wanted to upgrade to >12.3.18 mainline in order to have a GD IOS in my AS5300s (and avoid
some security vulnerabilities), but i guess this GD wasn't meant for all routers. Quite strange that
some ED/LD releases are preferred over GD ones.

--
Tassos

Aaron Leonard wrote on 26/5/2006 20:28:
> Hi Tassos,
>
>>> Modem recovery can automatically busy out modems, if it should see
>>> excessive failures on certain modems and decide that it wants to
>>> "recover" the module.
>>>
>>
>> I have neven seen on 12.2(15)T16 busied out calls (we're talking about
>> 20 routers with this IOS), although i get messages like:
>>
>> May 25 04:00:00: %MODEM-1-MODEMOK: Modem (2/96) Running old firmware
>> May 25 04:38:00: %MICA-5-MODEM_RECOVERY: Modem (2/96) is being
>> recovered by asap-redownload
>> May 25 04:38:06: %MODEM-5-DL_START: Modem (2/96) started firmware
>> download
>> May 25 04:38:19: %MODEM-5-DL_GOOD: Modem (2/96) completed firmware
>> download:
>>
>> On the other hand, in 12.3.18 (only one router and in the same hunt
>> group with the other 20), i never got such messages, but i had quite a
>> lot of busied out calls.
>
> Doesn't the modem log show *anything* when the busied counter increments
> for a modem?
>
>>> There are probably some other reasons why the router will decide
>>> automatically to busy out modems, ... oh, I just though of one, that
>>> little featurette where where we proactively busy out modems to
>>> signal to the switch that the trunks are busy to force the switch to
>>> hunt to a different router.
>>>
>>
>> Can you explain it a little bit more?
>> Why do you proactively busy out the modems in this case (signal to the
>> switch), when the isdn channels are the ones that should be busied out?
>
> Um, you're right, it was my foggy memory that was faulty.
>
>>> Maybe if you look at the modem logs, they'll provide some
>>> explanation, unless they have rolled over past the busyout event.
>>>
>>
>> I just reloaded the box :( and loaded 12.3.19....i'm waiting for
>> busied out calls to appear again.
>> Then i'll try to grab the modem logs.
>>
>> --
>> Tassos
>
> OK.
>
> I do have one concern that I should raise here, about using current 12.3
> mainline code on MICA platforms - (such as the 5300, or the 3600/3700
> with NM-DM modems) - see below. I think if I were running such a
> platform, I would be tempted to run latest 12.3(6) throttle (12.3(6f))
> rather than a subsequent 12.3M release.
>
> Regards,
>
> Aaron
>
> ---
>
> CSCei63851
> mica modems randomly marked as bad in any version afer 12.3(6)
> Release-note: Modified 050919
>
> Click to see the enclosure with line wrapping enabled
>
>
> In 3640 and 5300 platform running any version after 12.3(6), mica modems
> may randomly
> marked as BAD
>
> The symptoms observed only in situations where call success rate(CSR) is
> very
> low and there are good amount of calls failing continously in modem to modem
> trainup
>
> Modem log of modem in bad state, showed that IOS put modem out of service
> because modem failed to go onhook/offhook
>
> Workaround
> -Use 12.3(6)d or any version before it
>
> -Manually download the modem portware to spe that has bad modem using
> "firmware location.." command in spe mode
>
> -Reboot the router
>
> -Make sure many calls are not failing in trainup that is CSR is good
>
> ---
>
> This is marked Unreproducible, although it probably should be reopened,
> as this problem is seen on 5300s and 36/3700s on all 12.3 mainline after
> the 12.3(6) throttle.
>
> ------------------------------------------------------------------------
>
>>
>>> Aaron
>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>>> Router1 is AS5300 with 12.3(18.8) - 2.9.5.0
>>>>
>>>> router1>sh mode sum
>>>> Avg Hold Incoming calls Outgoing calls Busied
>>>> Failed No Succ
>>>> Time Succ Fail Avail Succ Fail Avail Out
>>>> Dial Ans Pct.
>>>> 00:32:21 65667 4330 91 0 0 0 54
>>>> 0 0 94%
>>>>
>>>> Router2 is AS5300 with 12.2(15)T16 - 2.9.5.0
>>>>
>>>> router2>sh modem sum
>>>> Avg Hold Incoming calls Outgoing calls Busied
>>>> Failed No Succ
>>>> Time Succ Fail Avail Succ Fail Avail Out
>>>> Dial Ans Pct.
>>>> 00:31:52 435989 25314 76 0 0 0 0
>>>> 0 4 95%
>>>>
>>>>
>>>> Router1 is the only AS5300 running 12.3(18.8) and is the only one
>>>> showing busied out calls during the last 2 months. Previously, when
>>>> it was running 12.2(15)T16 too, no busied calls were there for over
>>>> a year.
>>>>
>>>> So, is there a known bug with "busied out" calls in 12.3.18 IOS? I
>>>> couldn't find anything in bugtool.
>>>>
>>>>
>>>> --
>>>> Tassos
>>>> _______________________________________________
>>>> cisco-nas mailing list
>>>> cisco-nas@puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-nas
>>>>
>>>
>
_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas

Tassos,

OK, I figure that you are hitting the CSCei63851 issue as noted below.

CSCei63851 AS5300: mica modems randomly marked as bad in any version
after 12.3(6)

This bug is marked closed meaning that we do not intend to fix it, as
the AS5300 reached end of engineering (software maintenance) a year ago.

So, yes, if you have an AS5300, you're probably best off using a 12.3(6)
throttle or (if you prefer) 12.2(15)T throttle rebuild.

It still would be good, if feasible for you, to open a case and have it
linked to CSCei63851. Even though we are not going to fix CSCei63851,
this would at least let us gauge how widespread its impact is. Which
would in turn be relevant for PSIRT ... if we have a PSIRT-forced
bugfix, we would want to put it into a pre-CSCei63851 throttle for the
benefit of MICA users.

Regards,

Aaron

---

> After trying 12.3.19 for 40 days, it showed the same behavior.
>
> Avg Hold Inc calls Out calls Busied Failed
> No Succ
> Mdm Time Succ Fail Succ Fail Out Dial
> Answer Pct.
> .........
> * 2/30 00:29:40 344 47 0 0 1 0
> 0 88%
> * 2/31 00:30:08 370 21 0 0 1 0
> 0 95%
> 2/32 00:30:54 374 24 0 0 1 0
> 0 94%
> 2/33 00:27:56 370 24 0 0 1 0
> 0 94%
> * 2/34 00:29:04 368 25 0 0 1 0
> 0 94%
> * 2/35 00:32:53 357 18 0 0 1 0
> 0 95%
> .........
> Total: 00:30:12 43876 2605 0 0 6 0
> 1 94%
>
> Unfortunately the modem logs didn't show anything, because their log
> buffer (which was the default) was overwritten with the latest
> entries. I increased the modem log buffer (modem buffer-size 500), so
> i'm waiting for it to appear again.
>
>
> About the "on AS5300 better stick with <12.3(6)d" issue, i think if
> that was an option i would prefer to move back to 12.2(15)T since it
> has proven quite stable all these years.
>
> I just wanted to upgrade to >12.3.18 mainline in order to have a GD
> IOS in my AS5300s (and avoid some security vulnerabilities), but i
> guess this GD wasn't meant for all routers. Quite strange that some
> ED/LD releases are preferred over GD ones.
>
> --
> Tassos
>
> Aaron Leonard wrote on 26/5/2006 20:28:
>> Hi Tassos,
>>
>>>> Modem recovery can automatically busy out modems, if it should see
>>>> excessive failures on certain modems and decide that it wants to
>>>> "recover" the module.
>>>>
>>>
>>> I have neven seen on 12.2(15)T16 busied out calls (we're talking
>>> about 20 routers with this IOS), although i get messages like:
>>>
>>> May 25 04:00:00: %MODEM-1-MODEMOK: Modem (2/96) Running old firmware
>>> May 25 04:38:00: %MICA-5-MODEM_RECOVERY: Modem (2/96) is being
>>> recovered by asap-redownload
>>> May 25 04:38:06: %MODEM-5-DL_START: Modem (2/96) started firmware
>>> download
>>> May 25 04:38:19: %MODEM-5-DL_GOOD: Modem (2/96) completed firmware
>>> download:
>>>
>>> On the other hand, in 12.3.18 (only one router and in the same hunt
>>> group with the other 20), i never got such messages, but i had quite
>>> a lot of busied out calls.
>>
>> Doesn't the modem log show *anything* when the busied counter
>> increments for a modem?
>>
>>>> There are probably some other reasons why the router will decide
>>>> automatically to busy out modems, ... oh, I just though of one,
>>>> that little featurette where where we proactively busy out modems
>>>> to signal to the switch that the trunks are busy to force the
>>>> switch to hunt to a different router.
>>>>
>>>
>>> Can you explain it a little bit more?
>>> Why do you proactively busy out the modems in this case (signal to
>>> the switch), when the isdn channels are the ones that should be
>>> busied out?
>>
>> Um, you're right, it was my foggy memory that was faulty.
>>
>>>> Maybe if you look at the modem logs, they'll provide some
>>>> explanation, unless they have rolled over past the busyout event.
>>>>
>>>
>>> I just reloaded the box :( and loaded 12.3.19....i'm waiting for
>>> busied out calls to appear again.
>>> Then i'll try to grab the modem logs.
>>>
>>> --
>>> Tassos
>>
>> OK.
>>
>> I do have one concern that I should raise here, about using current
>> 12.3 mainline code on MICA platforms - (such as the 5300, or the
>> 3600/3700 with NM-DM modems) - see below. I think if I were running
>> such a platform, I would be tempted to run latest 12.3(6) throttle
>> (12.3(6f)) rather than a subsequent 12.3M release.
>>
>> Regards,
>>
>> Aaron
>>
>> ---
>>
>> CSCei63851
>> mica modems randomly marked as bad in any version afer 12.3(6)
>> Release-note: Modified 050919
>>
>> Click to see the enclosure with line wrapping enabled
>>
>>
>> In 3640 and 5300 platform running any version after 12.3(6), mica
>> modems may randomly
>> marked as BAD
>>
>> The symptoms observed only in situations where call success rate(CSR)
>> is very
>> low and there are good amount of calls failing continously in modem
>> to modem
>> trainup
>>
>> Modem log of modem in bad state, showed that IOS put modem out of
>> service
>> because modem failed to go onhook/offhook
>>
>> Workaround
>> -Use 12.3(6)d or any version before it
>>
>> -Manually download the modem portware to spe that has bad modem using
>> "firmware location.." command in spe mode
>>
>> -Reboot the router
>>
>> -Make sure many calls are not failing in trainup that is CSR is good
>>
>> ---
>>
>> This is marked Unreproducible, although it probably should be
>> reopened, as this problem is seen on 5300s and 36/3700s on all 12.3
>> mainline after the 12.3(6) throttle.
>>
>> ------------------------------------------------------------------------
>>
>>>
>>>> Aaron
>>>>
>>>> ------------------------------------------------------------------------
>>>>
>>>>
>>>>
>>>>> Router1 is AS5300 with 12.3(18.8) - 2.9.5.0
>>>>>
>>>>> router1>sh mode sum
>>>>> Avg Hold Incoming calls Outgoing calls Busied
>>>>> Failed No Succ
>>>>> Time Succ Fail Avail Succ Fail Avail Out
>>>>> Dial Ans Pct.
>>>>> 00:32:21 65667 4330 91 0 0 0 54
>>>>> 0 0 94%
>>>>>
>>>>> Router2 is AS5300 with 12.2(15)T16 - 2.9.5.0
>>>>>
>>>>> router2>sh modem sum
>>>>> Avg Hold Incoming calls Outgoing calls Busied
>>>>> Failed No Succ
>>>>> Time Succ Fail Avail Succ Fail Avail Out
>>>>> Dial Ans Pct.
>>>>> 00:31:52 435989 25314 76 0 0 0 0
>>>>> 0 4 95%
>>>>>
>>>>>
>>>>> Router1 is the only AS5300 running 12.3(18.8) and is the only one
>>>>> showing busied out calls during the last 2 months. Previously,
>>>>> when it was running 12.2(15)T16 too, no busied calls were there
>>>>> for over a year.
>>>>>
>>>>> So, is there a known bug with "busied out" calls in 12.3.18 IOS? I
>>>>> couldn't find anything in bugtool.
>>>>>
>>>>>
>>>>> --
>>>>> Tassos
>>>>> _______________________________________________
>>>>> cisco-nas mailing list
>>>>> cisco-nas@puck.nether.net
>>>>> https://puck.nether.net/mailman/listinfo/cisco-nas
>>>>>
>>>>
>>

_______________________________________________
cisco-nas mailing list
cisco-nas@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nas