Mailing List Archive: ASR920 and EEM:Mandatory.dualrate

ASR920 and EEM:Mandatory.dualrate_eem.tcl

Aug 26, 2019, 5:46 AM

Post #1 of 15 (3020 views)

Hi,

does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?

We have an ASR920 that grew an unexpected config change upon insertion
of a DAC cable into port ten0/0/12, and "unexpected config change" always
triggers an investigation here (who, why, what). One part of it was
somewhat related

interface TenGigabitEthernet0/0/12
description ...
no ip address
+ negotiation auto
service instance 200 ethernet

... but the other part was more interesting

line vty 0 4
access-class 9 in
- exec-timeout 240 0
ipv6 access-class VTY-v6 in
- transport input telnet ssh
+ transport preferred none
+ transport input none
+ transport output none
escape-character 3

"uh, what?". So we investigated and found a few log messages about that
script...

Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd: transceiver module inserted in TenGigabitEthernet0/0/12
<SNIP>
Aug 20 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE: TenGigabitEthernet0/0/12: MODE_1G
Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:Mandatory.dualrate_eem.tcl)
Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:Mandatory.dualrate_eem.tcl)
Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on vty0 (EEM:Mandatory.dualrate_eem.tcl)
Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd: Transceiver module removed from TenGigabitEthernet0/0/12
Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20 %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1
Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20 %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was granted to user <anon>; Trace file: , /harddisk/tracelogs/system_shell_R0-0.2264_0.20190820134619.bin
ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl: DUAL_RATE_CHANGE Re-configuration of interface TenGigabitEthernet0/0/12 to start re-configuring
Aug 20 13:46:28 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1 (EEM:Mandatory.dualrate_eem.tcl)
Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is Modified

... and 441 (!!) lines in the tacacs command accounting log, which
mostly looked like "it replayed the whole config, line by line"...
until it hit the vty section, which then got messed up...

Aug 20 13:47:08 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2166 timezone=CEST service=shell start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2167 timezone=CEST service=shell start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2168 timezone=CEST service=shell start_time=1566301629 priv-lvl=15 cmd=no login authentication <cr>
Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2169 timezone=CEST service=shell start_time=1566301629 priv-lvl=15 cmd=no authorization exec <cr>
Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2170 timezone=CEST service=shell start_time=1566301629 priv-lvl=15 cmd=no authorization commands 15 <cr>
Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2171 timezone=CEST service=shell start_time=1566301630 priv-lvl=15 cmd=no transport preferred <cr>
...
Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2174 timezone=CEST service=shell start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
Aug 20 13:47:11 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl stop task_id=2175 timezone=CEST service=shell start_time=1566301631 priv-lvl=1 cmd=no length <cr>
Aug 20 13:47:11 router unknown tty2 EEM:Mandatory.dualrate_eem.tcl stop task_id=2177 timezone=CEST service=shell start_time=1566301631 priv-lvl=15 cmd=write memory <cr>

shall I state that I find this a somewhat surprising behaviour?

Haven't opened a TAC case yet (no time) but hopefully someone here
has see this before and found some more useful results.

gert
--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

dudepron at gmail

Aug 26, 2019, 6:07 AM

Post #2 of 15 (3020 views)

Permalink

Any unexpected config change should be an automatic tac case.
Totally unexpected. Reminds me of the days when swapping a flash card on a
gsr could crash it.
This is a new one .

On Monday, August 26, 2019, Gert Doering <gert@greenie.muc.de> wrote:

> Hi,
>
> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>
> We have an ASR920 that grew an unexpected config change upon insertion
> of a DAC cable into port ten0/0/12, and "unexpected config change" always
> triggers an investigation here (who, why, what). One part of it was
> somewhat related
>
> interface TenGigabitEthernet0/0/12
> description ...
> no ip address
> + negotiation auto
> service instance 200 ethernet
>
> ... but the other part was more interesting
>
> line vty 0 4
> access-class 9 in
> - exec-timeout 240 0
> ipv6 access-class VTY-v6 in
> - transport input telnet ssh
> + transport preferred none
> + transport input none
> + transport output none
> escape-character 3
>
> "uh, what?". So we investigated and found a few log messages about that
> script...
>
> Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd: transceiver
> module inserted in TenGigabitEthernet0/0/12
> <SNIP>
> Aug 20 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE:
> TenGigabitEthernet0/0/12: MODE_1G
> Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
> (EEM:Mandatory.dualrate_eem.tcl)
> Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
> (EEM:Mandatory.dualrate_eem.tcl)
> Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on vty0
> (EEM:Mandatory.dualrate_eem.tcl)
> Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd: Transceiver
> module removed from TenGigabitEthernet0/0/12
> Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1
> Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was granted
> to user <anon>; Trace file: , /harddisk/tracelogs/system_
> shell_R0-0.2264_0.20190820134619.bin
> ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl:
> DUAL_RATE_CHANGE Re-configuration of interface TenGigabitEthernet0/0/12 to
> start re-configuring
> Aug 20 13:46:28 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
> (EEM:Mandatory.dualrate_eem.tcl)
> Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is Modified
>
>
> ... and 441 (!!) lines in the tacacs command accounting log, which
> mostly looked like "it replayed the whole config, line by line"...
> until it hit the vty section, which then got messed up...
>
> Aug 20 13:47:08 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2166 timezone=CEST service=shell
> start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2167 timezone=CEST service=shell
> start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2168 timezone=CEST service=shell
> start_time=1566301629 priv-lvl=15 cmd=no login authentication <cr>
> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2169 timezone=CEST service=shell
> start_time=1566301629 priv-lvl=15 cmd=no authorization exec <cr>
> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2170 timezone=CEST service=shell
> start_time=1566301629 priv-lvl=15 cmd=no authorization commands 15
> <cr>
> Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2171 timezone=CEST service=shell
> start_time=1566301630 priv-lvl=15 cmd=no transport preferred <cr>
> ...
> Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2174 timezone=CEST service=shell
> start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
> Aug 20 13:47:11 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2175 timezone=CEST service=shell
> start_time=1566301631 priv-lvl=1 cmd=no length <cr>
> Aug 20 13:47:11 router unknown tty2 EEM:Mandatory.dualrate_eem.tcl
> stop task_id=2177 timezone=CEST service=shell
> start_time=1566301631 priv-lvl=15 cmd=write memory <cr>
>
>
> shall I state that I find this a somewhat surprising behaviour?
>
> Haven't opened a TAC case yet (no time) but hopefully someone here
> has see this before and found some more useful results.
>
> gert
> --
> "If was one thing all people took for granted, was conviction that if you
> feed honest figures into a computer, honest figures come out. Never
> doubted
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh
> Mistress
>
> Gert Doering - Munich, Germany
> gert@greenie.muc.de
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

jared at puck

Aug 26, 2019, 6:09 AM

Post #3 of 15 (3020 views)

Permalink

I’ll say this in public (now) - Changing the security posture on the VTYs is a great reason to not use this product at the moment. I’ve seen many people not monitor their devices for these types of changes, and this is a great case to study.

Time for some retraining of people.

- Jared

> On Aug 26, 2019, at 9:07 AM, Aaron <dudepron@gmail.com> wrote:
>
> Any unexpected config change should be an automatic tac case.
> Totally unexpected. Reminds me of the days when swapping a flash card on a
> gsr could crash it.
> This is a new one .
>
> On Monday, August 26, 2019, Gert Doering <gert@greenie.muc.de> wrote:
>
>> Hi,
>>
>> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>>
>> We have an ASR920 that grew an unexpected config change upon insertion
>> of a DAC cable into port ten0/0/12, and "unexpected config change" always
>> triggers an investigation here (who, why, what). One part of it was
>> somewhat related
>>
>> interface TenGigabitEthernet0/0/12
>> description ...
>> no ip address
>> + negotiation auto
>> service instance 200 ethernet
>>
>> ... but the other part was more interesting
>>
>> line vty 0 4
>> access-class 9 in
>> - exec-timeout 240 0
>> ipv6 access-class VTY-v6 in
>> - transport input telnet ssh
>> + transport preferred none
>> + transport input none
>> + transport output none
>> escape-character 3
>>
>> "uh, what?". So we investigated and found a few log messages about that
>> script...
>>
>> Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd: transceiver
>> module inserted in TenGigabitEthernet0/0/12
>> <SNIP>
>> Aug 20 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE:
>> TenGigabitEthernet0/0/12: MODE_1G
>> Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
>> (EEM:Mandatory.dualrate_eem.tcl)
>> Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
>> (EEM:Mandatory.dualrate_eem.tcl)
>> Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on vty0
>> (EEM:Mandatory.dualrate_eem.tcl)
>> Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd: Transceiver
>> module removed from TenGigabitEthernet0/0/12
>> Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
>> %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1
>> Aug 20 13:46:20 CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
>> %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was granted
>> to user <anon>; Trace file: , /harddisk/tracelogs/system_
>> shell_R0-0.2264_0.20190820134619.bin
>> ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl:
>> DUAL_RATE_CHANGE Re-configuration of interface TenGigabitEthernet0/0/12 to
>> start re-configuring
>> Aug 20 13:46:28 CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
>> (EEM:Mandatory.dualrate_eem.tcl)
>> Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is Modified
>>
>>
>> ... and 441 (!!) lines in the tacacs command accounting log, which
>> mostly looked like "it replayed the whole config, line by line"...
>> until it hit the vty section, which then got messed up...
>>
>> Aug 20 13:47:08 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2166 timezone=CEST service=shell
>> start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
>> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2167 timezone=CEST service=shell
>> start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
>> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2168 timezone=CEST service=shell
>> start_time=1566301629 priv-lvl=15 cmd=no login authentication <cr>
>> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2169 timezone=CEST service=shell
>> start_time=1566301629 priv-lvl=15 cmd=no authorization exec <cr>
>> Aug 20 13:47:09 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2170 timezone=CEST service=shell
>> start_time=1566301629 priv-lvl=15 cmd=no authorization commands 15
>> <cr>
>> Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2171 timezone=CEST service=shell
>> start_time=1566301630 priv-lvl=15 cmd=no transport preferred <cr>
>> ...
>> Aug 20 13:47:10 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2174 timezone=CEST service=shell
>> start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
>> Aug 20 13:47:11 router unknown tty3 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2175 timezone=CEST service=shell
>> start_time=1566301631 priv-lvl=1 cmd=no length <cr>
>> Aug 20 13:47:11 router unknown tty2 EEM:Mandatory.dualrate_eem.tcl
>> stop task_id=2177 timezone=CEST service=shell
>> start_time=1566301631 priv-lvl=15 cmd=write memory <cr>
>>
>>
>> shall I state that I find this a somewhat surprising behaviour?
>>
>> Haven't opened a TAC case yet (no time) but hopefully someone here
>> has see this before and found some more useful results.
>>
>> gert
>> --
>> "If was one thing all people took for granted, was conviction that if you
>> feed honest figures into a computer, honest figures come out. Never
>> doubted
>> it myself till I met a computer with a sense of humor."
>> Robert A. Heinlein, The Moon is a Harsh
>> Mistress
>>
>> Gert Doering - Munich, Germany
>> gert@greenie.muc.de
>>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

b.turnbow at twt

Aug 26, 2019, 6:21 AM

Post #4 of 15 (3020 views)

Permalink

The dualrate script is for changing from 1G to 10G and vice versa.
So asr920 needs a vty access to run the script in telnet and since there is
not one available it removes ssh
Nice workaround!

More info here
https://www.cisco.com/c/en/us/td/docs/routers/asr920/b_Chassis_Guide_asr920/console-port.html

Brian

> -----Original Message-----
> From: cisco-nsp [mailto:cisco-nsp-bounces@puck.nether.net] On Behalf Of
> Jared Mauch
> Sent: lunedì 26 agosto 2019 15:10
> To: Aaron
> Cc: Gert Doering; cisco-nsp@puck.nether.net
> Subject: Re: [c-nsp] ASR920 and EEM:Mandatory.dualrate_eem.tcl
>
> I’ll say this in public (now) - Changing the security posture on the VTYs
> is a
> great reason to not use this product at the moment. I’ve seen many people
> not monitor their devices for these types of changes, and this is a great
> case
> to study.
>
> Time for some retraining of people.
>
> - Jared
>
> > On Aug 26, 2019, at 9:07 AM, Aaron <dudepron@gmail.com> wrote:
> >
> > Any unexpected config change should be an automatic tac case.
> > Totally unexpected. Reminds me of the days when swapping a flash card
> > on a gsr could crash it.
> > This is a new one .
> >
> > On Monday, August 26, 2019, Gert Doering <gert@greenie.muc.de> wrote:
> >
> >> Hi,
> >>
> >> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
> >>
> >> We have an ASR920 that grew an unexpected config change upon
> >> insertion of a DAC cable into port ten0/0/12, and "unexpected config
> >> change" always triggers an investigation here (who, why, what). One
> >> part of it was somewhat related
> >>
> >> interface TenGigabitEthernet0/0/12
> >> description ...
> >> no ip address
> >> + negotiation auto
> >> service instance 200 ethernet
> >>
> >> ... but the other part was more interesting
> >>
> >> line vty 0 4
> >> access-class 9 in
> >> - exec-timeout 240 0
> >> ipv6 access-class VTY-v6 in
> >> - transport input telnet ssh
> >> + transport preferred none
> >> + transport input none
> >> + transport output none
> >> escape-character 3
> >>
> >> "uh, what?". So we investigated and found a few log messages about
> >> that script...
> >>
> >> Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd:
> >> transceiver module inserted in TenGigabitEthernet0/0/12 <SNIP> Aug 20
> >> 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE:
> >> TenGigabitEthernet0/0/12: MODE_1G
> >> Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on
> >> vty1
> >> (EEM:Mandatory.dualrate_eem.tcl)
> >> Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on
> >> vty1
> >> (EEM:Mandatory.dualrate_eem.tcl)
> >> Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on
> >> vty0
> >> (EEM:Mandatory.dualrate_eem.tcl)
> >> Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd:
> Transceiver
> >> module removed from TenGigabitEthernet0/0/12 Aug 20 13:46:20 CEST:
> >> %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> >> %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1 Aug 20 13:46:20
> >> CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> >> %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was
> >> granted to user <anon>; Trace file: , /harddisk/tracelogs/system_
> >> shell_R0-0.2264_0.20190820134619.bin
> >> ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl:
> >> DUAL_RATE_CHANGE Re-configuration of interface
> >> TenGigabitEthernet0/0/12 to start re-configuring Aug 20 13:46:28
> >> CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
> >> (EEM:Mandatory.dualrate_eem.tcl)
> >> Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is
> >> Modified
> >>
> >>
> >> ... and 441 (!!) lines in the tacacs command accounting log, which
> >> mostly looked like "it replayed the whole config, line by line"...
> >> until it hit the vty section, which then got messed up...
> >>
> >> Aug 20 13:47:08 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2166 timezone=CEST service=shell
> >> start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
> >> Aug 20 13:47:09 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2167 timezone=CEST service=shell
> >> start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
> >> Aug 20 13:47:09 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2168 timezone=CEST service=shell
> >> start_time=1566301629 priv-lvl=15 cmd=no login authentication
> >> <cr>
> >> Aug 20 13:47:09 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2169 timezone=CEST service=shell
> >> start_time=1566301629 priv-lvl=15 cmd=no authorization exec <cr>
> >> Aug 20 13:47:09 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2170 timezone=CEST service=shell
> >> start_time=1566301629 priv-lvl=15 cmd=no authorization commands
> 15
> >> <cr>
> >> Aug 20 13:47:10 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2171 timezone=CEST service=shell
> >> start_time=1566301630 priv-lvl=15 cmd=no transport preferred
> >> <cr>
> >> ...
> >> Aug 20 13:47:10 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2174 timezone=CEST service=shell
> >> start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
> >> Aug 20 13:47:11 router unknown tty3
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2175 timezone=CEST service=shell
> >> start_time=1566301631 priv-lvl=1 cmd=no length <cr>
> >> Aug 20 13:47:11 router unknown tty2
> EEM:Mandatory.dualrate_eem.tcl
> >> stop task_id=2177 timezone=CEST service=shell
> >> start_time=1566301631 priv-lvl=15 cmd=write memory <cr>
> >>
> >>
> >> shall I state that I find this a somewhat surprising behaviour?
> >>
> >> Haven't opened a TAC case yet (no time) but hopefully someone here
> >> has see this before and found some more useful results.
> >>
> >> gert
> >> --
> >> "If was one thing all people took for granted, was conviction that if
> >> you feed honest figures into a computer, honest figures come out.
> >> Never doubted it myself till I met a computer with a sense of humor."
> >> Robert A. Heinlein, The Moon is a Harsh
> >> Mistress
> >>
> >> Gert Doering - Munich, Germany
> >> gert@greenie.muc.de
> >>
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

dudepron at gmail

Aug 26, 2019, 7:00 AM

Post #5 of 15 (3020 views)

Permalink

And to not reset the configuration back... How is that for security....

On Mon, Aug 26, 2019 at 9:21 AM Brian Turnbow <b.turnbow@twt.it> wrote:

> The dualrate script is for changing from 1G to 10G and vice versa.
> So asr920 needs a vty access to run the script in telnet and since there
> is
> not one available it removes ssh
> Nice workaround!
>
> More info here
>
> https://www.cisco.com/c/en/us/td/docs/routers/asr920/b_Chassis_Guide_asr920/console-port.html
>
>
>
>
> Brian
>
> > -----Original Message-----
> > From: cisco-nsp [mailto:cisco-nsp-bounces@puck.nether.net] On Behalf Of
> > Jared Mauch
> > Sent: lunedì 26 agosto 2019 15:10
> > To: Aaron
> > Cc: Gert Doering; cisco-nsp@puck.nether.net
> > Subject: Re: [c-nsp] ASR920 and EEM:Mandatory.dualrate_eem.tcl
> >
> > I’ll say this in public (now) - Changing the security posture on the
> VTYs
> > is a
> > great reason to not use this product at the moment. I’ve seen many
> people
> > not monitor their devices for these types of changes, and this is a
> great
> > case
> > to study.
> >
> > Time for some retraining of people.
> >
> > - Jared
> >
> > > On Aug 26, 2019, at 9:07 AM, Aaron <dudepron@gmail.com> wrote:
> > >
> > > Any unexpected config change should be an automatic tac case.
> > > Totally unexpected. Reminds me of the days when swapping a flash card
> > > on a gsr could crash it.
> > > This is a new one .
> > >
> > > On Monday, August 26, 2019, Gert Doering <gert@greenie.muc.de> wrote:
> > >
> > >> Hi,
> > >>
> > >> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
> > >>
> > >> We have an ASR920 that grew an unexpected config change upon
> > >> insertion of a DAC cable into port ten0/0/12, and "unexpected config
> > >> change" always triggers an investigation here (who, why, what). One
> > >> part of it was somewhat related
> > >>
> > >> interface TenGigabitEthernet0/0/12
> > >> description ...
> > >> no ip address
> > >> + negotiation auto
> > >> service instance 200 ethernet
> > >>
> > >> ... but the other part was more interesting
> > >>
> > >> line vty 0 4
> > >> access-class 9 in
> > >> - exec-timeout 240 0
> > >> ipv6 access-class VTY-v6 in
> > >> - transport input telnet ssh
> > >> + transport preferred none
> > >> + transport input none
> > >> + transport output none
> > >> escape-character 3
> > >>
> > >> "uh, what?". So we investigated and found a few log messages about
> > >> that script...
> > >>
> > >> Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd:
> > >> transceiver module inserted in TenGigabitEthernet0/0/12 <SNIP> Aug 20
> > >> 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE:
> > >> TenGigabitEthernet0/0/12: MODE_1G
> > >> Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on
> > >> vty1
> > >> (EEM:Mandatory.dualrate_eem.tcl)
> > >> Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on
> > >> vty1
> > >> (EEM:Mandatory.dualrate_eem.tcl)
> > >> Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on
> > >> vty0
> > >> (EEM:Mandatory.dualrate_eem.tcl)
> > >> Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd:
> > Transceiver
> > >> module removed from TenGigabitEthernet0/0/12 Aug 20 13:46:20 CEST:
> > >> %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> > >> %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1 Aug 20 13:46:20
> > >> CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
> > >> %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was
> > >> granted to user <anon>; Trace file: , /harddisk/tracelogs/system_
> > >> shell_R0-0.2264_0.20190820134619.bin
> > >> ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl:
> > >> DUAL_RATE_CHANGE Re-configuration of interface
> > >> TenGigabitEthernet0/0/12 to start re-configuring Aug 20 13:46:28
> > >> CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
> > >> (EEM:Mandatory.dualrate_eem.tcl)
> > >> Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is
> > >> Modified
> > >>
> > >>
> > >> ... and 441 (!!) lines in the tacacs command accounting log, which
> > >> mostly looked like "it replayed the whole config, line by line"...
> > >> until it hit the vty section, which then got messed up...
> > >>
> > >> Aug 20 13:47:08 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2166 timezone=CEST service=shell
> > >> start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
> > >> Aug 20 13:47:09 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2167 timezone=CEST service=shell
> > >> start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
> > >> Aug 20 13:47:09 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2168 timezone=CEST service=shell
> > >> start_time=1566301629 priv-lvl=15 cmd=no login authentication
> > >> <cr>
> > >> Aug 20 13:47:09 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2169 timezone=CEST service=shell
> > >> start_time=1566301629 priv-lvl=15 cmd=no authorization exec
> <cr>
> > >> Aug 20 13:47:09 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2170 timezone=CEST service=shell
> > >> start_time=1566301629 priv-lvl=15 cmd=no authorization commands
> > 15
> > >> <cr>
> > >> Aug 20 13:47:10 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2171 timezone=CEST service=shell
> > >> start_time=1566301630 priv-lvl=15 cmd=no transport preferred
> > >> <cr>
> > >> ...
> > >> Aug 20 13:47:10 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2174 timezone=CEST service=shell
> > >> start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
> > >> Aug 20 13:47:11 router unknown tty3
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2175 timezone=CEST service=shell
> > >> start_time=1566301631 priv-lvl=1 cmd=no length <cr>
> > >> Aug 20 13:47:11 router unknown tty2
> > EEM:Mandatory.dualrate_eem.tcl
> > >> stop task_id=2177 timezone=CEST service=shell
> > >> start_time=1566301631 priv-lvl=15 cmd=write memory <cr>
> > >>
> > >>
> > >> shall I state that I find this a somewhat surprising behaviour?
> > >>
> > >> Haven't opened a TAC case yet (no time) but hopefully someone here
> > >> has see this before and found some more useful results.
> > >>
> > >> gert
> > >> --
> > >> "If was one thing all people took for granted, was conviction that if
> > >> you feed honest figures into a computer, honest figures come out.
> > >> Never doubted it myself till I met a computer with a sense of humor."
> > >> Robert A. Heinlein, The Moon is a Harsh
> > >> Mistress
> > >>
> > >> Gert Doering - Munich, Germany
> > >> gert@greenie.muc.de
> > >>
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp@puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp@puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

adrian.minta at gmail

Aug 26, 2019, 8:21 AM

Post #6 of 15 (3020 views)

Permalink

If the TCL script fails, the RX part of the 10G transceiver still works
and in some circumstances this could lead to a STP loop.

On 8/26/19 5:00 PM, Aaron wrote:
> And to not reset the configuration back... How is that for security....
>
> On Mon, Aug 26, 2019 at 9:21 AM Brian Turnbow <b.turnbow@twt.it> wrote:
>
>> The dualrate script is for changing from 1G to 10G and vice versa.
>> So asr920 needs a vty access to run the script in telnet and since there
>> is
>> not one available it removes ssh
>> Nice workaround!
>>
>> More info here
>>
>> https://www.cisco.com/c/en/us/td/docs/routers/asr920/b_Chassis_Guide_asr920/console-port.html
>>
>>
>>
>>
>> Brian
>>
>>> -----Original Message-----
>>> From: cisco-nsp [mailto:cisco-nsp-bounces@puck.nether.net] On Behalf Of
>>> Jared Mauch
>>> Sent: lunedì 26 agosto 2019 15:10
>>> To: Aaron
>>> Cc: Gert Doering; cisco-nsp@puck.nether.net
>>> Subject: Re: [c-nsp] ASR920 and EEM:Mandatory.dualrate_eem.tcl
>>>
>>> I’ll say this in public (now) - Changing the security posture on the
>> VTYs
>>> is a
>>> great reason to not use this product at the moment. I’ve seen many
>> people
>>> not monitor their devices for these types of changes, and this is a
>> great
>>> case
>>> to study.
>>>
>>> Time for some retraining of people.
>>>
>>> - Jared
>>>
>>>> On Aug 26, 2019, at 9:07 AM, Aaron <dudepron@gmail.com> wrote:
>>>>
>>>> Any unexpected config change should be an automatic tac case.
>>>> Totally unexpected. Reminds me of the days when swapping a flash card
>>>> on a gsr could crash it.
>>>> This is a new one .
>>>>
>>>> On Monday, August 26, 2019, Gert Doering <gert@greenie.muc.de> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>>>>>
>>>>> We have an ASR920 that grew an unexpected config change upon
>>>>> insertion of a DAC cable into port ten0/0/12, and "unexpected config
>>>>> change" always triggers an investigation here (who, why, what). One
>>>>> part of it was somewhat related
>>>>>
>>>>> interface TenGigabitEthernet0/0/12
>>>>> description ...
>>>>> no ip address
>>>>> + negotiation auto
>>>>> service instance 200 ethernet
>>>>>
>>>>> ... but the other part was more interesting
>>>>>
>>>>> line vty 0 4
>>>>> access-class 9 in
>>>>> - exec-timeout 240 0
>>>>> ipv6 access-class VTY-v6 in
>>>>> - transport input telnet ssh
>>>>> + transport preferred none
>>>>> + transport input none
>>>>> + transport output none
>>>>> escape-character 3
>>>>>
>>>>> "uh, what?". So we investigated and found a few log messages about
>>>>> that script...
>>>>>
>>>>> Aug 20 13:45:30 CEST: %TRANSCEIVER-6-INSERTED: F0: iomd:
>>>>> transceiver module inserted in TenGigabitEthernet0/0/12 <SNIP> Aug 20
>>>>> 13:45:45 CEST: %IOSXE_SPA-6-DUAL_RATE_CHANGE:
>>>>> TenGigabitEthernet0/0/12: MODE_1G
>>>>> Aug 20 13:45:47 CEST: %SYS-5-CONFIG_I: Configured from console by on
>>>>> vty1
>>>>> (EEM:Mandatory.dualrate_eem.tcl)
>>>>> Aug 20 13:46:14 CEST: %SYS-5-CONFIG_I: Configured from console by on
>>>>> vty1
>>>>> (EEM:Mandatory.dualrate_eem.tcl)
>>>>> Aug 20 13:46:15 CEST: %SYS-5-CONFIG_I: Configured from console by on
>>>>> vty0
>>>>> (EEM:Mandatory.dualrate_eem.tcl)
>>>>> Aug 20 13:46:17 CEST: %TRANSCEIVER-6-REMOVED: F0: iomd:
>>> Transceiver
>>>>> module removed from TenGigabitEthernet0/0/12 Aug 20 13:46:20 CEST:
>>>>> %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
>>>>> %SYSTEM-3-SYSTEM_SHELL_LOG: Shell started: vty 1 Aug 20 13:46:20
>>>>> CEST: %IOSXE-5-PLATFORM: F0: Aug 20 13:46:20
>>>>> %SYSTEM-3-SYSTEM_SHELL_LOG: 2019/08/20 13:46:19 : Shell access was
>>>>> granted to user <anon>; Trace file: , /harddisk/tracelogs/system_
>>>>> shell_R0-0.2264_0.20190820134619.bin
>>>>> ug 20 13:46:26 CEST: %HA_EM-6-LOG: Mandatory.dualrate_eem.tcl:
>>>>> DUAL_RATE_CHANGE Re-configuration of interface
>>>>> TenGigabitEthernet0/0/12 to start re-configuring Aug 20 13:46:28
>>>>> CEST: %SYS-5-CONFIG_I: Configured from console by on vty1
>>>>> (EEM:Mandatory.dualrate_eem.tcl)
>>>>> Aug 20 13:46:39 CEST: %SYS-5-CONFIG_C: Running-config file is
>>>>> Modified
>>>>>
>>>>>
>>>>> ... and 441 (!!) lines in the tacacs command accounting log, which
>>>>> mostly looked like "it replayed the whole config, line by line"...
>>>>> until it hit the vty section, which then got messed up...
>>>>>
>>>>> Aug 20 13:47:08 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2166 timezone=CEST service=shell
>>>>> start_time=1566301628 priv-lvl=15 cmd=configure terminal <cr>
>>>>> Aug 20 13:47:09 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2167 timezone=CEST service=shell
>>>>> start_time=1566301629 priv-lvl=15 cmd=line vty 0 4 <cr>
>>>>> Aug 20 13:47:09 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2168 timezone=CEST service=shell
>>>>> start_time=1566301629 priv-lvl=15 cmd=no login authentication
>>>>> <cr>
>>>>> Aug 20 13:47:09 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2169 timezone=CEST service=shell
>>>>> start_time=1566301629 priv-lvl=15 cmd=no authorization exec
>> <cr>
>>>>> Aug 20 13:47:09 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2170 timezone=CEST service=shell
>>>>> start_time=1566301629 priv-lvl=15 cmd=no authorization commands
>>> 15
>>>>> <cr>
>>>>> Aug 20 13:47:10 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2171 timezone=CEST service=shell
>>>>> start_time=1566301630 priv-lvl=15 cmd=no transport preferred
>>>>> <cr>
>>>>> ...
>>>>> Aug 20 13:47:10 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2174 timezone=CEST service=shell
>>>>> start_time=1566301630 priv-lvl=15 cmd=no exec-timeout <cr>
>>>>> Aug 20 13:47:11 router unknown tty3
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2175 timezone=CEST service=shell
>>>>> start_time=1566301631 priv-lvl=1 cmd=no length <cr>
>>>>> Aug 20 13:47:11 router unknown tty2
>>> EEM:Mandatory.dualrate_eem.tcl
>>>>> stop task_id=2177 timezone=CEST service=shell
>>>>> start_time=1566301631 priv-lvl=15 cmd=write memory <cr>
>>>>>
>>>>>
>>>>> shall I state that I find this a somewhat surprising behaviour?
>>>>>
>>>>> Haven't opened a TAC case yet (no time) but hopefully someone here
>>>>> has see this before and found some more useful results.
>>>>>
>>>>> gert
>>>>> --
>>>>> "If was one thing all people took for granted, was conviction that if
>>>>> you feed honest figures into a computer, honest figures come out.
>>>>> Never doubted it myself till I met a computer with a sense of humor."
>>>>> Robert A. Heinlein, The Moon is a Harsh
>>>>> Mistress
>>>>>
>>>>> Gert Doering - Munich, Germany
>>>>> gert@greenie.muc.de
>>>>>
>>>> _______________________________________________
>>>> cisco-nsp mailing list cisco-nsp@puck.nether.net
>>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>> _______________________________________________
>>> cisco-nsp mailing list cisco-nsp@puck.nether.net
>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/

--
Best regards,
Adrian Minta

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

lists at ltri

Aug 26, 2019, 9:28 AM

Post #7 of 15 (3020 views)

Permalink

Hello Gert,

On Mon, 26 Aug 2019 at 14:47, Gert Doering <gert@greenie.muc.de> wrote:
>
> Hi,
>
> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>
> We have an ASR920 that grew an unexpected config change upon insertion
> of a DAC cable into port ten0/0/12, and "unexpected config change" always
> triggers an investigation here (who, why, what). One part of it was
> somewhat related
>
> interface TenGigabitEthernet0/0/12
> description ...
> no ip address
> + negotiation auto
> service instance 200 ethernet

It seems to me dual-rate ports on the ASR920 are a bit of an
afterthought, which is why it needs integrated duct-tape EEM scripts
to work around code bugs. Because those EEM scripts are as fragile as
you would expect from your average "expect" style CLI scraping tool,
you get things like this:

CSCvb61075 Dual-rate EEM script expires: when the hostname has a dot
('.') character in it
CSCuz01963 "negotiation auto" changes to "no negotiation auto" under
the Tengig interface.
CSCvf97552 BDI ping failure will happen and dual rate EEM policy wont succeed
CSCuy55664 Dual-rate port SFP optics swap leads to egress forwarding issues
CSCvm21736 "negotiation auto" config gets removed from dual rate ports
post node Hard reset/Power cycle
CSCux89058 Negotiation auto - negated after OIR of 1G-SFP on Tengig
port in ASR920
CSCvf36147 'negotiation auto' command is not saved in startup configs
after reload in Striker-C
CSCur65014 ASR920:"No negotiation auto" config is not retained on reload
CSCuy55658 ASR920: Dual-rate port SFP optics swap leads to egress
forwarding issues

But my personal favorite is CSCvm57265:

> Status: Open
> Severity: 4 Minor
>
> Symptom:
> Higher numbered loopback configuration is changed and put into the MGMT VRF automatically when SPF of dual rate interfaces of ASR920 is changed from SFP (1GE) to SFP+ (10GE) or viceversa
>
> Conditions:
> ASR920 running version asr920igp-universalk9.03.18.04.SP.156-2.SP4-ext.
> Dual rate interface up with SFP or SFP+ inserted.
>
> SFP is changed either from SFP to SFP+ or SFP+ to SFP.
>
> If there are several loopbacks configured on the device, the higher numbered one will be changed and automatically put into the MGMT VRF erasing the rest of the configuration. If only one loopback is configured, then that loopback will be changed automatically.
>
> Workaround:
> If all loopbacks are used for communication or signalling, configure a temporal loopback with a higher number that the rest. Only the higher loopback will get changed:
>
> Loopbacl10
> Loopback20
> Loopback40 --- This one will be changed
>
> During the SFP change this temporal loopback will be changed but this will not impact any service since this is only a dummy interface that can be erased after the activity is complete.

You better put that Loopback40 in your templates ;)

cheers,
lukas
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

nick at foobar

Aug 26, 2019, 9:40 AM

Post #8 of 15 (3020 views)

Permalink

Lukas Tribus wrote on 26/08/2019 17:28:
> You better put that Loopback40 in your templates ;)

But wait! It's only Severity: 4 Minor. No need to be alarmed that this
might accidentally kill your entire network.

Nick

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

andrew at vianet

Aug 26, 2019, 11:51 AM

Post #9 of 15 (3018 views)

Permalink

Bug CSCvm57265 has happened to me. The ASR920 in question had its
OSPF/MPLS Router-ID on Lo0 (only loopback configured) and the TCL script
changed it into the MGMT VRF (from the GRT) when inserting a 10G optic
for the first time after enabling the license. Needless to say it took
the site down.

It only happen once, TAC told me they could not figure out the issue
unless I reproduce it.

On 8/26/2019 12:28 PM, Lukas Tribus wrote:
> Hello Gert,
>
> On Mon, 26 Aug 2019 at 14:47, Gert Doering <gert@greenie.muc.de> wrote:
>> Hi,
>>
>> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>>
>> We have an ASR920 that grew an unexpected config change upon insertion
>> of a DAC cable into port ten0/0/12, and "unexpected config change" always
>> triggers an investigation here (who, why, what). One part of it was
>> somewhat related
>>
>> interface TenGigabitEthernet0/0/12
>> description ...
>> no ip address
>> + negotiation auto
>> service instance 200 ethernet
> It seems to me dual-rate ports on the ASR920 are a bit of an
> afterthought, which is why it needs integrated duct-tape EEM scripts
> to work around code bugs. Because those EEM scripts are as fragile as
> you would expect from your average "expect" style CLI scraping tool,
> you get things like this:
>
> CSCvb61075 Dual-rate EEM script expires: when the hostname has a dot
> ('.') character in it
> CSCuz01963 "negotiation auto" changes to "no negotiation auto" under
> the Tengig interface.
> CSCvf97552 BDI ping failure will happen and dual rate EEM policy wont succeed
> CSCuy55664 Dual-rate port SFP optics swap leads to egress forwarding issues
> CSCvm21736 "negotiation auto" config gets removed from dual rate ports
> post node Hard reset/Power cycle
> CSCux89058 Negotiation auto - negated after OIR of 1G-SFP on Tengig
> port in ASR920
> CSCvf36147 'negotiation auto' command is not saved in startup configs
> after reload in Striker-C
> CSCur65014 ASR920:"No negotiation auto" config is not retained on reload
> CSCuy55658 ASR920: Dual-rate port SFP optics swap leads to egress
> forwarding issues
>
>
> But my personal favorite is CSCvm57265:
>
>> Status: Open
>> Severity: 4 Minor
>>
>> Symptom:
>> Higher numbered loopback configuration is changed and put into the MGMT VRF automatically when SPF of dual rate interfaces of ASR920 is changed from SFP (1GE) to SFP+ (10GE) or viceversa
>>
>> Conditions:
>> ASR920 running version asr920igp-universalk9.03.18.04.SP.156-2.SP4-ext.
>> Dual rate interface up with SFP or SFP+ inserted.
>>
>> SFP is changed either from SFP to SFP+ or SFP+ to SFP.
>>
>> If there are several loopbacks configured on the device, the higher numbered one will be changed and automatically put into the MGMT VRF erasing the rest of the configuration. If only one loopback is configured, then that loopback will be changed automatically.
>>
>> Workaround:
>> If all loopbacks are used for communication or signalling, configure a temporal loopback with a higher number that the rest. Only the higher loopback will get changed:
>>
>> Loopbacl10
>> Loopback20
>> Loopback40 --- This one will be changed
>>
>> During the SFP change this temporal loopback will be changed but this will not impact any service since this is only a dummy interface that can be erased after the activity is complete.
>
> You better put that Loopback40 in your templates ;)
>
>
> cheers,
> lukas
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

gert at greenie

Aug 26, 2019, 1:24 PM

Post #10 of 15 (3012 views)

Permalink

Hi,

On Mon, Aug 26, 2019 at 02:46:19PM +0200, Gert Doering wrote:
> does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?

So, now I know - thanks to all who answered.

I'm not sure I wanted to know in the first place, and now I do not
know if I'm scared or morbidly fascinated.

The hacker in me ("let's see if we can make this work, somehow") is
morbidly fascinated by having a mechanism in the box that is able
to change config (loopback, vty) to telnet(!?) into the box, put
files into bootflash, mount a tmpfs from there, then do other things,
but is *not* able to "just do the right thing directly".

Also, what is wrong with the good old "these ports are not automatic
dual-speed ports, so if you insert an 1G transceiver, you get a log
message that you tells you to configure 'speed 1G' to make it work"?

The operator in me starts to understand how "you have MPLS configured
on one port" can lead to "all 1G ports stop forwarding" now (we had one
box where the 1G part just died, and TAC refused to swap it because
we had MPLS configured on the 10G links, so "IT MUST BE THAT BUG!").

And I do not want to be close to these boxes anymore. What is in there
scares me.

So, if I need EoMPLS/VPLS capability (12-24 1G ports, 2-4 10G ports,
plain point-to-point pseudowires, or pseudowires with a local switching
instance - VPLS or EVPN), what other vendors build such a box and do
not hide madness inside? Conceptually similar to the ASR920, that is,
no full table, small footprint / fairly low power consumption, etc.

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

laurentfdumont at gmail

Aug 26, 2019, 1:33 PM

Post #11 of 15 (3012 views)

Permalink

At $old_job we had issues with the 920 failing to switch between the 1G and
10G shaping internally after the SFP swap. It would report that everything
went well and negotiate to 10G to only shape the actual line rate at 1G.

Fun times.

On Mon, Aug 26, 2019, 4:25 PM Gert Doering <gert@greenie.muc.de> wrote:

> Hi,
>
> On Mon, Aug 26, 2019 at 02:46:19PM +0200, Gert Doering wrote:
> > does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>
> So, now I know - thanks to all who answered.
>
> I'm not sure I wanted to know in the first place, and now I do not
> know if I'm scared or morbidly fascinated.
>
> The hacker in me ("let's see if we can make this work, somehow") is
> morbidly fascinated by having a mechanism in the box that is able
> to change config (loopback, vty) to telnet(!?) into the box, put
> files into bootflash, mount a tmpfs from there, then do other things,
> but is *not* able to "just do the right thing directly".
>
> Also, what is wrong with the good old "these ports are not automatic
> dual-speed ports, so if you insert an 1G transceiver, you get a log
> message that you tells you to configure 'speed 1G' to make it work"?
>
>
> The operator in me starts to understand how "you have MPLS configured
> on one port" can lead to "all 1G ports stop forwarding" now (we had one
> box where the 1G part just died, and TAC refused to swap it because
> we had MPLS configured on the 10G links, so "IT MUST BE THAT BUG!").
>
> And I do not want to be close to these boxes anymore. What is in there
> scares me.
>
>
> So, if I need EoMPLS/VPLS capability (12-24 1G ports, 2-4 10G ports,
> plain point-to-point pseudowires, or pseudowires with a local switching
> instance - VPLS or EVPN), what other vendors build such a box and do
> not hide madness inside? Conceptually similar to the ASR920, that is,
> no full table, small footprint / fairly low power consumption, etc.
>
> gert
>
> --
> "If was one thing all people took for granted, was conviction that if you
> feed honest figures into a computer, honest figures come out. Never
> doubted
> it myself till I met a computer with a sense of humor."
> Robert A. Heinlein, The Moon is a Harsh
> Mistress
>
> Gert Doering - Munich, Germany
> gert@greenie.muc.de
> _______________________________________________
> cisco-nsp mailing list cisco-nsp@puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

cisco-nsp at radu-adrian

Aug 26, 2019, 3:14 PM

Post #12 of 15 (3011 views)

Permalink

On Mon, Aug 26, 2019, at 22:25, Gert Doering wrote:
> So, if I need EoMPLS/VPLS capability (12-24 1G ports, 2-4 10G ports,
> plain point-to-point pseudowires, or pseudowires with a local switching
> instance - VPLS or EVPN), what other vendors build such a box and do
> not hide madness inside? Conceptually similar to the ASR920, that is,
> no full table, small footprint / fairly low power consumption, etc.

Don't know for VPLS/EVPN(*), but for simple VPWS and quite basic L3VPN the model with 24x1G + 4x10G (fixed rate) was pretty stable at the time I was using them. Maybe also because I quit before any box could reach the "xxx days uptime bug" (that could be prevented with a simple scheduled maintenance) :P

(*) Actually EVPN on A920, I tried once (because it was supposed to be the thing that does not need neither LDP nor RSVP) and the conclusion at the time (~2years ago) was to wait until they ship something that does work.

--
R.-A. Feurdean
_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

adamv0025 at netconsultings

Aug 27, 2019, 6:15 AM

Post #13 of 15 (2998 views)

Permalink

> -----Original Message-----
> Gert Doering
> Sent: Monday, August 26, 2019 9:25 PM
> Hi,
>
> On Mon, Aug 26, 2019 at 02:46:19PM +0200, Gert Doering wrote:
> > does anyone know what "EEM:Mandatory.dualrate_eem.tcl" is?
>
> So, now I know - thanks to all who answered.
>
> I'm not sure I wanted to know in the first place, and now I do not know if
I'm
> scared or morbidly fascinated.
>
Isn't it possible to disable/delete all these EEM scripts?

adam

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

b.turnbow at twt

Aug 27, 2019, 7:29 AM

Post #14 of 15 (2998 views)

Permalink

Hi,

> > I'm not sure I wanted to know in the first place, and now I do not
> > know if
> I'm
> > scared or morbidly fascinated.
> >
> Isn't it possible to disable/delete all these EEM scripts?
>

It is a registered policy even on ASR-920-24SZ-M with no dual rate
ports....

ASR920_JN1#sh event manager policy registered
No. Class Type Event Type Trap Time Registered
Name
1 script system syslog Off Mon Jul 24 09:47:42 2017
Mandatory.dualrate_eem_policy.tcl
occurs 1 pattern {IOSXE_SPA-6-DUAL_RATE_CHANGE:*}
nice 0 queue-priority normal maxrun 240.000 scheduler rp_primary Secu
none

and at least can be removed on this platform
ASR920_JN1(config)#no event manager policy
Mandatory.dualrate_eem_policy.tcl

ASR920_JN1#sh event manager policy registered
No. Class Type Event Type Trap Time Registered
Name
ASR920_JN1

Not sure what happens post upgrade ....

And to see the script
show event manager policy available detailed
Mandatory.dualrate_eem_policy.tcl

Brian

_______________________________________________
cisco-nsp mailing list cisco-nsp@puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

Re: ASR920 and EEM:Mandatory.dualrate_eem.tcl [ In reply to ]

gert at greenie

Aug 27, 2019, 11:40 AM

Post #15 of 15 (2996 views)

Permalink

Hi,

On Tue, Aug 27, 2019 at 04:29:14PM +0200, Brian Turnbow wrote:
> And to see the script
> show event manager policy available detailed
> Mandatory.dualrate_eem_policy.tcl

Interesting. In my version (16.06.05) it's called slightly different,
but your command to see it works...

#show event manager policy available detailed Mandatory.dualrate_eem.tcl
::cisco::eem::event_register_syslog occurs 1 pattern "IOSXE_SPA-6-DUAL_RATE_CHANGE:*" maxrun 240
...
#EEM Tcl script to detect SFP to SFP+ change and vice-versa on dual-rate port then remove config, default interface and then re-apply config
...
set clist [.list "config terminal" "line vty 0 4" "login authentication Mandatory.dualrate_eem.tcl" "authorization exec Mandatory.dualrate_eem.tcl" "authorization commands 15 Mandatory.dualrate_eem.tcl" "transport preferred none" "transport input none" "transport output none" "exec-timeout 10 0" "length 0" "exit"]
...
# AAA Bypass unconfigurations
...

Highly amazing.

gert

--
"If was one thing all people took for granted, was conviction that if you
feed honest figures into a computer, honest figures come out. Never doubted
it myself till I met a computer with a sense of humor."
Robert A. Heinlein, The Moon is a Harsh Mistress

Gert Doering - Munich, Germany gert@greenie.muc.de

Mailing List Archive

Attached Files:

Attached Files:

Attached Files: