Mailing List Archive

Interesting, this has never been my experience even with Cisco or Juniper, have always been able to escalate quickly to engineering. I wonder if it was related to the size of my accounts.

Shane

> On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
>
> ?Thought about it but so far I believe companies from China provide better and fast TAC responses to their customers than the likes of Cisco and perhaps that’s why some companies(where there are no restrictions)prefer them for critical services.
>
> For a short period in TAC call you can have over 10 R&D engineers and solutions provided in a matter of hours even if it involves software changes.. while these other companies even before you get in a call with a TAC engineer it’s hours and when they join you hear something like “my shift ended 15 minutes ago, hold let me look for another engineer”. WHY? Thoughts

For us this has been the experience to a point where 100s of nodes( from
vendor x) had to be swapped out because no one had the patience anymore…

On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:

> Interesting, this has never been my experience even with Cisco or Juniper,
> have always been able to escalate quickly to engineering. I wonder if it
> was related to the size of my accounts.
>
> Shane
>
> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
> >
> > ?Thought about it but so far I believe companies from China provide
> better and fast TAC responses to their customers than the likes of Cisco
> and perhaps that’s why some companies(where there are no
> restrictions)prefer them for critical services.
> >
> > For a short period in TAC call you can have over 10 R&D engineers and
> solutions provided in a matter of hours even if it involves software
> changes.. while these other companies even before you get in a call with a
> TAC engineer it’s hours and when they join you hear something like “my
> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
> Thoughts
>

Funny you should mention this now, we were just discussing (more like
lamenting...) if support is a dying industry. It seems as though vendor
budgets are shrinking to the point they only have a Sales/Pre-Sales
department, and from Day Two on you are on your own. Dramatic take of
course, but if we are speaking in trajectories....




michael brooks
Sr. Network Engineer
Adams 12 Five Star Schools
michael.brooks@adams12.org
::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
"flying is learning how to throw yourself at the ground and miss"



On Wed, Mar 6, 2024 at 11:25?AM Pascal Masha <pascalmasha@gmail.com> wrote:

> Thought about it but so far I believe companies from China provide better
> and fast TAC responses to their customers than the likes of Cisco and
> perhaps that’s why some companies(where there are no restrictions)prefer
> them for critical services.
>
> For a short period in TAC call you can have over 10 R&D engineers and
> solutions provided in a matter of hours even if it involves software
> changes.. while these other companies even before you get in a call with a
> TAC engineer it’s hours and when they join you hear something like “my
> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
> Thoughts
>

--
This is a staff email account managed by Adams 12 Five Star Schools.  This
email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender.

On Wed, 6 Mar 2024 at 22:57, michael brooks - ESC
<michael.brooks@adams12.org> wrote:

> Funny you should mention this now, we were just discussing (more like lamenting...) if support is a dying industry. It seems as though vendor budgets are shrinking to the point they only have a Sales/Pre-Sales department, and from Day Two on you are on your own. Dramatic take of course, but if we are speaking in trajectories....

My personal experience extending in three different decades is that
there is no meaningful change in support quality or amount of issues
encountered.

Support quality has always been very modest, unless you specifically
pay to have access to named engineers. And this is not because quality
of the engineers changes, this is because vast majority of support
cases are useless cases, and to handle this massive volume support
tries to assume which support cases are legitimate problems, which are
PEBKAC and in which cases the user already solved their problem by the
time you read their ticket and will never respond back. The last case
is so common that every first-line adopts the strategy of 'pinging'
you, regardless how good and clear information you provide, they ask
some soft-ball question, to see if you're still engaged.
Having a named engineer changes this process, because the engineer
will quickly learn that you don't open useless cases, that the issue
you're having is legitimate, and will actually read the ticket and
think about the problem.

To me this seems an inevitable outcome, if your product is popular,
most of its users are users who don't do their homework and do not
respect the support line's time, which ends up being a disservice to
the whole ecosystem, because legitimate problems will take longer to
fix, or in case of open source software, authors just burn out and
kill the project.

What shocks me more than the low quality support is the low quality
software, decades pass along, and everyone still is having
show-stopper level of issues in basic functions on a regular basis,
the software quality is absolutely abysmal. I fear low software
quality is organically market-driven, no one is trying to make poor
NOS, it's just market incentives drive poor quality NOS. When no one
has high quality NOS, there is no reason to develop one, because most
of your revenue is support contracts, not hardware sales, and if the
NOS wouldn't be out-right broken needing to be recompiled regularly to
get basic things working, lot of users might stop buying support,
because they don't need the hand-holding part of it, they just need
working software.
This is not something that vendors actively drive, I'm sure most
companies believe they are making an honest attempt to improve
quality, but it is visible in where investments are put. One vendor
had a very promising project to take a holistic look into their NOS
quality issue, by senior subject matter experts, this project was
killed (I'm sure funding was needed somewhere with better returns),
and the responsible senior person went to Amazon instead.



>
>
>
>
> michael brooks
> Sr. Network Engineer
> Adams 12 Five Star Schools
> michael.brooks@adams12.org
> ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
> "flying is learning how to throw yourself at the ground and miss"
>
>
>
> On Wed, Mar 6, 2024 at 11:25?AM Pascal Masha <pascalmasha@gmail.com> wrote:
>>
>> Thought about it but so far I believe companies from China provide better and fast TAC responses to their customers than the likes of Cisco and perhaps that’s why some companies(where there are no restrictions)prefer them for critical services.
>>
>> For a short period in TAC call you can have over 10 R&D engineers and solutions provided in a matter of hours even if it involves software changes.. while these other companies even before you get in a call with a TAC engineer it’s hours and when they join you hear something like “my shift ended 15 minutes ago, hold let me look for another engineer”. WHY? Thoughts
>
>
> This is a staff email account managed by Adams 12 Five Star Schools. This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.



--
++ytti

> On 7 Mar 2024, at 06:50, Saku Ytti <saku@ytti.fi> wrote:
>
> ?The last case is so common that every first-line adopts the strategy of 'pinging' you, regardless how good and clear information you provide, they ask some soft-ball question, to see if you're still engaged.

No way - I never understood why HP and VMware support would call me 12/24 hours after I raised a case, essentially read it back to me and conclude with ‘okay assigning to an engineer now’. I last dealt with either of them 7 years ago but things haven’t likely changed.

This is it, to check if I’m still there! Quite absurd though to create a dependency on a phone call to start working on a support case for which I selected the email/portal contact method if you ask me though.

G

* I am biased, I’m from Arista * but having said that have you guys experienced Arista TAC? Not propaganda, I truly see it very differently.
As you guys said scale may change things down the road, but at the current scale it’s still an engineer that answers your call, straight away.

Sent from my iPhone

> On 7 Mar 2024, at 08:15, Giorgio Bonfiglio via NANOG <nanog@nanog.org> wrote:
>
> ?
>
>> On 7 Mar 2024, at 06:50, Saku Ytti <saku@ytti.fi> wrote:
>>
>> ?The last case is so common that every first-line adopts the strategy of 'pinging' you, regardless how good and clear information you provide, they ask some soft-ball question, to see if you're still engaged.
>
> No way - I never understood why HP and VMware support would call me 12/24 hours after I raised a case, essentially read it back to me and conclude with ‘okay assigning to an engineer now’. I last dealt with either of them 7 years ago but things haven’t likely changed.
>
> This is it, to check if I’m still there! Quite absurd though to create a dependency on a phone call to start working on a support case for which I selected the email/portal contact method if you ask me though.
>
> G

With all honesty, if you ask me, my experience with most companies from
China-in relation to Support- has always been fast and super satisfactory
no matter the raised case or sensitivity of the impact to users. I have
always felt comfortable running their gear and gives some sort of
confidence in not having prolonged outages no matter the reasons( engineer
inexperienced, not knowledgeable)

On Thu, 7 Mar 2024 at 09:49, Saku Ytti <saku@ytti.fi> wrote:

> On Wed, 6 Mar 2024 at 22:57, michael brooks - ESC
> <michael.brooks@adams12.org> wrote:
>
> > Funny you should mention this now, we were just discussing (more like
> lamenting...) if support is a dying industry. It seems as though vendor
> budgets are shrinking to the point they only have a Sales/Pre-Sales
> department, and from Day Two on you are on your own. Dramatic take of
> course, but if we are speaking in trajectories....
>
> My personal experience extending in three different decades is that
> there is no meaningful change in support quality or amount of issues
> encountered.
>
> Support quality has always been very modest, unless you specifically
> pay to have access to named engineers. And this is not because quality
> of the engineers changes, this is because vast majority of support
> cases are useless cases, and to handle this massive volume support
> tries to assume which support cases are legitimate problems, which are
> PEBKAC and in which cases the user already solved their problem by the
> time you read their ticket and will never respond back. The last case
> is so common that every first-line adopts the strategy of 'pinging'
> you, regardless how good and clear information you provide, they ask
> some soft-ball question, to see if you're still engaged.
> Having a named engineer changes this process, because the engineer
> will quickly learn that you don't open useless cases, that the issue
> you're having is legitimate, and will actually read the ticket and
> think about the problem.
>
> To me this seems an inevitable outcome, if your product is popular,
> most of its users are users who don't do their homework and do not
> respect the support line's time, which ends up being a disservice to
> the whole ecosystem, because legitimate problems will take longer to
> fix, or in case of open source software, authors just burn out and
> kill the project.
>
> What shocks me more than the low quality support is the low quality
> software, decades pass along, and everyone still is having
> show-stopper level of issues in basic functions on a regular basis,
> the software quality is absolutely abysmal. I fear low software
> quality is organically market-driven, no one is trying to make poor
> NOS, it's just market incentives drive poor quality NOS. When no one
> has high quality NOS, there is no reason to develop one, because most
> of your revenue is support contracts, not hardware sales, and if the
> NOS wouldn't be out-right broken needing to be recompiled regularly to
> get basic things working, lot of users might stop buying support,
> because they don't need the hand-holding part of it, they just need
> working software.
> This is not something that vendors actively drive, I'm sure most
> companies believe they are making an honest attempt to improve
> quality, but it is visible in where investments are put. One vendor
> had a very promising project to take a holistic look into their NOS
> quality issue, by senior subject matter experts, this project was
> killed (I'm sure funding was needed somewhere with better returns),
> and the responsible senior person went to Amazon instead.
>
>
>
> >
> >
> >
> >
> > michael brooks
> > Sr. Network Engineer
> > Adams 12 Five Star Schools
> > michael.brooks@adams12.org
> > ::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::
> > "flying is learning how to throw yourself at the ground and miss"
> >
> >
> >
> > On Wed, Mar 6, 2024 at 11:25?AM Pascal Masha <pascalmasha@gmail.com>
> wrote:
> >>
> >> Thought about it but so far I believe companies from China provide
> better and fast TAC responses to their customers than the likes of Cisco
> and perhaps that’s why some companies(where there are no
> restrictions)prefer them for critical services.
> >>
> >> For a short period in TAC call you can have over 10 R&D engineers and
> solutions provided in a matter of hours even if it involves software
> changes.. while these other companies even before you get in a call with a
> TAC engineer it’s hours and when they join you hear something like “my
> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
> Thoughts
> >
> >
> > This is a staff email account managed by Adams 12 Five Star Schools.
> This email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the sender.
>
>
>
> --
> ++ytti
>

----- On Mar 6, 2024, at 10:49 PM, Saku Ytti saku@ytti.fi wrote:

Hi,

> Support quality has always been very modest, unless you specifically
> pay to have access to named engineers. And this is not because quality
> of the engineers changes, this is because vast majority of support
> cases are useless cases, and to handle this massive volume support
> tries to assume which support cases are legitimate problems, which are
> PEBKAC and in which cases the user already solved their problem by the
> time you read their ticket and will never respond back. The last case
> is so common that every first-line adopts the strategy of 'pinging'
> you, regardless how good and clear information you provide, they ask
> some soft-ball question, to see if you're still engaged.
> Having a named engineer changes this process, because the engineer
> will quickly learn that you don't open useless cases, that the issue
> you're having is legitimate, and will actually read the ticket and
> think about the problem.

This. Absolutely this. I've been a TAC engineer at a major vendor for
a few years in the late 2000s. I found it interesting to observe that
the quality of cases is related to the size of the customer. In my
experience at that time, smaller customers tended to create low quality
cases but scream the loudest.

Following my experiences in TAC and hiring by several large networks, I
would give operations people guidance on how to actually open a TAC
case. More specifically, you know what the first questions will usually
be a canned response like "how long has this been happening, what is the
impact on production", etc. So, I've trained people to include that,
and all relevant logs that a TAC engineer can ask for, in the case to
begin with. And, of course, add a proper synopsis. "Router down" is
not.

Despite not having a named engineer, our cases were handled a lot
quicker all of a sudden.

Lastly, not every vendor has a first line group of juniors. Some
vendors you call will have the phone answered within 30 seconds by
an actual proper TAC engineer who will open the case for you if one
does not exist yet.

Thanks,

Sabri

It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.
— Sent from my iPhone

On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:

?For us this has been the experience to a point where 100s of nodes( from vendor x) had to be swapped out because no one had the patience anymore…
On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:

Interesting, this has never been my experience even with Cisco or Juniper, have always been able to escalate quickly to engineering. I wonder if it was related to the size of my accounts.

Shane

> On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
>
> ?Thought about it but so far I believe companies from China provide better and fast TAC responses to their customers than the likes of Cisco and perhaps that’s why some companies(where there are no restrictions)prefer them for critical services.
>
> For a short period in TAC call you can have over 10 R&D engineers and solutions provided in a matter of hours even if it involves software changes.. while these other companies even before you get in a call with a TAC engineer it’s hours and when they join you hear something like “my shift ended 15 minutes ago, hold let me look for another engineer”. WHY? Thoughts

That honestly is what my experience used to be but this has not been my observation recently, even when we as a large NSP provide all detail and literally ask about possible bugs.

From: NANOG <nanog-bounces+john=vanoppen.com@nanog.org> On Behalf Of Joel Esler
Sent: Thursday, March 7, 2024 11:46 AM
To: Pascal Masha <pascalmasha@gmail.com>
Cc: nanog <nanog@nanog.org>
Subject: Re: Best TAC Services from Equipment Vendors

It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.

>It may be a pain in the butt to get Cisco equipment, but their TAC is
sublime. If something is critical enough, and you push hard enough, Cisco
will move heaven and earth to solve your issue.

But is this not the problem itself?

Strap in for an "I remember when" ...
Once upon a time, I could call (specifically) Cisco TAC, knowing, without
any doubt, that my quality problem resolution was inbound, no questions
asked. That came to a crashing halt about 15 years ago when a TAC engineer
wanted to bounce our Voice VLAN SVI in the middle of an *airport* production
day. I about turned over my desk trying to wrest the remote control session
back from him before he hit enter on the shut. Since then, I have had to go
through a not insignificant evaluation period of TAC engineers before I let
them take control of a remote session, and it is now simply pure instinct
to log SSH sessions.

Fast forward and my user base is now ~45k people, not massive by any
stretch, but certainly not a small org, and we tend to buy Premium/Platinum
support on all of our critical applications. I truly do have to "push hard"
and long to get the attention of our support teams to get a seemingly
simple problem supported and solved. Our support is still in the dumper.
Take, for instance, a recent case with our replication/DR vendor. Our DR
environment has been offline for ~3 months, does that not scream
"critical?" And yet, we are still having engineers jump on a call to
collect .... the same data that the last engineer jumped on a call to
collect. One of our engineers, as has been not-so-subtly alluded to,
resorted to a screamfest to get the attention of TAC, and not surprisingly
that dressing-down got upper levels involved.

For good measure, major networking vendor possibly mentioned earlier spent
9 *months*, at the developer level, troubleshooting a multicast issue which
ultimately turned out to be PIM not configured on all interfaces in the
path. Yes, I felt like an idiot for not already checking that, but should
not have the first engineer on the phone asked this?

Admittedly, we are going through a rough patch in terms of support, but it
is not out of line with the past decade's experiences.


michael brooks

On Thu, Mar 7, 2024 at 12:47?PM Joel Esler <joel@joelesler.net> wrote:

> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime. If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
> —
> Sent from my iPhone
>
> On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:
>
> ?
> For us this has been the experience to a point where 100s of nodes( from
> vendor x) had to be swapped out because no one had the patience anymore…
>
> On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:
>
>> Interesting, this has never been my experience even with Cisco or
>> Juniper, have always been able to escalate quickly to engineering. I wonder
>> if it was related to the size of my accounts.
>>
>> Shane
>>
>> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
>> >
>> > ?Thought about it but so far I believe companies from China provide
>> better and fast TAC responses to their customers than the likes of Cisco
>> and perhaps that’s why some companies(where there are no
>> restrictions)prefer them for critical services.
>> >
>> > For a short period in TAC call you can have over 10 R&D engineers and
>> solutions provided in a matter of hours even if it involves software
>> changes.. while these other companies even before you get in a call with a
>> TAC engineer it’s hours and when they join you hear something like “my
>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>> Thoughts
>>
>

--
This is a staff email account managed by Adams 12 Five Star Schools.  This
email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender.

To be honest, if your DR environment has been offline for 3 months and you are just now opening a case, I would not consider that critical.
Shane

On Mar 11, 2024, at 10:08?AM, michael brooks - ESC <michael.brooks@adams12.org> wrote:

?
>It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.

But is this not the problem itself?
Strap in for an "I remember when" ...Once upon a time, I could call (specifically) Cisco TAC, knowing, without any doubt, that my quality problem resolution was inbound, no questions asked. That came to a crashing halt about 15 years ago when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of an airport production day. I about turned over my desk trying to wrest the remote control session back from him before he hit enter on the shut. Since then, I have had to go through a not insignificant evaluation period of TAC engineers before I let them take control of a remote session, and it is now simply pure instinct to log SSH sessions.
Fast forward and my user base is now ~45k people, not massive by any stretch, but certainly not a small org, and we tend to buy Premium/Platinum support on all of our critical applications. I truly do have to "push hard" and long to get the attention of our support teams to get a seemingly simple problem supported and solved. Our support is still in the dumper. Take, for instance, a recent case with our replication/DR vendor. Our DR environment has been offline for ~3 months, does that not scream "critical?" And yet, we are still having engineers jump on a call to collect .... the same data that the last engineer jumped on a call to collect. One of our engineers, as has been not-so-subtly alluded to, resorted to a screamfest to get the attention of TAC, and not surprisingly that dressing-down got upper levels involved.
For good measure, major networking vendor possibly mentioned earlier spent 9 months, at the developer level, troubleshooting a multicast issue which ultimately turned out to be PIM not configured on all interfaces in the path. Yes, I felt like an idiot for not already checking that, but should not have the first engineer on the phone asked this?
Admittedly, we are going through a rough patch in terms of support, but it is not out of line with the past decade's experiences.

michael brooks

On Thu, Mar 7, 2024 at 12:47?PM Joel Esler <joel@joelesler.net> wrote:

It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.
— Sent from my iPhone

On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:

?For us this has been the experience to a point where 100s of nodes( from vendor x) had to be swapped out because no one had the patience anymore…
On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:

Interesting, this has never been my experience even with Cisco or Juniper, have always been able to escalate quickly to engineering. I wonder if it was related to the size of my accounts.

Shane

> On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
>
> ?Thought about it but so far I believe companies from China provide better and fast TAC responses to their customers than the likes of Cisco and perhaps that’s why some companies(where there are no restrictions)prefer them for critical services.
>
> For a short period in TAC call you can have over 10 R&D engineers and solutions provided in a matter of hours even if it involves software changes.. while these other companies even before you get in a call with a TAC engineer it’s hours and when they join you hear something like “my shift ended 15 minutes ago, hold let me look for another engineer”. WHY? Thoughts

This is a staff email account managed by Adams 12 Five Star Schools. This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.

You are missing the point, we opened the case 3 months ago




michael brooks


On Mon, Mar 11, 2024 at 8:24?AM <sronan@ronan-online.com> wrote:

> To be honest, if your DR environment has been offline for 3 months and you
> are just now opening a case, I would not consider that critical.
>
> Shane
>
> On Mar 11, 2024, at 10:08?AM, michael brooks - ESC <
> michael.brooks@adams12.org> wrote:
>
> ?
>
> >It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime. If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
>
> But is this not the problem itself?
>
> Strap in for an "I remember when" ...
> Once upon a time, I could call (specifically) Cisco TAC, knowing, without
> any doubt, that my quality problem resolution was inbound, no questions
> asked. That came to a crashing halt about 15 years ago when a TAC engineer
> wanted to bounce our Voice VLAN SVI in the middle of an *airport* production
> day. I about turned over my desk trying to wrest the remote control session
> back from him before he hit enter on the shut. Since then, I have had to go
> through a not insignificant evaluation period of TAC engineers before I let
> them take control of a remote session, and it is now simply pure instinct
> to log SSH sessions.
>
> Fast forward and my user base is now ~45k people, not massive by any
> stretch, but certainly not a small org, and we tend to buy Premium/Platinum
> support on all of our critical applications. I truly do have to "push hard"
> and long to get the attention of our support teams to get a seemingly
> simple problem supported and solved. Our support is still in the dumper.
> Take, for instance, a recent case with our replication/DR vendor. Our DR
> environment has been offline for ~3 months, does that not scream
> "critical?" And yet, we are still having engineers jump on a call to
> collect .... the same data that the last engineer jumped on a call to
> collect. One of our engineers, as has been not-so-subtly alluded to,
> resorted to a screamfest to get the attention of TAC, and not surprisingly
> that dressing-down got upper levels involved.
>
> For good measure, major networking vendor possibly mentioned earlier spent
> 9 *months*, at the developer level, troubleshooting a multicast issue
> which ultimately turned out to be PIM not configured on all interfaces in
> the path. Yes, I felt like an idiot for not already checking that, but
> should not have the first engineer on the phone asked this?
>
> Admittedly, we are going through a rough patch in terms of support, but it
> is not out of line with the past decade's experiences.
>
>
> michael brooks
>
> On Thu, Mar 7, 2024 at 12:47?PM Joel Esler <joel@joelesler.net> wrote:
>
>> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime. If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>> —
>> Sent from my iPhone
>>
>> On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:
>>
>> ?
>> For us this has been the experience to a point where 100s of nodes( from
>> vendor x) had to be swapped out because no one had the patience anymore…
>>
>> On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:
>>
>>> Interesting, this has never been my experience even with Cisco or
>>> Juniper, have always been able to escalate quickly to engineering. I wonder
>>> if it was related to the size of my accounts.
>>>
>>> Shane
>>>
>>> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com>
>>> wrote:
>>> >
>>> > ?Thought about it but so far I believe companies from China provide
>>> better and fast TAC responses to their customers than the likes of Cisco
>>> and perhaps that’s why some companies(where there are no
>>> restrictions)prefer them for critical services.
>>> >
>>> > For a short period in TAC call you can have over 10 R&D engineers and
>>> solutions provided in a matter of hours even if it involves software
>>> changes.. while these other companies even before you get in a call with a
>>> TAC engineer it’s hours and when they join you hear something like “my
>>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>>> Thoughts
>>>
>>
> This is a staff email account managed by Adams 12 Five Star Schools. This
> email and any files transmitted with it are confidential and intended
> solely for the use of the individual or entity to whom they are addressed.
> If you have received this email in error please notify the sender.
>
>

--
This is a staff email account managed by Adams 12 Five Star Schools.  This
email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender.

>
> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime. If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
>

This was an amazing laugh on a Monday morning. Thanks!

On Thu, Mar 7, 2024 at 2:47?PM Joel Esler <joel@joelesler.net> wrote:

> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime. If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
> —
> Sent from my iPhone
>
> On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:
>
> ?
> For us this has been the experience to a point where 100s of nodes( from
> vendor x) had to be swapped out because no one had the patience anymore…
>
> On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:
>
>> Interesting, this has never been my experience even with Cisco or
>> Juniper, have always been able to escalate quickly to engineering. I wonder
>> if it was related to the size of my accounts.
>>
>> Shane
>>
>> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com> wrote:
>> >
>> > ?Thought about it but so far I believe companies from China provide
>> better and fast TAC responses to their customers than the likes of Cisco
>> and perhaps that’s why some companies(where there are no
>> restrictions)prefer them for critical services.
>> >
>> > For a short period in TAC call you can have over 10 R&D engineers and
>> solutions provided in a matter of hours even if it involves software
>> changes.. while these other companies even before you get in a call with a
>> TAC engineer it’s hours and when they join you hear something like “my
>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>> Thoughts
>>
>

Once upon a time, michael brooks - ESC <michael.brooks@adams12.org> said:
> Strap in for an "I remember when" ...

My Cisco TAC experiences (which were few) were not great... probably
around 2000 I opened a case with all the details, it was assigned, and
promptly closed "can't reproduce". I didn't have a real lab setup, but
I hauled a spare 7500 and 2924 to my cube, loaded the exact versions in
the ticket, set the config from the ticket, and easily reproduced.
Reopened the ticket and that time they could reproduce it (I believe
they didn't even try the first time) and eventually I got a fix.

The early days of Juniper were much better; I hit a bug and ended up on
a call with the actual developer. I did have to read an RFC section to
them, but they did then acknowledge the mistake and fix it. My SE sent
me an SSG for that one. :) Later Juniper could be hit or miss; I did
have a JTAC engineer go extra to help me with a work-around to an
obscure issue.

--
Chris Adams <cma@cmadams.net>

>
> It may be a pain in the butt to get Cisco equipment, but their TAC is
> sublime. If something is critical enough, and you push hard enough, Cisco
> will move heaven and earth to solve your issue.
>

>This was an amazing laugh on a Monday morning. Thanks!

O crap, did I miss the sarcasm?




michael brooks

On Mon, Mar 11, 2024 at 8:55?AM Tom Beecher <beecher@beecher.cc> wrote:

> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime. If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>>
>
> This was an amazing laugh on a Monday morning. Thanks!
>
> On Thu, Mar 7, 2024 at 2:47?PM Joel Esler <joel@joelesler.net> wrote:
>
>> It may be a pain in the butt to get Cisco equipment, but their TAC is
>> sublime. If something is critical enough, and you push hard enough, Cisco
>> will move heaven and earth to solve your issue.
>> —
>> Sent from my iPhone
>>
>> On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com> wrote:
>>
>> ?
>> For us this has been the experience to a point where 100s of nodes( from
>> vendor x) had to be swapped out because no one had the patience anymore…
>>
>> On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com> wrote:
>>
>>> Interesting, this has never been my experience even with Cisco or
>>> Juniper, have always been able to escalate quickly to engineering. I wonder
>>> if it was related to the size of my accounts.
>>>
>>> Shane
>>>
>>> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com>
>>> wrote:
>>> >
>>> > ?Thought about it but so far I believe companies from China provide
>>> better and fast TAC responses to their customers than the likes of Cisco
>>> and perhaps that’s why some companies(where there are no
>>> restrictions)prefer them for critical services.
>>> >
>>> > For a short period in TAC call you can have over 10 R&D engineers and
>>> solutions provided in a matter of hours even if it involves software
>>> changes.. while these other companies even before you get in a call with a
>>> TAC engineer it’s hours and when they join you hear something like “my
>>> shift ended 15 minutes ago, hold let me look for another engineer”. WHY?
>>> Thoughts
>>>
>>

--
This is a staff email account managed by Adams 12 Five Star Schools.  This
email and any files transmitted with it are confidential and intended
solely for the use of the individual or entity to whom they are addressed.
If you have received this email in error please notify the sender.

On 2024-03-07 05:08, Pedro Prado wrote:
> * I am biased, I’m from Arista * but having said that have you guys experienced Arista TAC?

Yes.

> As you guys said scale may change things down the road, but at the current scale it’s still an engineer that answers your call, straight away.

I think the scale has worsened this, slightly, already. It was great
initially.

Opening a ticket via web/email after hours seems to get clueless
first-level support.

Calling during U.S. business hours has been fine, but I'm generally only
doing that to get a ticket moving after it has stalled.

If I have an issue that needs to be escalated to
engineering/development, I've had very good to great experiences with
the engineers/developers in India after hours. It's a bad day if I need
them, but it's great to have them.

FWIW, I haven't tried calling after hours.

On the other hand, the sales engineers have been great!

--
Richard

You sir, not wrong.

> On Mar 11, 2024, at 10:04, michael brooks - ESC <michael.brooks@adams12.org> wrote:
>
>
> >It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.
>
> But is this not the problem itself?
>
> Strap in for an "I remember when" ...
> Once upon a time, I could call (specifically) Cisco TAC, knowing, without any doubt, that my quality problem resolution was inbound, no questions asked. That came to a crashing halt about 15 years ago when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of an airport production day. I about turned over my desk trying to wrest the remote control session back from him before he hit enter on the shut. Since then, I have had to go through a not insignificant evaluation period of TAC engineers before I let them take control of a remote session, and it is now simply pure instinct to log SSH sessions.
>
> Fast forward and my user base is now ~45k people, not massive by any stretch, but certainly not a small org, and we tend to buy Premium/Platinum support on all of our critical applications. I truly do have to "push hard" and long to get the attention of our support teams to get a seemingly simple problem supported and solved. Our support is still in the dumper. Take, for instance, a recent case with our replication/DR vendor. Our DR environment has been offline for ~3 months, does that not scream "critical?" And yet, we are still having engineers jump on a call to collect .... the same data that the last engineer jumped on a call to collect. One of our engineers, as has been not-so-subtly alluded to, resorted to a screamfest to get the attention of TAC, and not surprisingly that dressing-down got upper levels involved.
>
> For good measure, major networking vendor possibly mentioned earlier spent 9 months, at the developer level, troubleshooting a multicast issue which ultimately turned out to be PIM not configured on all interfaces in the path. Yes, I felt like an idiot for not already checking that, but should not have the first engineer on the phone asked this?
>
> Admittedly, we are going through a rough patch in terms of support, but it is not out of line with the past decade's experiences.
>
>
> michael brooks
>
> On Thu, Mar 7, 2024 at 12:47?PM Joel Esler <joel@joelesler.net <mailto:joel@joelesler.net>> wrote:
>> It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.
>> —
>> Sent from my iPhone
>>
>>> On Mar 6, 2024, at 13:42, Pascal Masha <pascalmasha@gmail.com <mailto:pascalmasha@gmail.com>> wrote:
>>>
>>> ?
>>> For us this has been the experience to a point where 100s of nodes( from vendor x) had to be swapped out because no one had the patience anymore…
>>>
>>> On Wed, 6 Mar 2024 at 21:29, <sronan@ronan-online.com <mailto:sronan@ronan-online.com>> wrote:
>>>> Interesting, this has never been my experience even with Cisco or Juniper, have always been able to escalate quickly to engineering. I wonder if it was related to the size of my accounts.
>>>>
>>>> Shane
>>>>
>>>> > On Mar 6, 2024, at 1:27?PM, Pascal Masha <pascalmasha@gmail.com <mailto:pascalmasha@gmail.com>> wrote:
>>>> >
>>>> > ?Thought about it but so far I believe companies from China provide better and fast TAC responses to their customers than the likes of Cisco and perhaps that’s why some companies(where there are no restrictions)prefer them for critical services.
>>>> >
>>>> > For a short period in TAC call you can have over 10 R&D engineers and solutions provided in a matter of hours even if it involves software changes.. while these other companies even before you get in a call with a TAC engineer it’s hours and when they join you hear something like “my shift ended 15 minutes ago, hold let me look for another engineer”. WHY? Thoughts
>
> This is a staff email account managed by Adams 12 Five Star Schools. This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the sender.

> On Mar 11, 2024, at 12:54, michael brooks - ESC <michael.brooks@adams12.org> wrote:
>
>> It may be a pain in the butt to get Cisco equipment, but their TAC is sublime. If something is critical enough, and you push hard enough, Cisco will move heaven and earth to solve your issue.
>
> >This was an amazing laugh on a Monday morning. Thanks!
>
> O crap, did I miss the sarcasm?
>

You did not. I think you have to know the magic incantation to get Cisco TAC to escalate to the magic level.

> when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of an *airport* production day.
> I about turned over my desk trying to wrest the remote control session back from him before he hit enter
> on the shut. Since then, I have had to go through a not insignificant evaluation period of TAC engineers
> before I let them take control of a remote session, and it is now simply pure instinct to log SSH sessions.

Picture it, Cisco TAC, on a troubleshooting call, runs 'no ip routing' and hits enter before our engineer could scream "NO" at 11:30AM on a core L3 on a college campus.

RCA afterwards:

1. "Always log all terminals (we prefer SecureCRT) from Windows bastion host to OneDrive or Google Drive"
2. New CiscoTAC TACACS login created allowing Enable but Denying "configure" as a command. When you troubleshoot, you log in as CiscoTAC.

The CiscoTAC tacacs profile description in Clearpass makes it clear why it's there. I left the curse words out.

-J

John C. Lyden
Associate Director, Network Operations
Division of Information Resources & Technology
Rowan University

We were one of the earlier adopters of Cisco ACI. Any issues with ACI were automatically escalated to an engineer that could fix almost anything. Now ACI tickets seem to go though a generic queue and the tech doesn't even know how to spell ACI.

We continue to have the same type of failure with Cisco DNA Center and TAC has to engage the business unit nearly every time to fix it. Sometimes it is like presenting a case to the supreme court to get the business unit to engage. They collect so much data that I wonder if it would be easier to ship the servers to them.

? ?? Curtis Parish
???? 615.494.8861
Senior Network Engineer




----IF CLASSIFICATION START----

----IF CLASSIFICATION END----

I've been reading the "${VENDOR}'s support has really gotten worse lately"
threads for pretty much every vendor for the past twenty years. That's not
to say they've all been wrong. But it reminds me of those quotes you'll see
about how "these kids today are awful and society is going to pot" and then
the big reveal is that it was written in the 1950s, or 1920s, or just
before the peak of Rome, or something like that. The general tendency for
people to view the past as the good ol' days.

My most memorable Cisco TAC disaster story. Taking away "configure" from
TAC wouldn't have saved us. The guy simply reloaded the switch without
asking. The core switch for a building with hundreds of end users. In the
middle of the day. The building with most of the C-level execs. Our
management was pi-i-i-issed. That got escalated pretty high, pretty
quickly. And quick policy change that we did not give TAC keyboard control.
This was about ten years ago.

On Tue, Mar 12, 2024 at 7:47?AM Lyden, John C <lyden@rowan.edu> wrote:

> > when a TAC engineer wanted to bounce our Voice VLAN SVI in the middle of
> an *airport* production day.
> > I about turned over my desk trying to wrest the remote control session
> back from him before he hit enter
> > on the shut. Since then, I have had to go through a not insignificant
> evaluation period of TAC engineers
> > before I let them take control of a remote session, and it is now simply
> pure instinct to log SSH sessions.
>
> Picture it, Cisco TAC, on a troubleshooting call, runs 'no ip routing' and
> hits enter before our engineer could scream "NO" at 11:30AM on a core L3 on
> a college campus.
>
> RCA afterwards:
>
> 1. "Always log all terminals (we prefer SecureCRT) from Windows bastion
> host to OneDrive or Google Drive"
> 2. New CiscoTAC TACACS login created allowing Enable but Denying
> "configure" as a command. When you troubleshoot, you log in as CiscoTAC.
>
> The CiscoTAC tacacs profile description in Clearpass makes it clear why
> it's there. I left the curse words out.
>
> -J
>
> John C. Lyden
> Associate Director, Network Operations
> Division of Information Resources & Technology
> Rowan University
>
>
>

In light of this thread's contents, I have to give a shout out to Nokia
TAC. Maybe because we buy a lotta stuff and have a lotta maint
contracts, but they don't do things like what has been mentioned. Of
course, I see some stuff from Level 1 folks where I think 'whaaat???'
but they haven't done anything like what I have heard on this thread in
the past 5 years I have been using them. Even for 'informational'
tickets they respond quickly.

scott

Richard Laager wrote:
> FWIW, I haven't tried calling after hours.
If I have to call after-hours, I get an answer from someone who is going
to be handling my case (assuming that it doesn't have an engineer who's
on-shift already assigned to it).  Yes, after-hours has traditionally
been when I talk to the folks who are still learning, but when hasn't
that been true?  Having said that, I always get the impression that
they're having a side-line conversation with a more senior engineer who
is helping them with any troubleshooting that they're not familiar with.

I admit, I don't use any particularly advanced features, so my
experience may not be typical, but I've never had an operational issue
that they haven't been able to solve fairly quickly.

Justin H.