Mailing List Archive

1 2  View All
Re: Cost of IPv6 for IT operations team [ In reply to ]
Hi Phil and list,

Phil Mayers <p.mayers@imperial.ac.uk> writes:

> On 13/04/15 09:55, Benedikt Stockebrand wrote:
>
>> Which is a major effort in some environments because---contrary to what
>> Nick wrote---pretty much anyone involved needs to be familiarized with
>> IPv6. The reason here is that if there is any problem once IPv6 is
>> enabled anywhere, then *all* people doing the troubleshooting need to
>> know how to exclude IPv6 from the list of possible problem causes.
>
> See, this makes sense - it's a reasonable assertion. But it's the
> opposite of what we've found.

and again, different environments *really* differ here:-)

> In all honesty, I doubt a large portion of the organisation
> understands how IPv*4* works. If you think about one of the many areas
> v6 differs -
> ARP/DHCP versus ND/SLAAC/DHCPv6 - how many people *really* understood
> the former that well?

That applies from a network operating point of view, but what about
making applications deal with multiple IP addresses, possibly mixed IPv4
and IPv6, the address selection algorithms, and how both affect the
getaddrinfo(3) library function?

And how about testing for IPv6 compliance in a given environment?
RIPE-554 and similar documents are good starting points, but if you
don't understand them and just throw them at your vendors as is, then
the only result you get is that you buy from the ones that lie to you
most convincingly.

>> [Blame games]
>
> OTOH, we often find the network blamed for basically everything. So
> this is only a small extension ;o)

With IPv6 this can and quite likely will get worse, except of course
that now it's "an IPv6 problem" rather than "a network problem".

For once however, I've long since made it a habit to get out of these
kinds of environments before I can get any reliably experience with this
issue:-)

>> While it's really convenient to use product refresh cycles to enable in
>> some cases, from what I've seen at least in enterprise environments this
>> frequently doesn't work as a general approach.
>
> What problems have you seen? Insufficiently demanding RFP or lack of
> awareness of the need to put it in an RFP? Or something else?

Not so much, but exceedingly long life cycles (think telephony setups in
arbitrary enterprises, 40+ year old Cobol software in banks, building
automation and so on) and---what's essential to really make this
ugly---environments where all these are so tightly interconnected that
it's pretty much impossible to update anything at all without bringing
the entire enterprise to a stop.

While I haven't personally worked with these, I've been involved with
environments where BS2000's are still strong. And apparently there's a
lively second hand market for PDP-11 components as well...

That's actually one of several reasons why ISPs aren't as badly hit by
IPv6 as a lot of enterprises: Their legacy stuff is orders of magnitude
less painful than what you find in many enterprises.


Cheers,

Benedikt

--
Benedikt Stockebrand, Stepladder IT Training+Consulting
Dipl.-Inform. http://www.stepladder-it.com/

Business Grade IPv6 --- Consulting, Training, Projects

BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/
Re: Cost of IPv6 for IT operations team [ In reply to ]
> On Mar 27, 2015, at 9:37 AM, Jens Link <lists@quux.de> wrote:
>
> Ted Mittelstaedt <tedm@ipinc.net> writes:
>
>> But as for operations costs, I would say, zero
>
> I don't agree. In a dual-stacked environment there is more work to do.
>
> 1. Setting up Servers (and Services)
>
> You have at least two addresses to configure, which leads to two DNS
> records, two services to be monitored, more firewall rules, ... And in
> the end more things to document.

This sure is true in an environment where humans are still the gating factor
in setting the machines up. Where some amount of automation or database driven
configuration exist (and i define database loosely) it will not be measurably
more. The initial investment in the tooling/scripts/magic is the cost.

I’m reminded of this graphic:

http://www.geeksaresexy.net/2012/01/05/geeks-vs-non-geeks-picture/

This is true for many people stuck in the cut+paste world of configuring
devices, be they hosts or routers.

> 2. Troubleshooting
>
> When looking for problems you always have to remember that there are now two
> protocols and then the fun begins: Is this problem only v4? Or only
> v6? Or do have a completely different problem.

I think there are some issues here due to the divergent functions of tools.

Take ping/ping6 as an example. One returns the IP being pinged, the other the
hostname.

puck:~$ ping puck
PING puck.nether.net (204.42.254.5) 56(84) bytes of data.
64 bytes from puck.nether.net (204.42.254.5): icmp_seq=1 ttl=64 time=0.012 ms
^C
--- puck.nether.net ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.012/0.012/0.012/0.000 ms
puck:~$ ping6 puck
PING puck(puck.nether.net) 56 data bytes
64 bytes from puck.nether.net: icmp_seq=1 ttl=64 time=0.022 ms
64 bytes from puck.nether.net: icmp_seq=2 ttl=64 time=0.030 ms
^C

This means understanding how the tool works. I find this is a skill often
missing, and the subjective reasoning. This is difficult to teach but is
unrelated to IPv6.

> 3. Layer 8+9
>
> People have little or no experience with IPv6. They need more time to
> configure stuff and troubleshoot problems. And they will forget to
> configure one thing or the other and they will forget that there are
> two protocols to troubleshoot.And then there will always be people
> who don't want IPv6 an will always blame it[1].

Yes, this is the whole fun I’ve seen occur. Some places, e.g.:
www.thruway.ny.gov -> cname www.thruway.ny.gov when I report their
problems say “we don’t support ipv6” when i say “asking for aaaa you
send back SERVFAIL”.

This was interesting as it triggered a bug in the resolver of my ISP as well
so they responded from the unicast address of the host vs anycast.


>> The reason is if you don't deploy sooner or later you will have a
>> problem related to IPv6.
>
> Ack.
>
>> Then you will spends lots of time finding and correcting. That time
>> is roughly equal to the extremely small amount of additional time that
>> the techs deal with IPv6 on a network that has had it properly setup.
>
> It will be worse. When you start implementing IPv6 because the latest
> version of your critical application requires IPv6 and you need IPv6
> tomorrow you'll have a real problem. You have to touch many things at
> once, you may have to buy hardware and you'll be looking for qualified
> external support. Problem is: Many other companies will do the same.
>
> I'll have to remember to buy a big stash of T-Shirts with "Told you so!"
>
> Jens
>
> [1] Someone told me a story about a database server which stopped
> working after IPv6 was deployed. The DB amdin blamed the IPv6
> deployment. Of course it hat nothing to do with IPv6, the hard drive was
> full.

And these people need to be called out for not doing their jobs properly.

- Jared
Re: Cost of IPv6 for IT operations team [ In reply to ]
Jared Mauch <jared@puck.nether.net> writes:

>> You have at least two addresses to configure, which leads to two DNS
>> records, two services to be monitored, more firewall rules, ... And in
>> the end more things to document.
>
> This sure is true in an environment where humans are still the gating factor
> in setting the machines up.

Sure. But unfortunately that's the majority of customers I work for.

> Where some amount of automation or database driven configuration exist
> (and i define database loosely) it will not be measurably more. The
> initial investment in the tooling/scripts/magic is the cost.

I helped implementing IPv6 for a big German website (well two but the
same network, the same servers, same admins, same tools, different
developers) back in 2012. The Admins new their infrastructure,
developers knew their code, almost everything was automated. And most
important: They worked together, exchanged knowledge. After three month
everything was ready and there was not much extra work that had to be
done. Most work was on the development side (numbers with three dots in
between is not a very good regexp for finding valid IPs and integer is
most likely to small too save IPv6 addresses to a database).

On the other hand I know many enterprises / universities / ISPs / ...
who don't automate things. Everything is done manually and any many
tasks are just copy'n'paste (and if that fails they have a problem
because they don't know what they are doing). One of them is using
2001:db8::/32 internally (well with a typo in the db8 part). They do
have IPv4 PI. They could easily implement IPv6 PI. They don't want
to. They got used to the problems they have.

In the last couple of month I worked on two projects where I really
liked to have used some config management system like puppet, saltstack,
chef, ... The admins at the customers had heard that such tools exists
but haven't had time to decide which one to use. And IPv6 was absolutely
no topic (I got "Well the data sheet for the server says that it supports
IPb6 that's enough" and "No we don't need IPV6" as reply when I asked
about it.)

>> 2. Troubleshooting
>>
>> When looking for problems you always have to remember that there are now two
>> protocols and then the fun begins: Is this problem only v4? Or only
>> v6? Or do have a completely different problem.
>
> I think there are some issues here due to the divergent functions of tools.

Not only that. Remembering that there are two different protocols would
be a first step. Not blaming IPv6 when you only see IPv4 traffic in a
wireshark trace the next.

Talking to each other is also important. One of my favorite examples:

http://blog.quux.de/?p=1542

> This means understanding how the tool works. I find this is a skill often
> missing, and the subjective reasoning. This is difficult to teach but is
> unrelated to IPv6.

Troubleshooting is an art. And many people lack the basic
knowledge. Don't get me started about DNS problems, ssh-keys, monitoring
(or the lack of monitoring), updates, ...

>> [1] Someone told me a story about a database server which stopped
>> working after IPv6 was deployed. The DB amdin blamed the IPv6
>> deployment. Of course it hat nothing to do with IPv6, the hard drive was
>> full.
>
> And these people need to be called out for not doing their jobs properly.

But they won't listen to us. They probably would listen to their boss
but he (or she) doesn't know any better.

Jens
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------
Re: Cost of IPv6 for IT operations team [ In reply to ]
Ted Mittelstaedt <tedm@ipinc.net> writes:

sorry for the delayed answer. I was quite busy doing non IPv6 stuff ;-)

> Kind of like you drafted Billy Joe Bob from Hayseed to troubleshoot your
> 2015 Chevy Volt and it extended the time troubleshooting because he's
> still looking for the carburetor.

This pretty much describes some (many?) of the admins I sometimes
encounter. When I have to explain to the so called network guru (of an
Cisco only network) how to configure SNMP on a Nexus switch or how to
configure DNS on Windows to some Windows experts (and people who know
will know that I don't like windows and try not to touch windows) how to
configure DNS on windows something is wrong.

> We live in a complex world that's getting more complex every day. If
> you want to be a tech, then deal with it, up your game.

Don't tell that to me. Tell it to the majority of so called admins
running all those systems out in there . And tell it to their bosses.

A friend told me about a presentation about Exchange 2012 (or whatever
the current version is): "Learn powershell or start looking for a new
job!" was one of the things the guy from M$ said. My guess: We'll see
many old Exchange installations in the next years.

> When your spending the same effort hitting the books as you are
> playing Warcraft then I'll have some sympathy. Otherwise go join the
> line of people sucking off the social services.

At least here in Germany they will be most probably trained to do IT. A
friend was teaching one of these courses and he asked me to talk about
Nagios and network monitroíng for an afternoon. There were 2 or 3 people
really interested. The rest was sleeping (which was fine) or
complained. I think the worst thing I've said was "learn enough english
to understand the documentation."

Jens
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------
Re: Cost of IPv6 for IT operations team [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Am 13.04.15 um 22:51 schrieb Jens Link:
> I think the worst thing I've said was "learn enough english to
> understand the documentation."

Das ist nicht Teil der tariflichen Arbeitsplatzbeschreibung.


Cheers,
Jens


PS: SCNR: "English is not part of the unions description of my job."

-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iEYEARECAAYFAlUsL5QACgkQFscsa5fGRGRicwCfSM9Hg44b2pd5+A9Tg5S0Ql3I
rFYAn0pQICSLF+loGIDW/b3OdHCn2Vef
=g0FA
-----END PGP SIGNATURE-----
Re: Cost of IPv6 for IT operations team [ In reply to ]
Andy Davidson <andy@nosignal.org> writes:

> Stage one - write v6 support requirements into RFP for equipment before
> you plan to turn it on.

That should have started a couple of years ago. About a year ago I did
some IPv6 training for a customer. They just bought a very expensive
commercial cloud solution (BULLSHIT! I win.) with no IPv6 support.

> Stage two - training, get a tunnel into the lab, make comfort with v6
> part of your technical appraisal for the technology teams, so that when
> it’s time to turn it on your team are familiar and will make fewer
> mistakes.

That would be fine. Especially as I'm getting payed for IPv6
trainings. But you first have to realize that you need IPv6, than you
need the people how want to do IPv6. And you need the management who is
willing to pay for training ("We hired you to to your job. You don't
need any more training. If you want to do it on your own time and with
your own money").

> Stage three - roll it out at your network edge, core & dns. Yes this
> is a project which needs time management and planning and incurs cost.

Thats eays if you didn't skip 1 and 2.

> Stage four - utilise your new training and v6 capable edge to roll out
> NEW services dual-stack.

Or v6 only.

> The incremental cost of adding v6 support to a NEW rollout when you
> have to do a bunch of work to roll out a service at all is therefore
> zero.

Sure. But people out there don't see it this way. They still plan and
build IPv4 only. About 3 or 4 years ago I helped with setting up a large
network which was planned IPv6 only 2 or 3 years earlier. "IPv4 should
only be tunneled" was one of the thing written in the original concept.
When I joined most of the admins couldn't spell IPv6 and everything was
IPv4 only. One of the internal colleagues attended the Cisco CCNÜ
routing course at that time. They skipped IPv6 completely. Trainer: "You
only need IPv6 to get 100% in the test." :-(

Jens
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------
Re: Cost of IPv6 for IT operations team [ In reply to ]
Jens Hoffmann <jh@bofh.de> writes:

> PS: SCNR: "English is not part of the unions description of my job."

They didn't have a job. They trained to get one.

Jens
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------
Re: Cost of IPv6 for IT operations team [ In reply to ]
Benedikt Stockebrand <bs@stepladder-it.com> writes:

> And yes, back in 95 there were allegedly people testing the hardware
> they bought for being year 2000 compliant:-)

Back in '99 my employer send me to an Oracle training. I do not remember
much about Oracle but I do remember one quote from the trainer:

"I told my first customer about Y2k in 1980. He called last week. If
anyone of you speaks Cobol the customer is paying $large_amount per
hour for fixing his problems."

With IPv6 there is no fixed date but I will happen too.


Jens
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------
Re: Cost of IPv6 for IT operations team [ In reply to ]
Hi,

On Mon, Apr 13, 2015 at 11:31:57PM +0200, Jens Link wrote:
> "I told my first customer about Y2k in 1980. He called last week. If
> anyone of you speaks Cobol the customer is paying $large_amount per
> hour for fixing his problems."
>
> With IPv6 there is no fixed date but I will happen too.

"customer called, and is offering $large_amount for anyone capable and
willing to fix their quad-nat-ipv4-only mission critical network"

Gert Doering
-- NetMaster
--
have you enabled IPv6 on something today...?

SpaceNet AG Vorstand: Sebastian v. Bomhard
Joseph-Dollinger-Bogen 14 Aufsichtsratsvors.: A. Grundner-Culemann
D-80807 Muenchen HRB: 136055 (AG Muenchen)
Tel: +49 (0)89/32356-444 USt-IdNr.: DE813185279
Re: Cost of IPv6 for IT operations team [ In reply to ]
On Mon, 13 Apr 2015, Jens Link wrote:

> Back in '99 my employer send me to an Oracle training. I do not remember
> much about Oracle but I do remember one quote from the trainer:
>
> "I told my first customer about Y2k in 1980. He called last week. If
> anyone of you speaks Cobol the customer is paying $large_amount per
> hour for fixing his problems."
>
> With IPv6 there is no fixed date but I will happen too.

Correct, there is no hard date, and the timing all depends on the
organization and their philosophy, internal politics, skill level, and
other factors.

Some will argue that the longer you wait, the cheaper it will get because
others will have already done the hard work, and you can benefit from that
work without paying. The executive might think that his bonus this year
will be higher because expenses weren't taken this year, and he probably
won't be around in 3-5 years anyway, so why should he care?

Some people believe things should be done that is purely customer driven,
others say success can be had by being proactive. They're probably partly
right both of them. In some cases, there is no right answer, for instance
as outlined in the process of going from mechanical to electronic
calculators around 1970
(http://www.slideshare.net/Christiansandstrom/calculators-disrupted-presentation-780029),
a lot of value was destroyed by this technological shift, making customers
a lot better off (cheaper and more reliable devices) but the makers were
just gutted.

We're most likely going to see a similar shift in the car industry in the
next 5-30 years. Imagine whole divisions within companies that have
expertise in gearboxes, ignition systems, combustion engines, turbos,
their jobs will just go away when you replace this with an electric motor,
a simpler fixed-ratio gearbox and batteries.

At least someone with IPv4 expertise can apply a lot of this for IPv6 when
the time comes for their company to adopt, and a person trained in IPv6 in
15 years, will most likely be able to handle IPv4 just fine, at least more
easily than having a newbie Java programmer try to fix Y2K problem in
Cobol.

So I still am hopeful and I keep hearing people discussing "when" IPv6
will be adopted, I don't really here "if" anymore.

--
Mikael Abrahamsson email: swmike@swm.pp.se
Re: Cost of IPv6 for IT operations team [ In reply to ]
> On 14 Apr 2015, at 08:42, Gert Doering <gert@space.net> wrote:
>
> Hi,
>
> On Mon, Apr 13, 2015 at 11:31:57PM +0200, Jens Link wrote:
>> "I told my first customer about Y2k in 1980. He called last week. If
>> anyone of you speaks Cobol the customer is paying $large_amount per
>> hour for fixing his problems."
>>
>> With IPv6 there is no fixed date but I will happen too.
>
> "customer called, and is offering $large_amount for anyone capable and
> willing to fix their quad-nat-ipv4-only mission critical network"

There’s a Dilbert cartoon waiting to happen there…

tim

1 2  View All