Mailing List Archive

Hi,

Deployment work overload could be 5%.

For deployment of dual stack, you should minimise work overload by:
- taking advantage of normal refresh cycles and bundle enabling of dual
stack into the refresh or deployment of new management systems, procedures,
and services
- picking "low hanging fruit" tasks that will advance deployment with
minimum effort
(some legacy systems may never be capable of or required to be accessed
over IPv6 - leave them).
- don't provide some IPv6 network or service that doesn't have similar
performance and reliability as IPv4 has; wait until you can
- taking things slowly to allow time for staff to mentally adjust to
dual-stack as being normal,
and to keep IPv6 issues from being on the critical path

For run phase, I think it really depends upon what type of IT issues you
normally see:
- something used to work over network and now it doesn't == need to check
IPv6 as well as IPv4.
- database management, or PC malware == no IPv6 involvement
- IPv4 address space micromanagement == easy as IPv6 /64 subnet is never
too small

Run-time recommendations:
- make sure setup *and* changes to firewall / ACL rules permit/deny
equivalently for IPv4 and IPv6
- promptly investigate any network problems that could be blamed on IPv6 to
avoid users advising each other
"just turn off IPv6 to fix things", and then never turning it back on
again.

Thanks,
John









On 26 March 2015 at 20:04, BERENGUER Christophe <
Christophe.BERENGUER@solucom.fr> wrote:

> Hello everybody,
>
>
> I work for a consulting firm.
>
>
> For a client, I would like to estimate the work overload for IT
> operations team to deploy IPv6 dual stack and for day to day operations.
>
>
> On the internet, I have found an estimation around 20% of work overload
> for the run phase. But if you have operational feedback it would be the
> best!
>
>
> Thanks in advance for your answers,
>
> Have a nice day.
>
>
> Best regards,
>
>
> *Christophe BERENGUER*
> Consultant
> Fixe : +33 (0)1 49 03 85 86
> christophe.berenguer@solucom.fr
> solucom
> Tour Franklin : 100 - 101 terrasse Boieldieu
> 92042 Paris La Défense Cedex
>

This is a really good list, thanks for sharing. I just gave a talk on this
topic this week, this is timely. For operational reality, the
implementation phase is highly dependent on the environment. For an
environment that has full control of systems to peerings, the overhead will
be less than an environment that has less control on end points or has a
fractured internal structure.

The point of the protocols being ships in the night is very important and
is often forgotten, keeping that in mind is key as is minimizing the method
of users disabling IPv6 to fix issues. I often recommend someone on the
deployment team setting up an IPv6 only station and using it to see what
does and does not function, both internally and externally. A VM works well
for this but I find that an actual workstation makes me actually use it
more. This does a few things, but most importantly it makes it obvious what
doesn't work on v6, which will be at least 50% of your initial roll out
troubleshooting.

Keep good metrics and monitoring on *both* IPv4 and IPv6, this helps both
up front and over time. Anything that is measured and monitored in v4
should also by measured and monitored for v6 in my opinion. Make sure your
security policy is as close to v4 as you can reasonably get it (but be
mindful of ICMP) .his includes all packet captures, filters, ACLs and
security appliance provided services.

nb






---
Nick Buraglio
ESnet Network Engineering Group (AS293)
Lawrence Berkeley National Laboratory
buraglio@es.net
+1 (510) 995-6068

On Thu, Mar 26, 2015 at 6:33 AM, John Mann <john.mann@monash.edu> wrote:

> Hi,
>
> Deployment work overload could be 5%.
>
> For deployment of dual stack, you should minimise work overload by:
> - taking advantage of normal refresh cycles and bundle enabling of dual
> stack into the refresh or deployment of new management systems, procedures,
> and services
> - picking "low hanging fruit" tasks that will advance deployment with
> minimum effort
> (some legacy systems may never be capable of or required to be accessed
> over IPv6 - leave them).
> - don't provide some IPv6 network or service that doesn't have similar
> performance and reliability as IPv4 has; wait until you can
> - taking things slowly to allow time for staff to mentally adjust to
> dual-stack as being normal,
> and to keep IPv6 issues from being on the critical path
>
> For run phase, I think it really depends upon what type of IT issues you
> normally see:
> - something used to work over network and now it doesn't == need to check
> IPv6 as well as IPv4.
> - database management, or PC malware == no IPv6 involvement
> - IPv4 address space micromanagement == easy as IPv6 /64 subnet is never
> too small
>
> Run-time recommendations:
> - make sure setup *and* changes to firewall / ACL rules permit/deny
> equivalently for IPv4 and IPv6
> - promptly investigate any network problems that could be blamed on IPv6
> to avoid users advising each other
> "just turn off IPv6 to fix things", and then never turning it back on
> again.
>
> Thanks,
> John
>
>
>
>
>
>
>
>
>
> On 26 March 2015 at 20:04, BERENGUER Christophe <
> Christophe.BERENGUER@solucom.fr> wrote:
>
>> Hello everybody,
>>
>>
>> I work for a consulting firm.
>>
>>
>> For a client, I would like to estimate the work overload for IT
>> operations team to deploy IPv6 dual stack and for day to day operations.
>>
>>
>> On the internet, I have found an estimation around 20% of work overload
>> for the run phase. But if you have operational feedback it would be the
>> best!
>>
>>
>> Thanks in advance for your answers,
>>
>> Have a nice day.
>>
>>
>> Best regards,
>>
>>
>> *Christophe BERENGUER*
>> Consultant
>> Fixe : +33 (0)1 49 03 85 86
>> christophe.berenguer@solucom.fr
>> solucom
>> Tour Franklin : 100 - 101 terrasse Boieldieu
>> 92042 Paris La Défense Cedex
>>
>
>

On 26/03/2015 22:04, BERENGUER Christophe wrote:
> Hello everybody,
>
>
> I work for a consulting firm.
>
>
> For a client, I would like to estimate the work overload for IT operations team to deploy IPv6 dual stack and for day to day operations.
>
>
> On the internet, I have found an estimation around 20% of work overload for the run phase.

Is that evidence-based, or a hand-waving guess?
I would expect a bit of extra workload at the beginning of the run phase
but in the steady state are there really 20% more incidents?

Brian

> But if you have operational feedback it would be the best!
>
>
> Thanks in advance for your answers,
>
> Have a nice day.
>
>
> Best regards,
>
>
> Christophe BERENGUER
> Consultant
> Fixe : +33 (0)1 49 03 85 86
> christophe.berenguer@solucom.fr<mailto:christophe.berenguer@solucom.fr>
> solucom
> Tour Franklin : 100 - 101 terrasse Boieldieu
> 92042 Paris La Défense Cedex
>

> Op 27 mrt. 2015, om 00:23 heeft Brian E Carpenter <brian.e.carpenter@gmail.com> het volgende geschreven:
>
> On 26/03/2015 22:04, BERENGUER Christophe wrote:
>> Hello everybody,
>>
>>
>> I work for a consulting firm.
>>
>>
>> For a client, I would like to estimate the work overload for IT operations team to deploy IPv6 dual stack and for day to day operations.
>>
>>
>> On the internet, I have found an estimation around 20% of work overload for the run phase.
>
> Is that evidence-based, or a hand-waving guess?
> I would expect a bit of extra workload at the beginning of the run phase
> but in the steady state are there really 20% more incidents?

We use pfSense at work and I’m using hostnames and other DNS names in the firewall rules to great lengths so that they automatically adjust when a host changes IPs, be that 4 or 6. I can select IPv4 and IPv6 in the rule so the same rule applies to both.

Ofcourse, there is a security tradeoff, but considering the sheer amount of CDN hosting today it’s becoming harder to just assign a IP to the rule and have it work for over a week :)

Firewalling by (prefixes from) ASN would be something useful to have too, for abuse purposes.

I’m mostly talking about outbound firewall rules, the LAN is pretty much closed off. Proxy or bust.

Cheers,
Seth

>
> Brian
>
>> But if you have operational feedback it would be the best!
>>
>>
>> Thanks in advance for your answers,
>>
>> Have a nice day.
>>
>>
>> Best regards,
>>
>>
>> Christophe BERENGUER
>> Consultant
>> Fixe : +33 (0)1 49 03 85 86
>> christophe.berenguer@solucom.fr<mailto:christophe.berenguer@solucom.fr>
>> solucom
>> Tour Franklin : 100 - 101 terrasse Boieldieu
>> 92042 Paris La Défense Cedex
>>
>

Without a detailed look at the client this kind of question falls in the
realm of my kid's story problems in her mathematics book - pretty
sounding things that are utterly divorced from reality.

I will just say this, however:

If you do NOT deploy IPv6 then yes it will save labor. Depending on how
disorganized the clients network is, that could be a lot of time or a
little.

But as for operations costs, I would say, zero

The reason is if you don't deploy sooner or later you will have a
problem related to IPv6. Then you will spends lots of time finding and
correcting. That time is roughly equal to the extremely small amount
of additional time that the techs deal with IPv6 on a network that has
had it properly setup.

Ted

On 3/26/2015 2:04 AM, BERENGUER Christophe wrote:
> Hello everybody,
>
>
> I work for a consulting firm.
>
>
> For a client, I would like to estimate the work overload for IT
> operations team to deploy IPv6 dual stack and for day to day operations.
>
>
> On the internet, I have found an estimation around 20% of work overload
> for the run phase. But if you have operational feedback it would be the
> best!
>
>
> Thanks in advance for your answers,
>
> Have a nice day.
>
>
> Best regards,
>
>
> *Christophe BERENGUER**
> *Consultant
> Fixe : +33 (0)1 49 03 85 86
> christophe.berenguer@solucom.fr <mailto:christophe.berenguer@solucom.fr>
> solucom
> Tour Franklin : 100 - 101 terrasse Boieldieu
> 92042 Paris La Défense Cedex

Thanks everyone for your answers.

We already have raise the point that at some point IPv6 will need to be deploy and the sooner is the better to have time to fix problems and let time for teams to master the technology.

The 20% figure is based on an IETF document and SRI presentation. I think it is not limited to the amount of incident but also the time to duplicate firewall rules, configure all the interfaces, etc. for both v4 and v6.

For the build phase, I agree that it depends on my clients architecture. It has a good overview of it but the impact on application is not clear and hard to estimate.

Thanks again for your answers!

--
Christophe BERENGUER
Consultant
Fixe : +33 (0)1 49 03 85 86
christophe.berenguer@solucom.fr
solucom
Tour Franklin : 100 - 101 terrasse Boieldieu
92042 Paris La Défense Cedex

________________________________________
De : ipv6-ops-bounces+christophe.berenguer=solucom.fr@lists.cluenet.de <ipv6-ops-bounces+christophe.berenguer=solucom.fr@lists.cluenet.de> de la part de Ted Mittelstaedt <tedm@ipinc.net>
Envoyé : vendredi 27 mars 2015 09:27
À : ipv6-ops@lists.cluenet.de
Objet : Re: Cost of IPv6 for IT operations team

Without a detailed look at the client this kind of question falls in the
realm of my kid's story problems in her mathematics book - pretty
sounding things that are utterly divorced from reality.

I will just say this, however:

If you do NOT deploy IPv6 then yes it will save labor. Depending on how
disorganized the clients network is, that could be a lot of time or a
little.

But as for operations costs, I would say, zero

The reason is if you don't deploy sooner or later you will have a
problem related to IPv6. Then you will spends lots of time finding and
correcting. That time is roughly equal to the extremely small amount
of additional time that the techs deal with IPv6 on a network that has
had it properly setup.

Ted

On 3/26/2015 2:04 AM, BERENGUER Christophe wrote:
> Hello everybody,
>
>
> I work for a consulting firm.
>
>
> For a client, I would like to estimate the work overload for IT
> operations team to deploy IPv6 dual stack and for day to day operations.
>
>
> On the internet, I have found an estimation around 20% of work overload
> for the run phase. But if you have operational feedback it would be the
> best!
>
>
> Thanks in advance for your answers,
>
> Have a nice day.
>
>
> Best regards,
>
>
> *Christophe BERENGUER**
> *Consultant
> Fixe : +33 (0)1 49 03 85 86
> christophe.berenguer@solucom.fr <mailto:christophe.berenguer@solucom.fr>
> solucom
> Tour Franklin : 100 - 101 terrasse Boieldieu
> 92042 Paris La Défense Cedex

On 26/03/15 09:04, BERENGUER Christophe wrote:
> Hello everybody,
>
>
> I work for a consulting firm.
>
>
> For a client, I would like to estimate the work overload for IT
> operations team to deploy IPv6 dual stack and for day to day operations.
>
>
> On the internet, I have found an estimation around 20% of work overload
> for the run phase. But if you have operational feedback it would be the
> best!

I agree with others that this is a very hard thing to estimate.

I will say that we run our dual-stack network (fully deployed since ca.
2012) with exactly the same staffing levels, and actually a slight
reduction in our recurrent budget, as our older IPv4-only network.

I don't think our network is any less reliable, or suffers a higher
level of incidents. This suggests to me that, in our case, IPv6 has
added a very low operational cost. Our incidence of IPv6-related
problems, particularly rogue RA from machines configured for connection
sharing, has actually *decreased* substantially since we deployed native
IPv6.

I don't believe the rollout cost was high. We used refresh cycles to
upgrade to v6-capable gear, and rolled out slowly to grow our team
knowledge. But we don't have detailed cost breakdowns.

> The 20% figure is based on an IETF document

Which one? We can fix that if people think it's wrong.
(This comes just too late for yesterday's v6ops meeting
at the IETF in Dallas.)

Regards
Brian

It really depends on how automated you are at running things. It took me
a few hours to brainstorm and document an addressing scheme which lets
us look at the 64 bit suffix and be able to tell in which DC the host is
located and what is it's function.

The setup for our CDN hosts is completely automated, and it took our
ansible guy a couple of hours to write an implementation that checks if
a given DC has IPv6 or not and does the right thing in either case.

Even if you are looking at this in the time-frame of a month, that is
far less than 20%, and in the time-frame of a year, it is negligible.

Testing the in-house software that runs the CDN was more work, but that
came under developer testing, not under IT operations.

If we had been managing all our servers by hand, it would have taken far
more time, but then again, the number of servers we run makes ANY change
which assumes manual intervention unmanageable.


Best regards,
Matija Grabnar


On 03/27/2015 12:59 PM, Phil Mayers wrote:
> On 26/03/15 09:04, BERENGUER Christophe wrote:
>> Hello everybody,
>>
>>
>> I work for a consulting firm.
>>
>>
>> For a client, I would like to estimate the work overload for IT
>> operations team to deploy IPv6 dual stack and for day to day operations.
>>
>>
>> On the internet, I have found an estimation around 20% of work overload
>> for the run phase. But if you have operational feedback it would be the
>> best!
>
> I agree with others that this is a very hard thing to estimate.
>
> I will say that we run our dual-stack network (fully deployed since
> ca. 2012) with exactly the same staffing levels, and actually a slight
> reduction in our recurrent budget, as our older IPv4-only network.
>
> I don't think our network is any less reliable, or suffers a higher
> level of incidents. This suggests to me that, in our case, IPv6 has
> added a very low operational cost. Our incidence of IPv6-related
> problems, particularly rogue RA from machines configured for
> connection sharing, has actually *decreased* substantially since we
> deployed native IPv6.
>
> I don't believe the rollout cost was high. We used refresh cycles to
> upgrade to v6-capable gear, and rolled out slowly to grow our team
> knowledge. But we don't have detailed cost breakdowns.

Hi,

I have found the figure in this document : https://tools.ietf.org/html/draft-lopez-v6ops-dc-ipv6-05#section-3.4

Regards,

Christophe BERENGUER
Consultant
Fixe : +33 (0)1 49 03 85 86
christophe.berenguer@solucom.fr
solucom
Tour Franklin : 100 - 101 terrasse Boieldieu
92042 Paris La Défense Cedex

________________________________________
De : Brian E Carpenter <brian.e.carpenter@gmail.com>
Envoyé : vendredi 27 mars 2015 13:21
À : BERENGUER Christophe
Cc : Ted Mittelstaedt; ipv6-ops@lists.cluenet.de
Objet : Re: Cost of IPv6 for IT operations team

> The 20% figure is based on an IETF document

Which one? We can fix that if people think it's wrong.
(This comes just too late for yesterday's v6ops meeting
at the IETF in Dallas.)

Regards
Brian

Ted Mittelstaedt <tedm@ipinc.net> writes:

> But as for operations costs, I would say, zero

I don't agree. In a dual-stacked environment there is more work to do.

1. Setting up Servers (and Services)

You have at least two addresses to configure, which leads to two DNS
records, two services to be monitored, more firewall rules, ... And in
the end more things to document.

2. Troubleshooting

When looking for problems you always have to remember that there are now two
protocols and then the fun begins: Is this problem only v4? Or only
v6? Or do have a completely different problem.

3. Layer 8+9

People have little or no experience with IPv6. They need more time to
configure stuff and troubleshoot problems. And they will forget to
configure one thing or the other and they will forget that there are
two protocols to troubleshoot.And then there will always be people
who don't want IPv6 an will always blame it[1].

> The reason is if you don't deploy sooner or later you will have a
> problem related to IPv6.

Ack.

> Then you will spends lots of time finding and correcting. That time
> is roughly equal to the extremely small amount of additional time that
> the techs deal with IPv6 on a network that has had it properly setup.

It will be worse. When you start implementing IPv6 because the latest
version of your critical application requires IPv6 and you need IPv6
tomorrow you'll have a real problem. You have to touch many things at
once, you may have to buy hardware and you'll be looking for qualified
external support. Problem is: Many other companies will do the same.

I'll have to remember to buy a big stash of T-Shirts with "Told you so!"

Jens

[1] Someone told me a story about a database server which stopped
working after IPv6 was deployed. The DB amdin blamed the IPv6
deployment. Of course it hat nothing to do with IPv6, the hard drive was
full.
--
----------------------------------------------------------------------------
| Foelderichstr. 40 | 13595 Berlin, Germany | +49-151-18721264 |
| http://blog.quux.de | jabber: jenslink@jabber.quux.de | --------------- |
----------------------------------------------------------------------------

Christophe,

On 28/03/2015 01:56, BERENGUER Christophe wrote:
> Hi,
>
> I have found the figure in this document : https://tools.ietf.org/html/draft-lopez-v6ops-dc-ipv6-05#section-3.4

That is a very old and expired personal draft, not an IETF document. Very dangerous
to rely on such a document. It was eventually replaced by a working group draft
http://tools.ietf.org/html/draft-ietf-v6ops-dc-ipv6-01#section-2.5.4
but that has not been accepted to become an RFC and has also expired
after an unsuccessful WG Last Call.

(You can always check the status of an Internet-Draft at
https://datatracker.ietf.org/doc/ .)

But in any case, look at the full context:
Depending of the complexity of the DC network, provisioning and other
factors we estimate that the extra costs (and later savings) may be
around between 15 to 20%.

Note the parenthesis!

Regards
Brian

> Regards,
>
> Christophe BERENGUER
> Consultant
> Fixe : +33 (0)1 49 03 85 86
> christophe.berenguer@solucom.fr
> solucom
> Tour Franklin : 100 - 101 terrasse Boieldieu
> 92042 Paris La Défense Cedex
>
> ________________________________________
> De : Brian E Carpenter <brian.e.carpenter@gmail.com>
> Envoyé : vendredi 27 mars 2015 13:21
> À : BERENGUER Christophe
> Cc : Ted Mittelstaedt; ipv6-ops@lists.cluenet.de
> Objet : Re: Cost of IPv6 for IT operations team
>
>> The 20% figure is based on an IETF document
>
> Which one? We can fix that if people think it's wrong.
> (This comes just too late for yesterday's v6ops meeting
> at the IETF in Dallas.)
>
> Regards
> Brian
>
>

I don't think that having "things" in the mix extends troubleshooting.

My experience is that problems generally fall into 2 buckets:

1) Textbook ones (mistyped IP address, overload that's readily visible
if they had looked at the reports, or like you said full hard disk)

The time-suck in these are techs who aren't very experienced, or skilled
or organized (or all the above)

With these, adding more stuff seems like it extends troubleshooting -
but the real reason troubleshooting is extended is not because you added
more stuff it's because your troubleshooting staff isn't up to snuff.

Kind of like you drafted Billy Joe Bob from Hayseed to troubleshoot your
2015 Chevy Volt and it extended the time troubleshooting because he's
still looking for the carburetor.

People only blame the car for that because they don't want to face old
Billy Bob and say 'dude your incompetent, hit the books or get the F out
of the business"

We live in a complex world that's getting more complex every day. If
you want to be a tech, then deal with it, up your game. When your
spending the same effort hitting the books as you are playing Warcraft
then I'll have some sympathy. Otherwise go join the line of people
sucking off the social services.

2) really knotty ones that take a lot of time, like intermittent failures.

With those, it's very hard to draw correlations. I have seen some very
simple, basic, 1-protocol setups that had really oddball problems that
took a great while to track down.

Just my $0.02

Ted

On 3/27/2015 6:37 AM, Jens Link wrote:
> Ted Mittelstaedt<tedm@ipinc.net> writes:
>
>> But as for operations costs, I would say, zero
>
> I don't agree. In a dual-stacked environment there is more work to do.
>
> 1. Setting up Servers (and Services)
>
> You have at least two addresses to configure, which leads to two DNS
> records, two services to be monitored, more firewall rules, ... And in
> the end more things to document.
>
> 2. Troubleshooting
>
> When looking for problems you always have to remember that there are now two
> protocols and then the fun begins: Is this problem only v4? Or only
> v6? Or do have a completely different problem.
>
> 3. Layer 8+9
>
> People have little or no experience with IPv6. They need more time to
> configure stuff and troubleshoot problems. And they will forget to
> configure one thing or the other and they will forget that there are
> two protocols to troubleshoot.And then there will always be people
> who don't want IPv6 an will always blame it[1].
>
>> The reason is if you don't deploy sooner or later you will have a
>> problem related to IPv6.
>
> Ack.
>
>> Then you will spends lots of time finding and correcting. That time
>> is roughly equal to the extremely small amount of additional time that
>> the techs deal with IPv6 on a network that has had it properly setup.
>
> It will be worse. When you start implementing IPv6 because the latest
> version of your critical application requires IPv6 and you need IPv6
> tomorrow you'll have a real problem. You have to touch many things at
> once, you may have to buy hardware and you'll be looking for qualified
> external support. Problem is: Many other companies will do the same.
>
> I'll have to remember to buy a big stash of T-Shirts with "Told you so!"
>
> Jens
>
> [1] Someone told me a story about a database server which stopped
> working after IPv6 was deployed. The DB amdin blamed the IPv6
> deployment. Of course it hat nothing to do with IPv6, the hard drive was
> full.

On Fri, 27 Mar 2015, Phil Mayers wrote:

> I don't believe the rollout cost was high. We used refresh cycles to
> upgrade to v6-capable gear, and rolled out slowly to grow our team
> knowledge. But we don't have detailed cost breakdowns.

This is my experience as well, if someone already has decided to employ
skilled people and tell them (or gave them fairly free hands) to have IPv6
enabled on most services in the "next few years", then this will happen
without very little extra cost.

If one decides that "We need IPv6 on services in the next 6 months" then
there'll be lots of extra truck rolls and immediate capex and opex if this
has not been taken into account previously.

It takes 3-5 years to get IPv6 running with minimal extra cost, then it
can be done as part of normal hardware refresh cycles and slowly
increasing exposure to IPv6 by staff. So the entities who have not started
yet and want to get started quickly will have to face increased cost.

I was involved in IPv6-enablement of a fairly large ISP and in 2007-2010
most of the groundwork was done going full native IPv6 instead of tunnels
etc, testing all platforms, working bug cases with vendors that had
software bugs, waiting for the fixes, testing again, then looking at the
next aspect like IPv6-enabling one of the services with friendly users,
run that for a while, then turning on more customers, enabling more
services etc. The earlier one starts, the more can be done over normal
upgrade cycles of software, you can help the vendor to get everything done
in a way that'll actually for for you.

Another strategy is to wait until everybody else has done this work and
hope you were in luck with the things you purchased and that it'll just
work out of the box without any preparation. This is also a valid
strategy, as long as not everybody does this :)

--
Mikael Abrahamsson email: swmike@swm.pp.se

All of this assumes "normal hardware refresh cycles" That may be the
case for ordinary office users but I got a million dollar asphalt kiln
attached to a Windows 98 controller with a serial port that says
otherwise.

Just wanting to point out that "normal hardware refresh cycles" more
often comes out of the mouth of the salesguy wanting to sell you
something than out of the network manager. Most of them will say "if it
ain't broke don't fix it" first.

And switching infrastructure tends to be pretty bulletproof if you
buy good hardware to start with. I've got a lot of customers with
switch infrastructure that hasn't changed in years.

"normal hardware refresh cycles" is something the Fortune 1000 have.
It's a nice concept but not one universally adopted by everyone else in
the real world. Another one of these is "hardware service contracts"
That's not universally adopted, either.

Ted


On 4/2/2015 3:04 AM, Mikael Abrahamsson wrote:
> On Fri, 27 Mar 2015, Phil Mayers wrote:
>
>> I don't believe the rollout cost was high. We used refresh cycles to
>> upgrade to v6-capable gear, and rolled out slowly to grow our team
>> knowledge. But we don't have detailed cost breakdowns.
>
> This is my experience as well, if someone already has decided to employ
> skilled people and tell them (or gave them fairly free hands) to have
> IPv6 enabled on most services in the "next few years", then this will
> happen without very little extra cost.
>
> If one decides that "We need IPv6 on services in the next 6 months" then
> there'll be lots of extra truck rolls and immediate capex and opex if
> this has not been taken into account previously.
>
> It takes 3-5 years to get IPv6 running with minimal extra cost, then it
> can be done as part of normal hardware refresh cycles and slowly
> increasing exposure to IPv6 by staff. So the entities who have not
> started yet and want to get started quickly will have to face increased
> cost.
>
> I was involved in IPv6-enablement of a fairly large ISP and in 2007-2010
> most of the groundwork was done going full native IPv6 instead of
> tunnels etc, testing all platforms, working bug cases with vendors that
> had software bugs, waiting for the fixes, testing again, then looking at
> the next aspect like IPv6-enabling one of the services with friendly
> users, run that for a while, then turning on more customers, enabling
> more services etc. The earlier one starts, the more can be done over
> normal upgrade cycles of software, you can help the vendor to get
> everything done in a way that'll actually for for you.
>
> Another strategy is to wait until everybody else has done this work and
> hope you were in luck with the things you purchased and that it'll just
> work out of the box without any preparation. This is also a valid
> strategy, as long as not everybody does this :)
>

On Fri, 3 Apr 2015, Ted Mittelstaedt wrote:

> "normal hardware refresh cycles" is something the Fortune 1000 have.
> It's a nice concept but not one universally adopted by everyone else in
> the real world. Another one of these is "hardware service contracts"
> That's not universally adopted, either.

Obviously, otherwise Win XP market share wouldn't be where it is
considering MS has stopped supporting it.

The state of the SCADA and other "industry" applications is the result of
a near total disregard for security when it comes to application
programming. This will hopefully improve, but it'll take time, the same
way IPv6 adoption will take time. However, it's still the truth that if
you're buying equipment today that you intend to have around for 5-10
years and you don't check them for IPv6 functionality, you're being
short-sighted.

--
Mikael Abrahamsson email: swmike@swm.pp.se

I know plenty of industrial machines that speak the pre-IP Microsoft
protocols to get G code files into them. Some also depend on the clock
rate of their 486 CPU for timing. Some have a C compiler on disk that
was used to develop the control software they run.

IPv6-enabling these devices is rather irrelevant, as they never speak to
the outside world, only to some other almost-as-outdated hosts. Either you
keep rfc1918 v4 (or NetBIOS over IPX) enabled on the desktops speaking
to the machine control PC, or you make some kind of bastion host which
can have IPv6. Either way the protocol spoken among these hosts
(which are hopefully controlled by common administration) is quite
irrelevant for the big picture. Expecting to have 20 or 30 year old
software successfully sending and receiving packets on the wild wild
internet is not a good idea and IPv6 is not in the top 10 reasons for
this.

Jussi

On Sat, Apr 04, 2015 at 04:20:43PM +0200, Mikael Abrahamsson wrote:
> On Fri, 3 Apr 2015, Ted Mittelstaedt wrote:
>
> >"normal hardware refresh cycles" is something the Fortune 1000
> >have. It's a nice concept but not one universally adopted by
> >everyone else in the real world. Another one of these is
> >"hardware service contracts" That's not universally adopted,
> >either.
>
> Obviously, otherwise Win XP market share wouldn't be where it is
> considering MS has stopped supporting it.
>
> The state of the SCADA and other "industry" applications is the
> result of a near total disregard for security when it comes to
> application programming. This will hopefully improve, but it'll take
> time, the same way IPv6 adoption will take time. However, it's still
> the truth that if you're buying equipment today that you intend to
> have around for 5-10 years and you don't check them for IPv6
> functionality, you're being short-sighted.
>
> --
> Mikael Abrahamsson email: swmike@swm.pp.se

The industrial machine was only used as an example to keep the
fools from replying "why would anyone want to run Windows XP anymore"

There are
still a lot of orgs that run it and run Chrome on it. And a simple
registry tweak gets you all of the patches that MS released for
Windows Embedded for POS systems and all of those apply to XP.

Mikael was right in that it is a good idea to select for IPv6 for
new purchases.

But you are both ignoring the point that many if not the majority
of smaller ogs do not replace working gear just because it's old.

The usual replacement drivers are the desire for new must-have features.

A secondary replacement driver is simple hardware failure. But hardware
failure on the PC market takes a lot longer than most PC manufacturers
would like, despite the electrolytic capacitor issues of the last decade.

I would guess XP is going to truly be a niche OS in another 5 years.
But until then a lot of people are still using it.

Ted

On 4/4/2015 12:50 PM, Jussi Peltola wrote:
> I know plenty of industrial machines that speak the pre-IP Microsoft
> protocols to get G code files into them. Some also depend on the clock
> rate of their 486 CPU for timing. Some have a C compiler on disk that
> was used to develop the control software they run.
>
> IPv6-enabling these devices is rather irrelevant, as they never speak to
> the outside world, only to some other almost-as-outdated hosts. Either you
> keep rfc1918 v4 (or NetBIOS over IPX) enabled on the desktops speaking
> to the machine control PC, or you make some kind of bastion host which
> can have IPv6. Either way the protocol spoken among these hosts
> (which are hopefully controlled by common administration) is quite
> irrelevant for the big picture. Expecting to have 20 or 30 year old
> software successfully sending and receiving packets on the wild wild
> internet is not a good idea and IPv6 is not in the top 10 reasons for
> this.
>
> Jussi
>
> On Sat, Apr 04, 2015 at 04:20:43PM +0200, Mikael Abrahamsson wrote:
>> On Fri, 3 Apr 2015, Ted Mittelstaedt wrote:
>>
>>> "normal hardware refresh cycles" is something the Fortune 1000
>>> have. It's a nice concept but not one universally adopted by
>>> everyone else in the real world. Another one of these is
>>> "hardware service contracts" That's not universally adopted,
>>> either.
>>
>> Obviously, otherwise Win XP market share wouldn't be where it is
>> considering MS has stopped supporting it.
>>
>> The state of the SCADA and other "industry" applications is the
>> result of a near total disregard for security when it comes to
>> application programming. This will hopefully improve, but it'll take
>> time, the same way IPv6 adoption will take time. However, it's still
>> the truth that if you're buying equipment today that you intend to
>> have around for 5-10 years and you don't check them for IPv6
>> functionality, you're being short-sighted.
>>
>> --
>> Mikael Abrahamsson email: swmike@swm.pp.se

On 26 Mar 2015, at 09:04, BERENGUER Christophe <Christophe.BERENGUER@solucom.fr> wrote:

> For a client, I would like to estimate the work overload for IT operations team to deploy IPv6 dual stack and for day to day operations.
> On the internet, I have found an estimation around 20% of work overload for the run phase. But if you have operational feedback it would be the best!

This is a strange question, but I think as a consultant I would want to design a rollout methodology that made that overhead (cost) as close to zero for your customer as possible.

For example - and I speak generically, but this may be appropriate for some firms :

Stage one - write v6 support requirements into RFP for equipment before you plan to turn it on. Ideally this will be based on document RIPE-554 and be part of your buying process already (and have been in this process for at least the last buying cycle!) This way, when you want to turn v6 support on, the incremental cost of your hardware support is zero.

Stage two - training, get a tunnel into the lab, make comfort with v6 part of your technical appraisal for the technology teams, so that when it’s time to turn it on your team are familiar and will make fewer mistakes.

Stage three - roll it out at your network edge, core & dns. Yes this is a project which needs time management and planning and incurs cost.

Stage four - utilise your new training and v6 capable edge to roll out NEW services dual-stack. The incremental cost of adding v6 support to a NEW rollout when you have to do a bunch of work to roll out a service at all is therefore zero. v6 support for existing services can be added in product refreshes in time.

Andy

On 10/04/2015 21:36, Andy Davidson wrote:
> Stage one - [...]
> Stage two - [...]
> Stage three - [...]
>
> Stage four - utilise your new training and v6 capable edge to roll out
> NEW services dual-stack. The incremental cost of adding v6 support to a
> NEW rollout when you have to do a bunch of work to roll out a service at
> all is therefore zero. v6 support for existing services can be added in
> product refreshes in time.

Uh, lemme just drop this in here: http://imgur.com/AYbpRG2

Stage 4 might be a good way of burying deployment costs but I'm going to
assert that stages 1 through 3 are the easier, lower cost bits. The reason
is that stages 1-3 can be deployed relatively easily by a tiny number of
people even on reasonably large networks. Although stage 3 - where you
state the costs lie - is the first place which causes a direct and up-front
cost to be incurred in terms of resourcing and config, it can still be
rolled out relatively quickly and easily.

The problem with stage 4 is that it requires that the expertise garnered by
the initial deployment team is spread throughout the rest of the company,
ranging from product development to the FLS desk, right through to
customers. This is a prolonged and time-consuming process, and
consequently expensive. I'd love to see a larger scale discussion about
this, because it's one of the main blockages for ipv6 adoption and
discussion of live cases would help other organisations make the jump.

Nick

Thus wrote Andy Davidson (andy@nosignal.org):

> Stage one - write v6 support requirements into RFP for equipment before you plan to turn it on. Ideally this will be based on document RIPE-554 and be part of your buying process already (and have been in this process for at least the last buying cycle!) This way, when you want to turn v6 support on, the incremental cost of your hardware support is zero.
>
> Stage two - training, get a tunnel into the lab, make comfort with v6 part of your technical appraisal for the technology teams, so that when it’s time to turn it on your team are familiar and will make fewer mistakes.
>
> Stage three - roll it out at your network edge, core & dns. Yes this is a project which needs time management and planning and incurs cost.
>
> Stage four - utilise your new training and v6 capable edge to roll out NEW services dual-stack. The incremental cost of adding v6 support to a NEW rollout when you have to do a bunch of work to roll out a service at all is therefore zero. v6 support for existing services can be added in product refreshes in time.

No "identify gaps of core custom-written software asuming that servers
have exactly one IP address, and it matches \d+\.\d+\.\d+\.\d+ (or language
equivalents), and fix them", at all? "We have IPv6 in the network, but we
don't use it for anything (except Web surfing)" does seem not particularily
useful. Might as well just deploy v6 to the DMZ and save yourself
a lot of hassle if that is all you'll do.

Or were you just ignoring software as Someone Elses Problem (tm)? :)

regards,
spz

(currently doing devops for a manufactorer with ~5000 Unix servers running
the plants as a dayjob)
--
spz@serpens.de (S.P.Zeidler)

Hi Andy and list,

Andy Davidson <andy@nosignal.org> writes:

> This is a strange question, but I think as a consultant I would want
> to design a rollout methodology that made that overhead (cost) as
> close to zero for your customer as possible.

that depends; in some cases you're far more interested in minimizing the
time to delivery, and then things look entirely different.

> Stage one - write v6 support requirements into RFP for equipment
> before you plan to turn it on. Ideally this will be based on document
> RIPE-554 and be part of your buying process already (and have been in
> this process for at least the last buying cycle!) This way, when you
> want to turn v6 support on, the incremental cost of your hardware
> support is zero.

That requires a lot of time. Sure, if you've started this five years
before you actually deployed IPv6 then yes, the additional cost is
zero.

And yes, back in 95 there were allegedly people testing the hardware
they bought for being year 2000 compliant:-)

> Stage two - training, get a tunnel into the lab, make comfort with v6
> part of your technical appraisal for the technology teams, so that
> when it’s time to turn it on your team are familiar and will make
> fewer mistakes.

Which is a major effort in some environments because---contrary to what
Nick wrote---pretty much anyone involved needs to be familiarized with
IPv6. The reason here is that if there is any problem once IPv6 is
enabled anywhere, then *all* people doing the troubleshooting need to
know how to exclude IPv6 from the list of possible problem causes.

This is especially bad if you're doing 24/7. If you screw this up, then
"a tiny number of people" will be exceedingly busy figuring out that
(hopefully) the majority of problems are eventually unrelated to IPv6
while the rest stand around and watch until they can actually continue
with their part of the work.

And as soon as you add a round or so of blame game, that "tiny number of
people" will also have to communicate towards management that no, no
matter what some other people claim, it isn't an IPv6 fault---which is
exceedingly difficult since this situation usually leads to management
turning towards the trouble ticket management system only to discover
that (a) pretty much anytime anything went wrong IPv6 was somehow
involved according to the paper trail and (b) time to fix has
significantly increased due to this additional showstopper step.

> Stage three - roll it out at your network edge, core & dns. Yes this
> is a project which needs time management and planning and incurs cost.

In an enterprise network that's actually a minor issue compared to
e.g. the business applications that need to be fixed up, or the overdue
OS updates that need to be deployed.

> Stage four - utilise your new training and v6 capable edge to roll out
> NEW services dual-stack. The incremental cost of adding v6 support to
> a NEW rollout when you have to do a bunch of work to roll out a
> service at all is therefore zero. v6 support for existing services
> can be added in product refreshes in time.

Just one semi-quote here: "That code has been running nicely without any
updates since 1999; the one last PL/1 (or Algol, or Cobol, or BS2000
assembler) programmer we still had fixed it up for year 2000 and then
retired."

While it's really convenient to use product refresh cycles to enable in
some cases, from what I've seen at least in enterprise environments this
frequently doesn't work as a general approach.

And aside from that you also assume that you have the time to wait for
at least one refresh cycle (more if there are inconvenient dependencies
between services). As mentioned before that's way from universally
applicable.


In fact, coming up with a number for the effort needed to deploy or run
IPv6 involves a huge amount of work on the details. For example, if an
in-house software development team has been smart enough to use a single
high-level library for all network operations, then the choice of that
single library from twenty years ago can determine if enabling IPv6 on
all in-house software happens automatically or involves pretty much a
full rewrite of the entire source base.

In other words, just because someone claims some number measured in one
specific environment doesn't mean anything at all to yours.


Cheers,

Benedikt

--
Benedikt Stockebrand, Stepladder IT Training+Consulting
Dipl.-Inform. http://www.stepladder-it.com/

Business Grade IPv6 --- Consulting, Training, Projects

BIVBlog---Benedikt's IT Video Blog: http://www.stepladder-it.com/bivblog/

On 11/04/15 10:27, Nick Hilliard wrote:

> Uh, lemme just drop this in here: http://imgur.com/AYbpRG2

;o)

>
> The problem with stage 4 is that it requires that the expertise garnered by
> the initial deployment team is spread throughout the rest of the company,
> ranging from product development to the FLS desk, right through to
> customers. This is a prolonged and time-consuming process, and
> consequently expensive. I'd love to see a larger scale discussion about
> this, because it's one of the main blockages for ipv6 adoption and
> discussion of live cases would help other organisations make the jump.
>

University environment here, fully dual-stacked everywhere.

Interestingly enough, we've not done a huge amount of training across
the rest of the IT support function. It was always intended, but it's
actually surprisingly rare to need to refer to an IP address - v4 or v6
- when troubleshooting problems. MAC addresses are, generally, far more
useful, and with v4/v6 ARP/ND tracking, give you the IPs.

We have spread some knowledge into our "infrastructure applications"
team - exchange, webservers, etc. - and that's been fine.

We've had a lot less luck spreading the knowledge into the teams which
deal with COTS application stacks e.g. Oracle. The problem there is the
"upstream" support - if Oracle don't certify or deploy on IPv6, the
people running that stack have no interesting in doing so.

The major thing is "don't forget to configure the IPv6 vhost". But by
dual-stacking everyones desktop, including the developers, that gets
picked up early.

For supporting client connectivity (desktops, laptops) we've not found
the protocol differences very relevant. MAC address on wired, 802.1x
username on wireless or MAC if it's an auth-level problem.

We just haven't found it a problem. Maybe it's a University thing. I'm
sure consumer ISPs have a much harder time.

Cheers,
Phil

On 13/04/15 09:55, Benedikt Stockebrand wrote:

> Which is a major effort in some environments because---contrary to what
> Nick wrote---pretty much anyone involved needs to be familiarized with
> IPv6. The reason here is that if there is any problem once IPv6 is
> enabled anywhere, then *all* people doing the troubleshooting need to
> know how to exclude IPv6 from the list of possible problem causes.

See, this makes sense - it's a reasonable assertion. But it's the
opposite of what we've found.

In all honesty, I doubt a large portion of the organisation understands
how IPv*4* works. If you think about one of the many areas v6 differs -
ARP/DHCP versus ND/SLAAC/DHCPv6 - how many people *really* understood
the former that well?

IME, not many.

So I'm a bit conflicted. What you say makes sense, but contradicts my
experience. Still, data != sumof(anecdote) and all ;o)

> And as soon as you add a round or so of blame game, that "tiny number of
> people" will also have to communicate towards management that no, no
> matter what some other people claim, it isn't an IPv6 fault---which is
> exceedingly difficult since this situation usually leads to management
> turning towards the trouble ticket management system only to discover
> that (a) pretty much anytime anything went wrong IPv6 was somehow
> involved according to the paper trail and (b) time to fix has
> significantly increased due to this additional showstopper step.

OTOH, we often find the network blamed for basically everything. So this
is only a small extension ;o)

> In an enterprise network that's actually a minor issue compared to
> e.g. the business applications that need to be fixed up, or the overdue
> OS updates that need to be deployed.

+1

> While it's really convenient to use product refresh cycles to enable in
> some cases, from what I've seen at least in enterprise environments this
> frequently doesn't work as a general approach.

What problems have you seen? Insufficiently demanding RFP or lack of
awareness of the need to put it in an RFP? Or something else?

> In other words, just because someone claims some number measured in one
> specific environment doesn't mean anything at all to yours.

Very strongly agreed.

At this stage of "IT support" as a profession - encompassing the
technology, tools, management structures and so forth - it's clear to me
there's huge variation, making any comparisons dangerously close to
apples/oranges.

Cheers,
Phil