Mailing List Archive

<cough>spof</cough>

the vita folk have a history of 'not really understanding large scale
compute/network operations' :(

On Tue, Oct 13, 2020 at 11:06 AM Sean Donelan <sean@donelan.com> wrote:
>
>
> On the last day of Virginia voter registration, the state-wide voter
> registration system experienced a cable cut disrupting access to the
> state-wide database system.
>
> Absent clear and convincing evidence otherwise, the problems will likely
> be caused by the usual stupid stuff.
>
>
>
> VITA
> @VITAagency
> A fiber cut near Rt. 10 in Chester near the Commonwealth Enterprise
> Solutions Center (CESC) is impacting data circuits and virtual private
> network (VPN) connectivity for multiple Commonwealth agencies.
>
>
> VITA
> @VITAagency
> Technicians are on site and working to repair the cut; updates will be
> provided as work progresses.

On Tue, 13 Oct 2020, Christopher Morrow wrote:
> <cough>spof</cough>
>
> the vita folk have a history of 'not really understanding large scale
> compute/network operations' :(

Reportedly, the VITA data center and Virginia voter registration system is
back up.

According to VITA, a Verizon fiber was struck during a roadside utilities
construction project near Route 10 in Chester, VA. As network engineers
know, fiber cuts happen all the time due to construction. Malicious cuts
and sabatoge occur, but are rare and usually obvious. Absent clear and
compelling evidence, assume normal stupid reasons for outages.


There are various long-term structural problems with election
administration across 10,000+ jurisdictions in the United States. A lot
of duct-tape and heroic work needed by election administrators to keep
things running.


Election administration in Australia, Norway and Luxembourg tend to score
the best with 10 out of 10 according to international election observers.

Election administration in the United States of American tends to score
around 7 or 8. There will be problems and delays. Not perfect,
embarrassing and USA should do better. But still a full and free election.

On Tue, Oct 13, 2020 at 2:41 PM Sean Donelan <sean@donelan.com> wrote:
>
> On Tue, 13 Oct 2020, Christopher Morrow wrote:
> > <cough>spof</cough>
> >
> > the vita folk have a history of 'not really understanding large scale
> > compute/network operations' :(
>
> Reportedly, the VITA data center and Virginia voter registration system is
> back up.
>
> According to VITA, a Verizon fiber was struck during a roadside utilities
> construction project near Route 10 in Chester, VA. As network engineers
> know, fiber cuts happen all the time due to construction. Malicious cuts
> and sabatoge occur, but are rare and usually obvious. Absent clear and
> compelling evidence, assume normal stupid reasons for outages.
>

sorry I meant that: 1) yes clearly it's still the middle of
roadwork/backhoe season, 2) i'm surprised that a single path failure
for their production datacenter was enough to take the system offline.
'spof' there meant: "Wow, a single point of failure in their outside
plant?"

>
> There are various long-term structural problems with election
> administration across 10,000+ jurisdictions in the United States. A lot
> of duct-tape and heroic work needed by election administrators to keep
> things running.
>
>
> Election administration in Australia, Norway and Luxembourg tend to score
> the best with 10 out of 10 according to international election observers.
>
> Election administration in the United States of American tends to score
> around 7 or 8. There will be problems and delays. Not perfect,
> embarrassing and USA should do better. But still a full and free election.

On Tue, 13 Oct 2020 17:11:53 -0400, Christopher Morrow said:

> sorry I meant that: 1) yes clearly it's still the middle of
> roadwork/backhoe season, 2) i'm surprised that a single path failure
> for their production datacenter was enough to take the system offline.
> 'spof' there meant: "Wow, a single point of failure in their outside
> plant?"

Given that back in 2010, they suffered a *disastrous* outage when
a storage array failed and took multiple agencies with it....

https://www.computerworld.com/article/2515423/northrop-grumman-takes-blame-for-va--it-services-outage.html

my reaction was more like
<voice="Gomer Pyle">
Surprise, surprise, surprise...
</voice>

That one started when one storage array had a failed memory card, and
the backup array encountered issues as well. There were a number of state
agencies and universities that had fought for increased self-governance, and
a *huge* part of that was "not be forced to outsource their internal IT to VITA",
and those units were very glad they had won that fight....

On Tue, Oct 13, 2020 at 5:15 PM Christopher Morrow
<morrowc.lists@gmail.com> wrote:
>
> On Tue, Oct 13, 2020 at 2:41 PM Sean Donelan <sean@donelan.com> wrote:
> >
> > On Tue, 13 Oct 2020, Christopher Morrow wrote:
> > > <cough>spof</cough>
> > >
> > > the vita folk have a history of 'not really understanding large scale
> > > compute/network operations' :(
> >
> > Reportedly, the VITA data center and Virginia voter registration system is
> > back up.
> >
> > According to VITA, a Verizon fiber was struck during a roadside utilities
> > construction project near Route 10 in Chester, VA. As network engineers
> > know, fiber cuts happen all the time due to construction. Malicious cuts
> > and sabatoge occur, but are rare and usually obvious. Absent clear and
> > compelling evidence, assume normal stupid reasons for outages.
> >
>
> sorry I meant that: 1) yes clearly it's still the middle of
> roadwork/backhoe season, 2) i'm surprised that a single path failure
> for their production datacenter was enough to take the system offline.
> 'spof' there meant: "Wow, a single point of failure in their outside
> plant?"
>

... and there is now a lawsuit about this:
Groups file lawsuit after Virginia state websites shut down for hours
on last day of voter registration

"An accidentally severed fiber-optic cable in Virginia effectively
shut down most of the state’s online voter registration on its last
day Tuesday, prompting voter advocates to file a lawsuit in federal
court seeking an extension of the deadline that they argue thousands
of voters missed because of the disruption."...

https://www.washingtonpost.com/local/va-politics/virginia-state-websites-shut-down/2020/10/13/8e4443d8-0d5d-11eb-b1e8-16b59b92b36d_story.html


W
> >
> > There are various long-term structural problems with election
> > administration across 10,000+ jurisdictions in the United States. A lot
> > of duct-tape and heroic work needed by election administrators to keep
> > things running.
> >
> >
> > Election administration in Australia, Norway and Luxembourg tend to score
> > the best with 10 out of 10 according to international election observers.
> >
> > Election administration in the United States of American tends to score
> > around 7 or 8. There will be problems and delays. Not perfect,
> > embarrassing and USA should do better. But still a full and free election.



--
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
---maf

On Tue, 13 Oct 2020, Valdis Kl?tnieks wrote:
> my reaction was more like
> <voice="Gomer Pyle">
> Surprise, surprise, surprise...
> </voice>

S.N.A.F.U.

Other SNAFUs, Georgia had technical problems with its voter database
systems during the first couple of days of early voting. Expect all sorts
of minor problems throughout the election and afterwards. Nonetheless they
are unlikely to significantly impact the results (hopefully), but will
generate lots of noise.

Its not just underfunded state I.T. systems. Even very large social media
companies can have technical Oopsies. Again, hopefully Twitter won't
fall down again during the evening of November 3rd. The digeratti will
lose thier minds.

Even if Twitter or another major social media platform does go belly-up,
most likely it will be a normal technical problem. Wishing the FBI & CISA
& OGAs watch officers a very boring night on November 3rd.



In other news, New Zealand is having national elections this weekend. New
Zealand is usually ranked in the top 10 best election administrations
worldwide. NZ expects to have the majority of ballots counted within 2
hours of their polls closing on Saturday evening.

Jealous of the Kiwis and their competently run elections. :-)

    Hi,

    Beside being:

        . a country with 1/10th of the population;

        . centralized voting rules;

        . <yadi yada>...


    PS: And there is a lot in that <yadi yada> about the (publicly)
unreal amount of insanity being pulled by the GOP  this year.

-----
Alain Hebert ahebert@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443

On 2020-10-16 02:36, Sean Donelan wrote:
> On Tue, 13 Oct 2020, Valdis Kl?tnieks wrote:
>> my reaction was more like
>> <voice="Gomer Pyle">
>> Surprise, surprise, surprise...
>> </voice>
>
> S.N.A.F.U.
>
> Other SNAFUs, Georgia had technical problems with its voter database
> systems during the first couple of days of early voting. Expect all
> sorts of minor problems throughout the election and afterwards.
> Nonetheless they are unlikely to significantly impact the results
> (hopefully), but will generate lots of noise.
>
> Its not just underfunded state I.T. systems.  Even very large social
> media companies can have technical Oopsies.  Again, hopefully Twitter
> won't fall down again during the evening of November 3rd.  The
> digeratti will lose thier minds.
>
> Even if Twitter or another major social media platform does go
> belly-up, most likely it will be a normal technical problem. Wishing
> the FBI & CISA & OGAs watch officers a very boring night on November 3rd.
>
>
>
> In other news, New Zealand is having national elections this weekend. 
> New Zealand is usually ranked in the top 10 best election
> administrations worldwide. NZ expects to have the majority of ballots
> counted within 2 hours of their polls closing on Saturday evening.
>
> Jealous of the Kiwis and their competently run elections. :-)

It’s not the population. It’s the number of positions/things you are voting for. New Zealand doesn’t vote for sheriffs, judges, mayors etc. AFAIK so there is much less to count. More population should lead to more polling stations. These need to be collated but that is a relatively quick job compared to counting the votes.

Timezone spread also makes the night longer. If you have a result within 2 hours of the Hawaiian polls closing you are on par.

--
Mark Andrews

> On 17 Oct 2020, at 07:49, Alain Hebert <ahebert@pubnix.net> wrote:
>
> ? Hi,
>
> Beside being:
>
> . a country with 1/10th of the population;
>
> . centralized voting rules;
>
> . <yadi yada>...
>
>
> PS: And there is a lot in that <yadi yada> about the (publicly) unreal amount of insanity being pulled by the GOP this year.
>
> -----
> Alain Hebert ahebert@pubnix.net
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443
> On 2020-10-16 02:36, Sean Donelan wrote:
>> On Tue, 13 Oct 2020, Valdis Kl?tnieks wrote:
>>> my reaction was more like
>>> <voice="Gomer Pyle">
>>> Surprise, surprise, surprise...
>>> </voice>
>>
>> S.N.A.F.U.
>>
>> Other SNAFUs, Georgia had technical problems with its voter database systems during the first couple of days of early voting. Expect all sorts of minor problems throughout the election and afterwards. Nonetheless they are unlikely to significantly impact the results (hopefully), but will generate lots of noise.
>>
>> Its not just underfunded state I.T. systems. Even very large social media companies can have technical Oopsies. Again, hopefully Twitter won't fall down again during the evening of November 3rd. The digeratti will lose thier minds.
>>
>> Even if Twitter or another major social media platform does go belly-up, most likely it will be a normal technical problem. Wishing the FBI & CISA & OGAs watch officers a very boring night on November 3rd.
>>
>>
>>
>> In other news, New Zealand is having national elections this weekend. New Zealand is usually ranked in the top 10 best election administrations worldwide. NZ expects to have the majority of ballots counted within 2 hours of their polls closing on Saturday evening.
>>
>> Jealous of the Kiwis and their competently run elections. :-)
>

>In other news, New Zealand is having national elections this weekend.
>New Zealand is usually ranked in the top 10 best election administrations
>worldwide. NZ expects to have the majority of ballots counted within 2
>hours of their polls closing on Saturday evening.

I thought the HGIC (Head Ghoul In Charge) had cancelled all the elections in New Zealand and declared herself to the new Dictator Ghoul In Charge until ousted by violent uprising by the peasants? Has there been an uprising to oust the Head Ghoul via bullet-to-the-brain that was not reported in the socialist media?

--
Be decisive. Make a decision, right or wrong. The road of life is paved with flat squirrels who could not make a decision.

To keep this on topic, no mention of any candidate or political party.


Networks can impact elections, and elections can impact network operators.

If @netblocks <www.netblocks.org> publishes a special report about your
country's elections, that's a huge indicator of network (and election)
problems. In the 1950's, often the first indication something was
happening was the national radio station in a country suddenly started
playing uninterrupted classical music. Now the modern equivalent is when
mobile networks stop working and social media sites are blocked in a
country.


There are many day-to-day causes of network problems.

I don't expect backhoes to stop cutting fiber cables on election day.

I don't expect network engineers to stop making mistakes on election day.

I don't expect software bugs to stop occurring on election day.


In the USA, absent clear and convincing evidence otherwise, I expect any
outages will be due to the normal things that cause outages on election day.

If you do experience an outage, you probably want to have your P.R. people
on call for a quicker than normal response.

    Or what Matt?

-----
Alain Hebert ahebert@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443

On 2020-10-16 19:26, Matt Erculiani wrote:
> Non-network-related government politics are off-topic for NANOG.
> Please keep them to yourself.
>
> Thanks,
> -Matt
>
> On Fri, Oct 16, 2020 at 2:49 PM Alain Hebert <ahebert@pubnix.net
> <mailto:ahebert@pubnix.net>> wrote:
>
>     Hi,
>
>     Beside being:
>
>         . a country with 1/10th of the population;
>
>         . centralized voting rules;
>
>         . <yadi yada>...
>
>
>     PS: And there is a lot in that <yadi yada> about the
> (publicly) unreal amount of insanity being pulled by the GOP  this
> year.
>
> -----
> Alain Hebertahebert@pubnix.net <mailto:ahebert@pubnix.net>
> PubNIX Inc.
> 50 boul. St-Charles
> P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
> Tel: 514-990-5911http://www.pubnix.net Fax: 514-990-9443
>
> On 2020-10-16 02:36, Sean Donelan wrote:
>> On Tue, 13 Oct 2020, Valdis Kl?tnieks wrote:
>>> my reaction was more like
>>> <voice="Gomer Pyle">
>>> Surprise, surprise, surprise...
>>> </voice>
>>
>> S.N.A.F.U.
>>
>> Other SNAFUs, Georgia had technical problems with its voter
>> database systems during the first couple of days of early voting.
>> Expect all sorts of minor problems throughout the election and
>> afterwards. Nonetheless they are unlikely to significantly impact
>> the results (hopefully), but will generate lots of noise.
>>
>> Its not just underfunded state I.T. systems.  Even very large
>> social media companies can have technical Oopsies. Again,
>> hopefully Twitter won't fall down again during the evening of
>> November 3rd.  The digeratti will lose thier minds.
>>
>> Even if Twitter or another major social media platform does go
>> belly-up, most likely it will be a normal technical problem. 
>> Wishing the FBI & CISA & OGAs watch officers a very boring night
>> on November 3rd.
>>
>>
>>
>> In other news, New Zealand is having national elections this
>> weekend.  New Zealand is usually ranked in the top 10 best
>> election administrations worldwide. NZ expects to have the
>> majority of ballots counted within 2 hours of their polls closing
>> on Saturday evening.
>>
>> Jealous of the Kiwis and their competently run elections. :-)
>
>
>
> --
> Matt Erculiani
> ERCUL-ARIN

>
> Again, hopefully Twitter won't
> fall down again during the evening of November 3rd. The digeratti will
> lose thier minds.
>

No offense towards our many friends at Twitter, but I think the best thing
that could happen would be for Twitter to shut 'er down from November 1
through November 6. Remove one of the primary avenues for rapid
disinformation and confusion, and allow tabulations to proceed without
extra nonsense.

This of course won't happen, but one can dream.

On Fri, Oct 16, 2020 at 2:37 AM Sean Donelan <sean@donelan.com> wrote:

> On Tue, 13 Oct 2020, Valdis Kl?tnieks wrote:
> > my reaction was more like
> > <voice="Gomer Pyle">
> > Surprise, surprise, surprise...
> > </voice>
>
> S.N.A.F.U.
>
> Other SNAFUs, Georgia had technical problems with its voter database
> systems during the first couple of days of early voting. Expect all sorts
> of minor problems throughout the election and afterwards. Nonetheless they
> are unlikely to significantly impact the results (hopefully), but will
> generate lots of noise.
>
> Its not just underfunded state I.T. systems. Even very large social media
> companies can have technical Oopsies. Again, hopefully Twitter won't
> fall down again during the evening of November 3rd. The digeratti will
> lose thier minds.
>
> Even if Twitter or another major social media platform does go belly-up,
> most likely it will be a normal technical problem. Wishing the FBI & CISA
> & OGAs watch officers a very boring night on November 3rd.
>
>
>
> In other news, New Zealand is having national elections this weekend. New
> Zealand is usually ranked in the top 10 best election administrations
> worldwide. NZ expects to have the majority of ballots counted within 2
> hours of their polls closing on Saturday evening.
>
> Jealous of the Kiwis and their competently run elections. :-)
>

On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote:
> In the USA, absent clear and convincing evidence otherwise, I expect any
> outages will be due to the normal things that cause outages on election day.

One of those things is the chronic underfunding of the systems/personnel
involved. (This isn't the case everywhere of course but it's the case in
a lot of places.) This leads to operations that are working -- barely --
and thus not resistant to stress, outages, and mistakes.

We could argue that given the criticality of this particular function
that resources should be more generously allocated, but unfortunately
that's often an unsuccessful argument.

---rsk

In my experiences with both government IT and elections departments, funding may be a problem, but a definite problem is an unwillingness to relieve poorly performing staff. They could be doing so much better with the funding they have.




-----
Mike Hammett
Intelligent Computing Solutions

Midwest Internet Exchange

The Brothers WISP

----- Original Message -----

From: "Rich Kulawiec" <rsk@gsp.org>
To: nanog@nanog.org
Sent: Monday, October 19, 2020 11:58:23 AM
Subject: Re: Virginia voter registration down due to cable cut

On Sat, Oct 17, 2020 at 07:44:01PM -0400, Sean Donelan wrote:
> In the USA, absent clear and convincing evidence otherwise, I expect any
> outages will be due to the normal things that cause outages on election day.

One of those things is the chronic underfunding of the systems/personnel
involved. (This isn't the case everywhere of course but it's the case in
a lot of places.) This leads to operations that are working -- barely --
and thus not resistant to stress, outages, and mistakes.

We could argue that given the criticality of this particular function
that resources should be more generously allocated, but unfortunately
that's often an unsuccessful argument.

---rsk

Actually we count lots of things, though we keep civic separate from state (we did that last year).We delayed our election by 4 weeks because of covid.Voting is open for a number of weeks.  About 72% of voters had voted before the last day.Voting is on paper.Voters don't have to register before voting (though that is prefered).Count started on the last day and reporting started at about 7.30pm (polls closed at 7pm)Counting was mostly done by midnight.This term we also have two referendums but they won't report back until the end of October.We don't have queues because we have so many places you can vote.https://i.stuff.co.nz/national/politics/300124486/election-2020-christchurch-east-voting-booth-locationsThere were 3 locations within 750m of my house.Sorry to hear about your fibre...https://i.stuff.co.nz/business/industries/122715051/return-of-the-cyberattackers-nzx-website-goes-offline-againI don't think a fibre cut would take down anything to serious in New Zealand....  but it does depend on what you consider important. ;)DD-- Don Gould5 Cargill PlaceRichmondChristchurch, New ZealandMobile/Telegram: + 64 21 114 0699www.bowenvale.co.nz
-------- Original message --------From: Mark Andrews <marka@isc.org> Date: 17/10/20 11:29 am (GMT+12:00) To: ahebert@pubnix.net Cc: nanog@nanog.org Subject: Re: Virginia voter registration down due to cable cut It’s not the population. It’s the number of positions/things you are voting for.   New Zealand doesn’t vote for sheriffs, judges, mayors etc. AFAIK so there is much less to count.  More population should lead to more polling stations. These need to be collated but that is a relatively quick job compared to counting the votes. Timezone spread also makes the night longer.  If you have a result within 2 hours of the Hawaiian polls closing you are on par. -- Mark AndrewsOn 17 Oct 2020, at 07:49, Alain Hebert <ahebert@pubnix.net> wrote:?




    Hi,

    Beside being:

        . a country with 1/10th of the population;

        . centralized voting rules;

        . <yadi yada>...


    PS: And there is a lot in that <yadi yada> about the
(publicly) unreal amount of insanity being pulled by the GOP  this
year.

-----
Alain Hebert ahebert@pubnix.net
PubNIX Inc.
50 boul. St-Charles
P.O. Box 26770 Beaconsfield, Quebec H9W 6G7
Tel: 514-990-5911 http://www.pubnix.net Fax: 514-990-9443

On 2020-10-16 02:36, Sean Donelan
wrote:

On Tue,
13 Oct 2020, Valdis Kl?tnieks wrote:

my reaction was more like

<voice="Gomer Pyle">

Surprise, surprise, surprise...

</voice>



S.N.A.F.U.


Other SNAFUs, Georgia had technical problems with its voter
database systems during the first couple of days of early voting.
Expect all sorts of minor problems throughout the election and
afterwards. Nonetheless they are unlikely to significantly impact
the results (hopefully), but will generate lots of noise.


Its not just underfunded state I.T. systems.  Even very large
social media companies can have technical Oopsies.  Again,
hopefully Twitter won't fall down again during the evening of
November 3rd.  The digeratti will lose thier minds.


Even if Twitter or another major social media platform does go
belly-up, most likely it will be a normal technical problem. 
Wishing the FBI & CISA & OGAs watch officers a very boring
night on November 3rd.




In other news, New Zealand is having national elections this
weekend.  New Zealand is usually ranked in the top 10 best
election administrations worldwide. NZ expects to have the
majority of ballots counted within 2 hours of their polls closing
on Saturday evening.


Jealous of the Kiwis and their competently run elections. :-)