Mailing List Archive

Same stuff (with our ASN and our prefixes) detected here, coming
from AS2027 (Milkywan), for a short time...

Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a
écrit :

> We just had every single prefix in AS378 start being announced by AS2027.
>
> Every announcement by AS2027 is failing RPKI yet being propagated a bit.
> Is this yet another misbehaving device or an actual attack?
>

--
*Ce message et toutes les pièces jointes (ci-après le "message") sont
établis à l’intention exclusive des destinataires désignés. Il contient des
informations confidentielles et pouvant être protégé par le secret
professionnel. Si vous recevez ce message par erreur, merci d'en avertir
immédiatement l'expéditeur et de détruire le message. Toute utilisation de
ce message non conforme à sa destination, toute diffusion ou toute
publication, totale ou partielle, est interdite, sauf autorisation expresse
de l'émetteur*

Hank, all exact match for prefix length? Or longer subnets covering the
whole?
(Is this leakage of a optimizer or possibly censorship leakage?)

On Sun, Oct 22, 2023, 1:03?PM Olivier Benghozi <olivier.benghozi@wifirst.fr>
wrote:

> Same stuff (with our ASN and our prefixes) detected here, coming
> from AS2027 (Milkywan), for a short time...
>
> Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a
> écrit :
>
>> We just had every single prefix in AS378 start being announced by AS2027.
>>
>> Every announcement by AS2027 is failing RPKI yet being propagated a bit.
>> Is this yet another misbehaving device or an actual attack?
>>
>
> *Ce message et toutes les pièces jointes (ci-après le "message") sont
> établis à l’intention exclusive des destinataires désignés. Il contient des
> informations confidentielles et pouvant être protégé par le secret
> professionnel. Si vous recevez ce message par erreur, merci d'en avertir
> immédiatement l'expéditeur et de détruire le message. Toute utilisation de
> ce message non conforme à sa destination, toute diffusion ou toute
> publication, totale ou partielle, est interdite, sauf autorisation expresse
> de l'émetteur*
>

Hello everyone,
I'm working for MilkyWan / AS2027 and I wanted to give you some
explanations regarding this incident. Last week-end, during an upgrade
on our network configuration, it appears that some prefixes were
announced with an incorrect AS Path. Based on our analysis, none of
these routes seem to have been announced anywhere, but to some
route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG RING), and
therefore didn't effectively end up in the DFZ.
The issue was discovered quickly (in a few minutes) and corrected right
away.
The incident is now closed on our side; please reach out to us should
you see anything proving otherwise.

We deeply apologize for that and we can confirm it was not a BGP hijack
attempt.

Wishing you a very pleasant day.

Vincent F. for Milkywan Team

Le 2023-10-22 19:02, Olivier Benghozi a écrit :
> Same stuff (with our ASN and our prefixes) detected here, coming from
> AS2027 (Milkywan), for a short time...
>
> Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il>
> a écrit :
>
>> We just had every single prefix in AS378 start being announced by
>> AS2027.
>>
>> Every announcement by AS2027 is failing RPKI yet being propagated a
>> bit.
>> Is this yet another misbehaving device or an actual attack?
>
> _Ce message et toutes les pièces jointes (ci-après le "message")
> sont établis à l’intention exclusive des destinataires désignés.
> Il contient des informations confidentielles et pouvant être
> protégé par le secret professionnel. Si vous recevez ce message par
> erreur, merci d'en avertir immédiatement l'expéditeur et de
> détruire le message. Toute utilisation de ce message non conforme à
> sa destination, toute diffusion ou toute publication, totale ou
> partielle, est interdite, sauf autorisation expresse de l'émetteur_

Thanks for the transparency, Vincent. Are you able to share how the AS-Path
became mangled to begin with? I’m assuming this was some kind of route
optimizer, but maybe something else going on?

On Mon, Oct 23, 2023 at 07:32 <vincent@milkywan.fr> wrote:

> Hello everyone,
> I'm working for MilkyWan / AS2027 and I wanted to give you some
> explanations regarding this incident. Last week-end, during an upgrade
> on our network configuration, it appears that some prefixes were
> announced with an incorrect AS Path. Based on our analysis, none of
> these routes seem to have been announced anywhere, but to some
> route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG RING), and
> therefore didn't effectively end up in the DFZ.
> The issue was discovered quickly (in a few minutes) and corrected right
> away.
> The incident is now closed on our side; please reach out to us should
> you see anything proving otherwise.
>
> We deeply apologize for that and we can confirm it was not a BGP hijack
> attempt.
>
> Wishing you a very pleasant day.
>
> Vincent F. for Milkywan Team
>
> Le 2023-10-22 19:02, Olivier Benghozi a écrit :
> > Same stuff (with our ASN and our prefixes) detected here, coming from
> > AS2027 (Milkywan), for a short time...
> >
> > Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il>
> > a écrit :
> >
> >> We just had every single prefix in AS378 start being announced by
> >> AS2027.
> >>
> >> Every announcement by AS2027 is failing RPKI yet being propagated a
> >> bit.
> >> Is this yet another misbehaving device or an actual attack?
> >
> > _Ce message et toutes les pièces jointes (ci-après le "message")
> > sont établis à l’intention exclusive des destinataires désignés.
> > Il contient des informations confidentielles et pouvant être
> > protégé par le secret professionnel. Si vous recevez ce message par
> > erreur, merci d'en avertir immédiatement l'expéditeur et de
> > détruire le message. Toute utilisation de ce message non conforme à
> > sa destination, toute diffusion ou toute publication, totale ou
> > partielle, est interdite, sauf autorisation expresse de l'émetteur_
>

Hey everybody, I run bgp.tools, (And had a extremely busy alerting
engine for a few minutes)

From what bgp.tools can see it seems like they had a private asn in
the path like so

```
2027 4220270000 6696 6939 42615 212232
```

This can be valid for a number of reasons, ( they might have been
doing some homemade BGP confederation for example ), and I assume
then that they had enabled some kind of private asn filter that had
not quite done what they expected. I think what they are expecting
was the part to look like this:

```
2027 6696 6939 42615 212232
```

However instead the private AS stripping function instead did this,
and sent it to their customers/collector feeds:

```
2027
```

This then obviously made everything look like a BGP origin hijack to
all of the route collectors and alerting systems.

It's worth noting that bgp.tools saw this from more than MilkyWan
directly, but also from what I can assume are their customers. But I
don't see any indication this faulty routing information propagated
anywhere else than that. ( To sort of backup the response that Vincent
has already provided us)

Hope this provides some interesting insight, and maybe some future heads up :)

On Sun, Oct 22, 2023 at 10:04?PM Christopher Morrow
<morrowc.lists@gmail.com> wrote:
>
> Hank, all exact match for prefix length? Or longer subnets covering the whole?
> (Is this leakage of a optimizer or possibly censorship leakage?)
>
> On Sun, Oct 22, 2023, 1:03?PM Olivier Benghozi <olivier.benghozi@wifirst.fr> wrote:
>>
>> Same stuff (with our ASN and our prefixes) detected here, coming from AS2027 (Milkywan), for a short time...
>>
>> Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il> a écrit :
>>>
>>> We just had every single prefix in AS378 start being announced by AS2027.
>>>
>>> Every announcement by AS2027 is failing RPKI yet being propagated a bit.
>>> Is this yet another misbehaving device or an actual attack?
>>
>>
>> Ce message et toutes les pièces jointes (ci-après le "message") sont établis à l’intention exclusive des destinataires désignés. Il contient des informations confidentielles et pouvant être protégé par le secret professionnel. Si vous recevez ce message par erreur, merci d'en avertir immédiatement l'expéditeur et de détruire le message. Toute utilisation de ce message non conforme à sa destination, toute diffusion ou toute publication, totale ou partielle, est interdite, sauf autorisation expresse de l'émetteur

Hello,

Because we were migrating our default table containing the DFZ into a
VRF, we had a BGP session between 2 routers terminating on one side in
the main table and on the other in the VRF. We had to remove the
no-export from our redistribution route-map because of this private eBGP
peering.

As we were concerned about reachability with transits and peerings on
both sides, we tried to activate a route-leak between the main table and
the newly installed VRF DFZ.

However, as a route-leak doesn't retain BGP attributes, routes started
to be learned in originate from the router containing the route-leak. So
there was no hijacking on the DFZ. Moreover, my route collector
listening on the DFZ did not identify any hijack during the entire
migration.

On the Transit and Peering side, we are subject to max-pref on the
provider side, and we have a route-map in prefix-list specific to our AS
+ customer community.

However, we consider Route Collectors as customers, who redistribute
prefixes greater than or equal to /24 or /48, minus bogons, and based on
no-export, we don't send our internal routes. Except that in addition to
the route leak, the no-export was removed, which let through all IGP
originate routes to our customers, and route collectors.

The problem was quickly identified, we cut the route leak and stopped
all transits and peering in main table to leave only the DFZ and work on
the end of migration having only some LNS not configured in the good
VRF, and finished the migration direction from the new VRF DFZ.

So don't rely on the Route Collector, which is updated at the whim of
the various operators, but compare what's also in the DFZ before firing
up the mailing lists. What's more, route collectors are not necessarily
configured in the same way as standard peers, depending on the operator.
This remains a mistake on our part, which has only resulted in horrors
on the monitoring route collectors and not on the DFZ routes. So Route
Collectors don't behave at all like transit or peering, given the lack
of max pref, prefix-list or RPKI.

But hey, it's quicker to send a flaming mail before typing your show ip
route, I agree ;)

My 2 cents,

Nicolas


On 23/10/2023 16:21, Tyler Conrad wrote:
> Thanks for the transparency, Vincent. Are you able to share how the
> AS-Path became mangled to begin with? I’m assuming this was some kind
> of route optimizer, but maybe something else going on?
>
> On Mon, Oct 23, 2023 at 07:32 <vincent@milkywan.fr> wrote:
>
> Hello everyone,
> I'm working for MilkyWan / AS2027 and I wanted to give you some
> explanations regarding this incident. Last week-end, during an
> upgrade
> on our network configuration, it appears that some prefixes were
> announced with an incorrect AS Path. Based on our analysis, none of
> these routes seem to have been announced anywhere, but to some
> route-collectors (RIPE RIS, BGP.Tools, Qrator, HE.net, NLNOG
> RING), and
> therefore didn't effectively end up in the DFZ.
> The issue was discovered quickly (in a few minutes) and corrected
> right
> away.
> The incident is now closed on our side; please reach out to us should
> you see anything proving otherwise.
>
> We deeply apologize for that and we can confirm it was not a BGP
> hijack
> attempt.
>
> Wishing you a very pleasant day.
>
> Vincent F. for Milkywan Team
>
> Le 2023-10-22 19:02, Olivier Benghozi a écrit :
> > Same stuff (with our ASN and our prefixes) detected here, coming
> from
> > AS2027 (Milkywan), for a short time...
> >
> > Le dim. 22 oct. 2023 à 17:18, Hank Nussbacher <hank@efes.iucc.ac.il>
> > a écrit :
> >
> >> We just had every single prefix in AS378 start being announced by
> >> AS2027.
> >>
> >> Every announcement by AS2027 is failing RPKI yet being propagated a
> >> bit.
> >> Is this yet another misbehaving device or an actual attack?
> >
> > _Ce message et toutes les pièces jointes (ci-après le "message")
> > sont établis à l’intention exclusive des destinataires désignés.
> > Il contient des informations confidentielles et pouvant être
> > protégé par le secret professionnel. Si vous recevez ce message par
> > erreur, merci d'en avertir immédiatement l'expéditeur et de
> > détruire le message. Toute utilisation de ce message non conforme à
> > sa destination, toute diffusion ou toute publication, totale ou
> > partielle, est interdite, sauf autorisation expresse de l'émetteur_
>