Mailing List Archive

anyone use fbtracert successfully?
The tool fbtracert (http://github.com/facebookarchive/fbtracert) was mentioned here recently as a way to get visibility into multi-pathing.
Has anyone here ever used this tool successfully?

Supposedly Facebook uses this tool internally, but… that doesn’t help much.

I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian; OpenBSD), and versions of Go (v1.10 through v1.16), in three very different environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve yet to see it produce any meaningful output – each run/iteration/thread only detects one, single, hop out of the entire chain of routers, making it less than useful. Granted, that’s not a full regression test by any means, but if anyone here has ever used it successfully, could you please let me know what sort of environment you ran it in/on?

Thanks,
-Adam

Adam Thompson
Consultant, Infrastructure Services
[1593169877849]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>
Re: anyone use fbtracert successfully? [ In reply to ]
I have used it successfully in a test environment that I was using ECMP in.
Most of the public networks that I've worked with don't use ECMP as often
as other methods for steering traffic (LAGs, BGP MEDs, etc).

What I have seen it fantastically useful for was troubleshooting a transit
provider, or for when they were congested or had a flapping core link.
Granted I *think *it's still subject to ICMP deprioritization (most SP's
use it prodigiously), and most MPLS cores don't decrement TTL, but it was
still useful to be able to show them "no, at this IP, I *always* drop
traffic, when..."

- Thomas Scott | mr.thomas.scott@gmail.com


On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson <athompson@merlin.mb.ca>
wrote:

> The tool fbtracert (http://github.com/facebookarchive/fbtracert) was
> mentioned here recently as a way to get visibility into multi-pathing.
>
> Has anyone here ever used this tool successfully?
>
>
>
> Supposedly Facebook uses this tool internally, but… that doesn’t help much.
>
>
>
> I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian;
> OpenBSD), and versions of Go (v1.10 through v1.16), in three very different
> environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve
> yet to see it produce any meaningful output – each run/iteration/thread
> only detects one, single, hop out of the entire chain of routers, making it
> less than useful. Granted, that’s not a full regression test by any means,
> but if anyone here has ever used it successfully, could you please let me
> know what sort of environment you ran it in/on?
>
>
>
> Thanks,
>
> -Adam
>
>
>
> *Adam Thompson*
> Consultant, Infrastructure Services
> [image: 1593169877849]
> 100 - 135 Innovation Drive
> Winnipeg, MB, R3T 6A8
> (204) 977-6824 or 1-800-430-6404 (MB only)
> athompson@merlin.mb.ca
> www.merlin.mb.ca
>
>
>
Re: anyone use fbtracert successfully? [ In reply to ]
Ha, my apologies, I thought I was writing this for a Linux User Group, not
a NOG. Ignore my simplistic explanations.
- Thomas Scott | mr.thomas.scott@gmail.com


On Wed, Nov 24, 2021 at 12:47 PM Thomas Scott <mr.thomas.scott@gmail.com>
wrote:

> I have used it successfully in a test environment that I was using ECMP
> in. Most of the public networks that I've worked with don't use ECMP as
> often as other methods for steering traffic (LAGs, BGP MEDs, etc).
>
> What I have seen it fantastically useful for was troubleshooting a transit
> provider, or for when they were congested or had a flapping core link.
> Granted I *think *it's still subject to ICMP deprioritization (most SP's
> use it prodigiously), and most MPLS cores don't decrement TTL, but it was
> still useful to be able to show them "no, at this IP, I *always* drop
> traffic, when..."
>
> - Thomas Scott | mr.thomas.scott@gmail.com
>
>
> On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson <athompson@merlin.mb.ca>
> wrote:
>
>> The tool fbtracert (http://github.com/facebookarchive/fbtracert) was
>> mentioned here recently as a way to get visibility into multi-pathing.
>>
>> Has anyone here ever used this tool successfully?
>>
>>
>>
>> Supposedly Facebook uses this tool internally, but… that doesn’t help
>> much.
>>
>>
>>
>> I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian;
>> OpenBSD), and versions of Go (v1.10 through v1.16), in three very different
>> environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve
>> yet to see it produce any meaningful output – each run/iteration/thread
>> only detects one, single, hop out of the entire chain of routers, making it
>> less than useful. Granted, that’s not a full regression test by any means,
>> but if anyone here has ever used it successfully, could you please let me
>> know what sort of environment you ran it in/on?
>>
>>
>>
>> Thanks,
>>
>> -Adam
>>
>>
>>
>> *Adam Thompson*
>> Consultant, Infrastructure Services
>> [image: 1593169877849]
>> 100 - 135 Innovation Drive
>> Winnipeg, MB, R3T 6A8
>> (204) 977-6824 or 1-800-430-6404 (MB only)
>> athompson@merlin.mb.ca
>> www.merlin.mb.ca
>>
>>
>>
>
Re: anyone use fbtracert successfully? [ In reply to ]
What about some other options?

https://paris-traceroute.net/
https://dublin-traceroute.net/
https://github.com/rucarrol/traceflow

--
Hugo Slabbert


On Wed, Nov 24, 2021 at 9:54 AM Thomas Scott <mr.thomas.scott@gmail.com>
wrote:

> Ha, my apologies, I thought I was writing this for a Linux User Group, not
> a NOG. Ignore my simplistic explanations.
> - Thomas Scott | mr.thomas.scott@gmail.com
>
>
> On Wed, Nov 24, 2021 at 12:47 PM Thomas Scott <mr.thomas.scott@gmail.com>
> wrote:
>
>> I have used it successfully in a test environment that I was using ECMP
>> in. Most of the public networks that I've worked with don't use ECMP as
>> often as other methods for steering traffic (LAGs, BGP MEDs, etc).
>>
>> What I have seen it fantastically useful for was troubleshooting a
>> transit provider, or for when they were congested or had a flapping core
>> link. Granted I *think *it's still subject to ICMP deprioritization
>> (most SP's use it prodigiously), and most MPLS cores don't decrement TTL,
>> but it was still useful to be able to show them "no, at this IP, I
>> *always* drop traffic, when..."
>>
>> - Thomas Scott | mr.thomas.scott@gmail.com
>>
>>
>> On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson <athompson@merlin.mb.ca>
>> wrote:
>>
>>> The tool fbtracert (http://github.com/facebookarchive/fbtracert) was
>>> mentioned here recently as a way to get visibility into multi-pathing.
>>>
>>> Has anyone here ever used this tool successfully?
>>>
>>>
>>>
>>> Supposedly Facebook uses this tool internally, but… that doesn’t help
>>> much.
>>>
>>>
>>>
>>> I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian;
>>> OpenBSD), and versions of Go (v1.10 through v1.16), in three very different
>>> environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve
>>> yet to see it produce any meaningful output – each run/iteration/thread
>>> only detects one, single, hop out of the entire chain of routers, making it
>>> less than useful. Granted, that’s not a full regression test by any means,
>>> but if anyone here has ever used it successfully, could you please let me
>>> know what sort of environment you ran it in/on?
>>>
>>>
>>>
>>> Thanks,
>>>
>>> -Adam
>>>
>>>
>>>
>>> *Adam Thompson*
>>> Consultant, Infrastructure Services
>>> [image: 1593169877849]
>>> 100 - 135 Innovation Drive
>>> Winnipeg, MB, R3T 6A8
>>> (204) 977-6824 or 1-800-430-6404 (MB only)
>>> athompson@merlin.mb.ca
>>> www.merlin.mb.ca
>>>
>>>
>>>
>>
RE: anyone use fbtracert successfully? [ In reply to ]
Thank you!! Some of those tools are proving much more useful for me than fbtracert. (In particular, traceflow has been updated recently enough that it “just works” in common environments that have Python3. And while it may not be perfect, it’s good enough to show what I need.)
-Adam
(who apparently has lost the skills needed to Google usefully, in his decrepitude)

Adam Thompson
Consultant, Infrastructure Services
[1593169877849]
100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>

From: Hugo Slabbert <hugo@slabnet.com>
Sent: Thursday, November 25, 2021 10:39 AM
To: Thomas Scott <mr.thomas.scott@gmail.com>
Cc: Adam Thompson <athompson@merlin.mb.ca>; nanog <nanog@nanog.org>
Subject: Re: anyone use fbtracert successfully?

What about some other options?

https://paris-traceroute.net/
https://dublin-traceroute.net/
https://github.com/rucarrol/traceflow

--
Hugo Slabbert


On Wed, Nov 24, 2021 at 9:54 AM Thomas Scott <mr.thomas.scott@gmail.com<mailto:mr.thomas.scott@gmail.com>> wrote:
Ha, my apologies, I thought I was writing this for a Linux User Group, not a NOG. Ignore my simplistic explanations.
- Thomas Scott | mr.thomas.scott@gmail.com<mailto:mr.thomas.scott@gmail.com>


On Wed, Nov 24, 2021 at 12:47 PM Thomas Scott <mr.thomas.scott@gmail.com<mailto:mr.thomas.scott@gmail.com>> wrote:
I have used it successfully in a test environment that I was using ECMP in. Most of the public networks that I've worked with don't use ECMP as often as other methods for steering traffic (LAGs, BGP MEDs, etc).

What I have seen it fantastically useful for was troubleshooting a transit provider, or for when they were congested or had a flapping core link. Granted I think it's still subject to ICMP deprioritization (most SP's use it prodigiously), and most MPLS cores don't decrement TTL, but it was still useful to be able to show them "no, at this IP, I always drop traffic, when..."

- Thomas Scott | mr.thomas.scott@gmail.com<mailto:mr.thomas.scott@gmail.com>


On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson <athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>> wrote:
The tool fbtracert (http://github.com/facebookarchive/fbtracert) was mentioned here recently as a way to get visibility into multi-pathing.
Has anyone here ever used this tool successfully?

Supposedly Facebook uses this tool internally, but… that doesn’t help much.

I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian; OpenBSD), and versions of Go (v1.10 through v1.16), in three very different environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve yet to see it produce any meaningful output – each run/iteration/thread only detects one, single, hop out of the entire chain of routers, making it less than useful. Granted, that’s not a full regression test by any means, but if anyone here has ever used it successfully, could you please let me know what sort of environment you ran it in/on?

Thanks,
-Adam

Adam Thompson
Consultant, Infrastructure Services

100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca<mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca<http://www.merlin.mb.ca/>
RE: anyone use fbtracert successfully? [ In reply to ]
smokeping in master slave mode. A bit old school, but maybe still worth a try.



https://oss.oetiker.ch/smokeping/doc/smokeping_master_slave.en.html



Jean



From: NANOG <nanog-bounces+jean=ddostest.me@nanog.org> On Behalf Of Adam Thompson
Sent: November 25, 2021 1:31 PM
To: Hugo Slabbert <hugo@slabnet.com>; Thomas Scott <mr.thomas.scott@gmail.com>
Cc: nanog <nanog@nanog.org>
Subject: RE: anyone use fbtracert successfully?



Thank you!! Some of those tools are proving much more useful for me than fbtracert. (In particular, traceflow has been updated recently enough that it “just works” in common environments that have Python3. And while it may not be perfect, it’s good enough to show what I need.)

-Adam

(who apparently has lost the skills needed to Google usefully, in his decrepitude)



Adam Thompson
Consultant, Infrastructure Services

100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca <mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca <http://www.merlin.mb.ca/>



From: Hugo Slabbert <hugo@slabnet.com <mailto:hugo@slabnet.com> >
Sent: Thursday, November 25, 2021 10:39 AM
To: Thomas Scott <mr.thomas.scott@gmail.com <mailto:mr.thomas.scott@gmail.com> >
Cc: Adam Thompson <athompson@merlin.mb.ca <mailto:athompson@merlin.mb.ca> >; nanog <nanog@nanog.org <mailto:nanog@nanog.org> >
Subject: Re: anyone use fbtracert successfully?



What about some other options?



https://paris-traceroute.net/

https://dublin-traceroute.net/

https://github.com/rucarrol/traceflow



--

Hugo Slabbert





On Wed, Nov 24, 2021 at 9:54 AM Thomas Scott <mr.thomas.scott@gmail.com <mailto:mr.thomas.scott@gmail.com> > wrote:

Ha, my apologies, I thought I was writing this for a Linux User Group, not a NOG. Ignore my simplistic explanations.

- Thomas Scott | mr.thomas.scott@gmail.com <mailto:mr.thomas.scott@gmail.com>





On Wed, Nov 24, 2021 at 12:47 PM Thomas Scott <mr.thomas.scott@gmail.com <mailto:mr.thomas.scott@gmail.com> > wrote:

I have used it successfully in a test environment that I was using ECMP in. Most of the public networks that I've worked with don't use ECMP as often as other methods for steering traffic (LAGs, BGP MEDs, etc).



What I have seen it fantastically useful for was troubleshooting a transit provider, or for when they were congested or had a flapping core link. Granted I think it's still subject to ICMP deprioritization (most SP's use it prodigiously), and most MPLS cores don't decrement TTL, but it was still useful to be able to show them "no, at this IP, I always drop traffic, when..."



- Thomas Scott | mr.thomas.scott@gmail.com <mailto:mr.thomas.scott@gmail.com>





On Wed, Nov 24, 2021 at 12:23 PM Adam Thompson <athompson@merlin.mb.ca <mailto:athompson@merlin.mb.ca> > wrote:

The tool fbtracert (http://github.com/facebookarchive/fbtracert) was mentioned here recently as a way to get visibility into multi-pathing.

Has anyone here ever used this tool successfully?



Supposedly Facebook uses this tool internally, but… that doesn’t help much.



I’ve tried it on 4 different platforms/OSes (WSL Ubuntu; RedHat; Debian; OpenBSD), and versions of Go (v1.10 through v1.16), in three very different environments (on-prem public IP; on-prem NAT’d; cloud public IP), and I’ve yet to see it produce any meaningful output – each run/iteration/thread only detects one, single, hop out of the entire chain of routers, making it less than useful. Granted, that’s not a full regression test by any means, but if anyone here has ever used it successfully, could you please let me know what sort of environment you ran it in/on?



Thanks,

-Adam



Adam Thompson
Consultant, Infrastructure Services

100 - 135 Innovation Drive
Winnipeg, MB, R3T 6A8
(204) 977-6824 or 1-800-430-6404 (MB only)
athompson@merlin.mb.ca <mailto:athompson@merlin.mb.ca>
www.merlin.mb.ca <http://www.merlin.mb.ca/>