Mailing List Archive

n2disk automated capture archive workflow
Hi list,

I'm evaluating n2disk for a specific use-case but I'm not clear how to map
my workflow to its mental model so I wanted to ask your advice.

my desired workflow is:

* I start capturing with n2disk. I'm using the -I option to make
on-the-fly index
* when a certain session event occurs in my application I want to archive
the pcap for that session (in the event I will know which remote UDP
ip/port were used in the session)

I have had success performing the above steps manuall/independently but my
attempts to automate them have become protracted and I wanted to ask if
perhaps I'm missing something simpler way to do this.

the main issue I have is that when my application event occurs (lets say I
know that a given application session $sessionid just ended involving
remote $ip and $port) then I'd like the knowledge of that application
session ending to trigger a npcapextract for $ip and $port to archive the
session under a filename of $sessionid. I have all these variables and
have a script which is ready to automatically run the extract...

the problem I have is:
* n2disk has not flushed the data to disk yet and so I can't run the
extract yet. so how can I know when it is safe to run the extract?
* I read in release notes that it was possible to us " kill -USR1 to close
and flush the current pcap in order to make live traffic immediately
available" which works but I notice every time I call it it generates a new
index file. Which then leads me to the question of: how do I know which
index file to run npcapextract against? e.g. if the latest index was 1.idx
and I do a kill -USR do I have to guess that my application events would be
found in 1.idx / 1.pcap or is there a another way to do this?

* looking at all this another way. I'd be happy to defer the npcapextract
until the data is naturally flushed to disk. but this leads me to 2
questions:
- how can I know when all the relevant data is flushed to disk so I can
take action on the npcapextract? e.g. is there some concept of a
hook/trigger I can call when pcap / index data is flushed to disk?
- how can I know which index file is the "current" one

I am new to n2disk so I am likely just coming at this with some flawed
mental model. I hope at least my question articulates what I'm trying to
accomplish.

I'd be very grateful for any input on how I can accomplish my use-case.

Thanks,
RD
Re: n2disk automated capture archive workflow [ In reply to ]
Hi Raoul
please read my comments inline.

> On 30 Mar 2018, at 20:07, Raoul Duke <rduke496@gmail.com> wrote:
>
> Hi list,
>
> I'm evaluating n2disk for a specific use-case but I'm not clear how to map my workflow to its mental model so I wanted to ask your advice.
>
> my desired workflow is:
>
> * I start capturing with n2disk. I'm using the -I option to make on-the-fly index
> * when a certain session event occurs in my application I want to archive the pcap for that session (in the event I will know which remote UDP ip/port were used in the session)
>
> I have had success performing the above steps manuall/independently but my attempts to automate them have become protracted and I wanted to ask if perhaps I'm missing something simpler way to do this.
>
> the main issue I have is that when my application event occurs (lets say I know that a given application session $sessionid just ended involving remote $ip and $port) then I'd like the knowledge of that application session ending to trigger a npcapextract for $ip and $port to archive the session under a filename of $sessionid. I have all these variables and have a script which is ready to automatically run the extract...
>
> the problem I have is:
> * n2disk has not flushed the data to disk yet and so I can't run the extract yet. so how can I know when it is safe to run the extract?

You can send a SIGUSR1 to the n2disk process to flush the current PCAP to disk, if the disk is fast enough you should be able to extract traffic after a couple of seconds at most.

> * I read in release notes that it was possible to us " kill -USR1 to close and flush the current pcap in order to make live traffic immediately available" which works but I notice every time I call it it generates a new index file. Which then leads me to the question of: how do I know which index file to run npcapextract against? e.g. if the latest index was 1.idx and I do a kill -USR do I have to guess that my application events would be found in 1.idx / 1.pcap or is there a another way to do this?

I recommend you to enable the timeline, and just specify the time interval in npcapextract, using the timeline as data source instead of the specific pcap/index.

> * looking at all this another way. I'd be happy to defer the npcapextract until the data is naturally flushed to disk. but this leads me to 2 questions:
> - how can I know when all the relevant data is flushed to disk so I can take action on the npcapextract? e.g. is there some concept of a hook/trigger I can call when pcap / index data is flushed to disk?

You probably need to know what is the timestamp of the last packet dumped to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk stats>. If this works for you we can add it to the features list.

> - how can I know which index file is the "current” one

You can check the time of last index on disk, however as I said you probably do not need this if you use the timeline and we export the time of last packet dumped to disk in the stats.

> I am new to n2disk so I am likely just coming at this with some flawed mental model. I hope at least my question articulates what I'm trying to accomplish.
>
> I'd be very grateful for any input on how I can accomplish my use-case.
>
> Thanks,
> RD

Best Regards
Alfredo

>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: n2disk automated capture archive workflow [ In reply to ]
Hi Raoul
please read my comments inline.

> On 30 Mar 2018, at 20:07, Raoul Duke <rduke496@gmail.com> wrote:
>
> Hi list,
>
> I'm evaluating n2disk for a specific use-case but I'm not clear how to map my workflow to its mental model so I wanted to ask your advice.
>
> my desired workflow is:
>
> * I start capturing with n2disk. I'm using the -I option to make on-the-fly index
> * when a certain session event occurs in my application I want to archive the pcap for that session (in the event I will know which remote UDP ip/port were used in the session)
>
> I have had success performing the above steps manuall/independently but my attempts to automate them have become protracted and I wanted to ask if perhaps I'm missing something simpler way to do this.
>
> the main issue I have is that when my application event occurs (lets say I know that a given application session $sessionid just ended involving remote $ip and $port) then I'd like the knowledge of that application session ending to trigger a npcapextract for $ip and $port to archive the session under a filename of $sessionid. I have all these variables and have a script which is ready to automatically run the extract...
>
> the problem I have is:
> * n2disk has not flushed the data to disk yet and so I can't run the extract yet. so how can I know when it is safe to run the extract?

You can send a SIGUSR1 to the n2disk process to flush the current PCAP to disk, if the disk is fast enough you should be able to extract traffic after a couple of seconds at most.

> * I read in release notes that it was possible to us " kill -USR1 to close and flush the current pcap in order to make live traffic immediately available" which works but I notice every time I call it it generates a new index file. Which then leads me to the question of: how do I know which index file to run npcapextract against? e.g. if the latest index was 1.idx and I do a kill -USR do I have to guess that my application events would be found in 1.idx / 1.pcap or is there a another way to do this?

I recommend you to enable the timeline, and just specify the time interval in npcapextract, using the timeline as data source instead of the specific pcap/index.

> * looking at all this another way. I'd be happy to defer the npcapextract until the data is naturally flushed to disk. but this leads me to 2 questions:
> - how can I know when all the relevant data is flushed to disk so I can take action on the npcapextract? e.g. is there some concept of a hook/trigger I can call when pcap / index data is flushed to disk?

You probably need to know what is the timestamp of the last packet dumped to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk stats>. If this works for you we can add it to the features list.

> - how can I know which index file is the "current” one

You can check the time of last index on disk, however as I said you probably do not need this if you use the timeline and we export the time of last packet dumped to disk in the stats.

> I am new to n2disk so I am likely just coming at this with some flawed mental model. I hope at least my question articulates what I'm trying to accomplish.
>
> I'd be very grateful for any input on how I can accomplish my use-case.
>
> Thanks,
> RD

Best Regards
Alfredo

>
>
>
>
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: n2disk automated capture archive workflow [ In reply to ]
Hi Alfredo,


On Fri, Mar 30, 2018 at 7:20 PM, Alfredo Cardigliano <cardigliano@ntop.org>
wrote:

>
>
> * I read in release notes that it was possible to us " kill -USR1 to
> close and flush the current pcap in order to make live traffic immediately
> available" which works but I notice every time I call it it generates a
> new index file. Which then leads me to the question of: how do I know
> which index file to run npcapextract against? e.g. if the latest index
> was 1.idx and I do a kill -USR do I have to guess that my application
> events would be found in 1.idx / 1.pcap or is there a another way to do
> this?
>
>
> I recommend you to enable the timeline, and just specify the time interval
> in npcapextract, using the timeline as data source instead of the specific
> pcap/index.
>

Thanks for this useful info. I will experiment with timeline.


>
> * looking at all this another way. I'd be happy to defer the npcapextract
> until the data is naturally flushed to disk. but this leads me to 2
> questions:
> - how can I know when all the relevant data is flushed to disk so I can
> take action on the npcapextract? e.g. is there some concept of a
> hook/trigger I can call when pcap / index data is flushed to disk?
>
>
> You probably need to know what is the timestamp of the last packet dumped
> to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk
> stats>. If this works for you we can add it to the features list.
>

this seems like a good feature to have in the general case could be
potentially used in my case.

another idea I had was: is there an option to control the flush frequency?
e.g. lets say I captured 100 packets but they have not yet flushed to disk
and no traffic happens for (say) 10 more minutes. would there still be no
flush to disk during that 10 minutes? i.e. would the flush only happen
when sufficient traffic has occured to fill up the memory buffer or is
there a way to say "always flush to disk every X seconds". in that way I
could defer the npcapextract for X seconds after I know the application
session has ended and could guarantee the packets would be flushed to disk
by then.

Thanks so much for such a quick and helpful response.

RD
Re: n2disk automated capture archive workflow [ In reply to ]
Hi Alfredo,


On Fri, Mar 30, 2018 at 7:20 PM, Alfredo Cardigliano <cardigliano@ntop.org>
wrote:

>
>
> * I read in release notes that it was possible to us " kill -USR1 to
> close and flush the current pcap in order to make live traffic immediately
> available" which works but I notice every time I call it it generates a
> new index file. Which then leads me to the question of: how do I know
> which index file to run npcapextract against? e.g. if the latest index
> was 1.idx and I do a kill -USR do I have to guess that my application
> events would be found in 1.idx / 1.pcap or is there a another way to do
> this?
>
>
> I recommend you to enable the timeline, and just specify the time interval
> in npcapextract, using the timeline as data source instead of the specific
> pcap/index.
>

Thanks for this useful info. I will experiment with timeline.


>
> * looking at all this another way. I'd be happy to defer the npcapextract
> until the data is naturally flushed to disk. but this leads me to 2
> questions:
> - how can I know when all the relevant data is flushed to disk so I can
> take action on the npcapextract? e.g. is there some concept of a
> hook/trigger I can call when pcap / index data is flushed to disk?
>
>
> You probably need to know what is the timestamp of the last packet dumped
> to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk
> stats>. If this works for you we can add it to the features list.
>

this seems like a good feature to have in the general case could be
potentially used in my case.

another idea I had was: is there an option to control the flush frequency?
e.g. lets say I captured 100 packets but they have not yet flushed to disk
and no traffic happens for (say) 10 more minutes. would there still be no
flush to disk during that 10 minutes? i.e. would the flush only happen
when sufficient traffic has occured to fill up the memory buffer or is
there a way to say "always flush to disk every X seconds". in that way I
could defer the npcapextract for X seconds after I know the application
session has ended and could guarantee the packets would be flushed to disk
by then.

Thanks so much for such a quick and helpful response.

RD
Re: n2disk automated capture archive workflow [ In reply to ]
What you need is [--max-file-duration|-t] <seconds>, however please note that
this is automatically set to 10 minutes when enabling the timeline (-A)

Alfredo

> On 30 Mar 2018, at 23:59, Raoul Duke <rduke496@gmail.com> wrote:
>
>
> Hi Alfredo,
>
>
> On Fri, Mar 30, 2018 at 7:20 PM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
>
>
>> * I read in release notes that it was possible to us " kill -USR1 to close and flush the current pcap in order to make live traffic immediately available" which works but I notice every time I call it it generates a new index file. Which then leads me to the question of: how do I know which index file to run npcapextract against? e.g. if the latest index was 1.idx and I do a kill -USR do I have to guess that my application events would be found in 1.idx / 1.pcap or is there a another way to do this?
>
> I recommend you to enable the timeline, and just specify the time interval in npcapextract, using the timeline as data source instead of the specific pcap/index.
>
> Thanks for this useful info. I will experiment with timeline.
>
>
>> * looking at all this another way. I'd be happy to defer the npcapextract until the data is naturally flushed to disk. but this leads me to 2 questions:
>> - how can I know when all the relevant data is flushed to disk so I can take action on the npcapextract? e.g. is there some concept of a hook/trigger I can call when pcap / index data is flushed to disk?
>
> You probably need to know what is the timestamp of the last packet dumped to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk stats>. If this works for you we can add it to the features list.
>
> this seems like a good feature to have in the general case could be potentially used in my case.
>
> another idea I had was: is there an option to control the flush frequency? e.g. lets say I captured 100 packets but they have not yet flushed to disk and no traffic happens for (say) 10 more minutes. would there still be no flush to disk during that 10 minutes? i.e. would the flush only happen when sufficient traffic has occured to fill up the memory buffer or is there a way to say "always flush to disk every X seconds". in that way I could defer the npcapextract for X seconds after I know the application session has ended and could guarantee the packets would be flushed to disk by then.
>
> Thanks so much for such a quick and helpful response.
>
> RD
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop
Re: n2disk automated capture archive workflow [ In reply to ]
What you need is [--max-file-duration|-t] <seconds>, however please note that
this is automatically set to 10 minutes when enabling the timeline (-A)

Alfredo

> On 30 Mar 2018, at 23:59, Raoul Duke <rduke496@gmail.com> wrote:
>
>
> Hi Alfredo,
>
>
> On Fri, Mar 30, 2018 at 7:20 PM, Alfredo Cardigliano <cardigliano@ntop.org <mailto:cardigliano@ntop.org>> wrote:
>
>
>> * I read in release notes that it was possible to us " kill -USR1 to close and flush the current pcap in order to make live traffic immediately available" which works but I notice every time I call it it generates a new index file. Which then leads me to the question of: how do I know which index file to run npcapextract against? e.g. if the latest index was 1.idx and I do a kill -USR do I have to guess that my application events would be found in 1.idx / 1.pcap or is there a another way to do this?
>
> I recommend you to enable the timeline, and just specify the time interval in npcapextract, using the timeline as data source instead of the specific pcap/index.
>
> Thanks for this useful info. I will experiment with timeline.
>
>
>> * looking at all this another way. I'd be happy to defer the npcapextract until the data is naturally flushed to disk. but this leads me to 2 questions:
>> - how can I know when all the relevant data is flushed to disk so I can take action on the npcapextract? e.g. is there some concept of a hook/trigger I can call when pcap / index data is flushed to disk?
>
> You probably need to know what is the timestamp of the last packet dumped to disk, maybe we can write it under /proc/net/pf_ring/stats/<n2disk stats>. If this works for you we can add it to the features list.
>
> this seems like a good feature to have in the general case could be potentially used in my case.
>
> another idea I had was: is there an option to control the flush frequency? e.g. lets say I captured 100 packets but they have not yet flushed to disk and no traffic happens for (say) 10 more minutes. would there still be no flush to disk during that 10 minutes? i.e. would the flush only happen when sufficient traffic has occured to fill up the memory buffer or is there a way to say "always flush to disk every X seconds". in that way I could defer the npcapextract for X seconds after I know the application session has ended and could guarantee the packets would be flushed to disk by then.
>
> Thanks so much for such a quick and helpful response.
>
> RD
> _______________________________________________
> Ntop mailing list
> Ntop@listgateway.unipi.it
> http://listgateway.unipi.it/mailman/listinfo/ntop