Mailing List Archive: [PATCHv2 00/25] perf tool: Add support for multiple data file storage

[PATCHv2 00/25] perf tool: Add support for multiple data file storage

Sep 1, 2013, 3:36 AM

Post #1 of 15 (1759 views)

hi,
sending the support for multiple file storage. Initial
RFC is here:
http://marc.info/?l=linux-kernel&m=137408381902423&w=2

v2 changes:
- reworked perf mmap size setup to be able to get
the mmap size value easily later
- added perf.data read/write test for v2 and v3
for both endianity
- added record '-M time' support

It's reachable here:
git://git.kernel.org/pub/scm/linux/kernel/git/jolsa/perf.git
perf/core_file

thanks,
jirka

Signed-off-by: Jiri Olsa <jolsa@redhat.com>
Cc: Corey Ashford <cjashfor@linux.vnet.ibm.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
---
Jiri Olsa (25):
perf tools: Check mmap pages value early
perf tools: Add possibility to specify mmap size
perf tools: Introduce perf_evlist__new_default function
perf tools: Adding throttle event data struct support
perf tests: Add simple session read/write test
perf tests: Add session reading test for little endian perf data
perf tests: Add session reading test for big endian perf data
perf doc: Add perf data file documentation
perf tools: Introduce perf data file version CHECK macro
perf tools: Introduce swap_features function
perf tools: Introduce swap_header function
perf tools: Separate version 2 specific perf data header bits
perf tools: Using evlist as a holder for event_desc feature
perf tools: Introduce perf.data version 3 format
perf tools: Add perf data version 3 header swap
perf tools: Add perf data version 3 header read
perf tools: Add perf.data version 3 header write
perf tools: Get rid of post_processing_offset in record command
perf tools: Move synthesizing into single function
perf tools: Add perf_data_file__open interface to data object
perf tools: Separating data file properties from session
perf tests: Add session reading test for little endian perf data v3
perf tests: Add session reading test for big endian perf data v3
perf tools: Add multi file '-M' option for record command
perf tools: Have the process properly sythesized in subsequent data files

tools/perf/Documentation/perf-data-file-v2.txt | 265 ++++++++++++++++++++++++++++++++++
tools/perf/Documentation/perf-data-file-v3.txt | 63 ++++++++
tools/perf/Documentation/perf-kvm.txt | 4 +-
tools/perf/Documentation/perf-record.txt | 18 ++-
tools/perf/Documentation/perf-top.txt | 4 +-
tools/perf/Documentation/perf-trace.txt | 4 +-
tools/perf/Makefile | 6 +
tools/perf/builtin-annotate.c | 11 +-
tools/perf/builtin-buildid-cache.c | 8 +-
tools/perf/builtin-buildid-list.c | 11 +-
tools/perf/builtin-diff.c | 19 ++-
tools/perf/builtin-evlist.c | 7 +-
tools/perf/builtin-inject.c | 10 +-
tools/perf/builtin-kmem.c | 7 +-
tools/perf/builtin-kvm.c | 18 ++-
tools/perf/builtin-lock.c | 8 +-
tools/perf/builtin-mem.c | 9 +-
tools/perf/builtin-record.c | 484 ++++++++++++++++++++++++++++++++++++++++++++------------------
tools/perf/builtin-report.c | 18 ++-
tools/perf/builtin-sched.c | 6 +-
tools/perf/builtin-script.c | 17 ++-
tools/perf/builtin-timechart.c | 10 +-
tools/perf/builtin-top.c | 12 +-
tools/perf/builtin-trace.c | 13 +-
tools/perf/perf.h | 15 +-
tools/perf/tests/builtin-test.c | 4 +
tools/perf/tests/perf-record.c | 12 +-
tools/perf/tests/perf.data.v2.be.h | 265 ++++++++++++++++++++++++++++++++++
tools/perf/tests/perf.data.v2.le.h | 219 ++++++++++++++++++++++++++++
tools/perf/tests/perf.data.v3.be.h | 258 +++++++++++++++++++++++++++++++++
tools/perf/tests/perf.data.v3.le.h | 212 +++++++++++++++++++++++++++
tools/perf/tests/session-simple.c | 658 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
tools/perf/tests/task-exit.c | 14 +-
tools/perf/tests/tests.h | 1 +
tools/perf/util/data.c | 120 ++++++++++++++++
tools/perf/util/data.h | 48 +++++++
tools/perf/util/event.h | 7 +
tools/perf/util/evlist.c | 78 ++++++++--
tools/perf/util/evlist.h | 9 +-
tools/perf/util/evsel.c | 24 ++++
tools/perf/util/header.c | 446 +++++++++++++++++++++++++++++++++------------------------
tools/perf/util/header.h | 31 +++-
tools/perf/util/python.c | 7 -
tools/perf/util/session.c | 138 ++++++++----------
tools/perf/util/session.h | 11 +-
tools/perf/util/util.c | 25 ++++
tools/perf/util/util.h | 14 ++
47 files changed, 3137 insertions(+), 511 deletions(-)
create mode 100644 tools/perf/Documentation/perf-data-file-v2.txt
create mode 100644 tools/perf/Documentation/perf-data-file-v3.txt
create mode 100644 tools/perf/tests/perf.data.v2.be.h
create mode 100644 tools/perf/tests/perf.data.v2.le.h
create mode 100644 tools/perf/tests/perf.data.v3.be.h
create mode 100644 tools/perf/tests/perf.data.v3.le.h
create mode 100644 tools/perf/tests/session-simple.c
create mode 100644 tools/perf/util/data.c
create mode 100644 tools/perf/util/data.h
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

ak at linux

Sep 1, 2013, 7:42 PM

Post #2 of 15 (1738 views)

Permalink

On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
> hi,
> sending the support for multiple file storage. Initial
> RFC is here:
> http://marc.info/?l=linux-kernel&m=137408381902423&w=2

Just a general comment. If you repost please always keep the overview
description/motivation in 0/0.

-Andi

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

peterz at infradead

Sep 9, 2013, 4:17 AM

Post #3 of 15 (1733 views)

Permalink

On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
> hi,
> sending the support for multiple file storage. Initial
> RFC is here:
> http://marc.info/?l=linux-kernel&m=137408381902423&w=2
>
> v2 changes:
> - reworked perf mmap size setup to be able to get
> the mmap size value easily later
> - added perf.data read/write test for v2 and v3
> for both endianity
> - added record '-M time' support

So this 0/n post seems to have forgotten to list the rationale for doing
all this..

The only reason I wanted this is so that each thread can write its own
data. The current one file thing is an immense bottle-neck for big
machines.

However your earlier patches didn't do it because of that -- I forgot
why they did it and you're now not telling.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

jolsa at redhat

Sep 9, 2013, 4:36 AM

Post #4 of 15 (1730 views)

Permalink

On Mon, Sep 09, 2013 at 01:17:49PM +0200, Peter Zijlstra wrote:
> On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
> > hi,
> > sending the support for multiple file storage. Initial
> > RFC is here:
> > http://marc.info/?l=linux-kernel&m=137408381902423&w=2
> >
> > v2 changes:
> > - reworked perf mmap size setup to be able to get
> > the mmap size value easily later
> > - added perf.data read/write test for v2 and v3
> > for both endianity
> > - added record '-M time' support
>
> So this 0/n post seems to have forgotten to list the rationale for doing
> all this..

here it is:

This patchset is actually doing 3 essential things:
1) adds new perf data format version (v3)
2) adds class/object to handle perf data file
3) adds record '-M size' option

ad 1) Along the way I realized it'd be much easier to have the
format simplified. I ended up with just 'DATA' and 'FEATURES'
sections making the file format more generic.

The easier/generic format will be for example useful for the
cpu hotplug notification (and maybe even for ftrace data files
features?)

The version 2 should be still readable by perf, while it
produces data in version 3 from this patchset onwards.

ad 2) I needed to unify perf data file handling to make the -M
implementation clear.

>
> The only reason I wanted this is so that each thread can write its own
> data. The current one file thing is an immense bottle-neck for big
> machines.
>

I haven't considered the thread based storage yet, but I
think having the simplified format and centralized file
handling will only help.

jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

peterz at infradead

Sep 9, 2013, 4:55 AM

Post #5 of 15 (1730 views)

Permalink

On Mon, Sep 09, 2013 at 01:36:41PM +0200, Jiri Olsa wrote:
> On Mon, Sep 09, 2013 at 01:17:49PM +0200, Peter Zijlstra wrote:
> > On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
> This patchset is actually doing 3 essential things:
> 1) adds new perf data format version (v3)
> 2) adds class/object to handle perf data file
> 3) adds record '-M size' option

So the -M thing is where the multiple files stuff comes from, right?
What's the purpose of this? Why would I care how big these files get,
surely filesystems handle 4g+ files these days.

> > The only reason I wanted this is so that each thread can write its own
> > data. The current one file thing is an immense bottle-neck for big
> > machines.
> >
>
> I haven't considered the thread based storage yet, but I
> think having the simplified format and centralized file
> handling will only help.

Please do consider it now, it'd suck to have to change the file format
thing yet again.

With both per-thread and size limited your filename generation needs
some extra bits I suppose.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

jolsa at redhat

Sep 9, 2013, 7:03 AM

Post #6 of 15 (1728 views)

Permalink

On Mon, Sep 09, 2013 at 01:55:18PM +0200, Peter Zijlstra wrote:
> On Mon, Sep 09, 2013 at 01:36:41PM +0200, Jiri Olsa wrote:
> > On Mon, Sep 09, 2013 at 01:17:49PM +0200, Peter Zijlstra wrote:
> > > On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
> > This patchset is actually doing 3 essential things:
> > 1) adds new perf data format version (v3)
> > 2) adds class/object to handle perf data file
> > 3) adds record '-M size' option
>
> So the -M thing is where the multiple files stuff comes from, right?
> What's the purpose of this? Why would I care how big these files get,
> surely filesystems handle 4g+ files these days.

my usage currently is to having this running during the day:
(well whenever I remember to run it.. ;-) )

[jolsa@krava perf]$ sudo ./perf record -g -M 1m -a

and checking report when the system or app get stuck

with multiple files I can just easily (or automaticaly)
remove old ones without restarting session

>
> > > The only reason I wanted this is so that each thread can write its own
> > > data. The current one file thing is an immense bottle-neck for big
> > > machines.
> > >
> >
> > I haven't considered the thread based storage yet, but I
> > think having the simplified format and centralized file
> > handling will only help.
>
> Please do consider it now, it'd suck to have to change the file format
> thing yet again.

ook

>
> With both per-thread and size limited your filename generation needs
> some extra bits I suppose.

yep, and probably some merging at the end

jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

dsahern at gmail

Sep 9, 2013, 7:11 AM

Post #7 of 15 (1729 views)

Permalink

On 9/9/13 7:03 AM, Jiri Olsa wrote:
> my usage currently is to having this running during the day:
> (well whenever I remember to run it.. ;-) )
>
> [jolsa@krava perf]$ sudo ./perf record -g -M 1m -a
>
> and checking report when the system or app get stuck
>
> with multiple files I can just easily (or automaticaly)
> remove old ones without restarting session

Aren't you losing potentially important events by doing that -- FORK,
COMM, MMAP?

I have a flight recorder style command that address this problem
(long-running/daemons) by processing task events and then stashing the
sample events on a time-ordered list with chopping to maintain the time
window.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

jolsa at redhat

Sep 9, 2013, 7:31 AM

Post #8 of 15 (1730 views)

Permalink

On Mon, Sep 09, 2013 at 07:11:11AM -0700, David Ahern wrote:
> On 9/9/13 7:03 AM, Jiri Olsa wrote:
> >my usage currently is to having this running during the day:
> >(well whenever I remember to run it.. ;-) )
> >
> > [jolsa@krava perf]$ sudo ./perf record -g -M 1m -a
> >
> >and checking report when the system or app get stuck
> >
> >with multiple files I can just easily (or automaticaly)
> >remove old ones without restarting session
>
> Aren't you losing potentially important events by doing that --
> FORK, COMM, MMAP?

those are synthesized for each file via synthesize_record
function, see:

[PATCH 19/25] perf tools: Move synthesizing into single function

>
> I have a flight recorder style command that address this problem
> (long-running/daemons) by processing task events and then stashing
> the sample events on a time-ordered list with chopping to maintain
> the time window.

so far I noticed there could be race among EXIT and remaining
SAMPLE events on another CPU mmap than EXIT event.. ending up
with EXIT being stored in the old file, while SAMPLEs will get
to the new one

I was thinking about some 'perf daemon' so I dont need to run that
manually.. seems similar to what you did

jirka
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

dsahern at gmail

Sep 9, 2013, 8:03 AM

Post #9 of 15 (1732 views)

Permalink

On 9/9/13 7:31 AM, Jiri Olsa wrote:
> On Mon, Sep 09, 2013 at 07:11:11AM -0700, David Ahern wrote:
>> On 9/9/13 7:03 AM, Jiri Olsa wrote:
>>> my usage currently is to having this running during the day:
>>> (well whenever I remember to run it.. ;-) )
>>>
>>> [jolsa@krava perf]$ sudo ./perf record -g -M 1m -a
>>>
>>> and checking report when the system or app get stuck
>>>
>>> with multiple files I can just easily (or automaticaly)
>>> remove old ones without restarting session
>>
>> Aren't you losing potentially important events by doing that --
>> FORK, COMM, MMAP?
>
> those are synthesized for each file via synthesize_record
> function, see:
>
> [PATCH 19/25] perf tools: Move synthesizing into single function

Ok. haven't had time to look through your 2 large patch sets.

Seems like a lot of repetitive work on a loaded system. The overhead of
the task events will dominate compared to the samples.

>
>>
>> I have a flight recorder style command that address this problem
>> (long-running/daemons) by processing task events and then stashing
>> the sample events on a time-ordered list with chopping to maintain
>> the time window.
>
> so far I noticed there could be race among EXIT and remaining
> SAMPLE events on another CPU mmap than EXIT event.. ending up
> with EXIT being stored in the old file, while SAMPLEs will get
> to the new one
>
> I was thinking about some 'perf daemon' so I dont need to run that
> manually.. seems similar to what you did

Right now I focus on scheduling events. This latest version of it can be
easily recycled for other use cases. Some work would be needed to dump
events to a file versus dumping processed information.

I am in San Jose this week. Not sure if I will have time to finish it to
a point of pushing out patches, but maybe I can push to github in the
next couple of days.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

mingo at kernel

Sep 9, 2013, 9:06 AM

Post #10 of 15 (1731 views)

Permalink

* David Ahern <dsahern@gmail.com> wrote:

> On 9/9/13 7:03 AM, Jiri Olsa wrote:
> >my usage currently is to having this running during the day:
> >(well whenever I remember to run it.. ;-) )
> >
> > [jolsa@krava perf]$ sudo ./perf record -g -M 1m -a
> >
> >and checking report when the system or app get stuck
> >
> >with multiple files I can just easily (or automaticaly)
> >remove old ones without restarting session
>
> Aren't you losing potentially important events by doing that -- FORK,
> COMM, MMAP?

I suspect these could/should be tracked and emitted fully (in bulk) when a
new data file is opened, so that each partial data file is fully
consistent?

> I have a flight recorder style command that address this problem
> (long-running/daemons) by processing task events and then stashing the
> sample events on a time-ordered list with chopping to maintain the time
> window.

Could this be used to emit currently relevant task context?

Btw., I also think it would be useful to have kernel support for that -
the 'collections' stuff I talked about a good while ago: the kernel would
work with user-space to iterate over all MMAPs and all running COMMs at
the opening of a tracing session.

That way we could avoid racy access to /proc, we could make sure that all
information that is emitted by FORK/COMM/MMAP is also emitted for the
'bulk' data, etc.

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

adrian.hunter at intel

Sep 9, 2013, 11:54 PM

Post #11 of 15 (1731 views)

Permalink

On 09/09/13 14:17, Peter Zijlstra wrote:
> On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
>> hi,
>> sending the support for multiple file storage. Initial
>> RFC is here:
>> http://marc.info/?l=linux-kernel&m=137408381902423&w=2
>>
>> v2 changes:
>> - reworked perf mmap size setup to be able to get
>> the mmap size value easily later
>> - added perf.data read/write test for v2 and v3
>> for both endianity
>> - added record '-M time' support
>
> So this 0/n post seems to have forgotten to list the rationale for doing
> all this..
>
> The only reason I wanted this is so that each thread can write its own
> data. The current one file thing is an immense bottle-neck for big
> machines.

Do you need multiple files for that? Why not just feed writes to a pool of
threads?

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

namhyung at kernel

Sep 10, 2013, 1:57 AM

Post #12 of 15 (1731 views)

Permalink

Hi Peter and Jiri,

On Mon, 9 Sep 2013 13:17:49 +0200, Peter Zijlstra wrote:
> On Sun, Sep 01, 2013 at 12:36:11PM +0200, Jiri Olsa wrote:
>> hi,
>> sending the support for multiple file storage. Initial
>> RFC is here:
>> http://marc.info/?l=linux-kernel&m=137408381902423&w=2
>>
>> v2 changes:
>> - reworked perf mmap size setup to be able to get
>> the mmap size value easily later
>> - added perf.data read/write test for v2 and v3
>> for both endianity
>> - added record '-M time' support
>
> So this 0/n post seems to have forgotten to list the rationale for doing
> all this..
>
> The only reason I wanted this is so that each thread can write its own
> data. The current one file thing is an immense bottle-neck for big
> machines.

Per-thread or per-cpu? In my perf ftrace patchset, I used to per-cpu
data file for this reason. Do you think per-thread approach is better
than per-cpu one?

Jiri, one of my colleagues asked me about the multiple file support
separated by time while ago. At that time I just added --time-filter
option to perf report, but it'd better if perf record can support it.
(Unfortunately the patch seems to buried in the list).

Anyway, as Peter said, please consider per-thread or per-cpu multiple
file support with your series. It will help further developments.

Thanks,
Namhyung
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

peterz at infradead

Sep 10, 2013, 2:15 AM

Post #13 of 15 (1731 views)

Permalink

On Tue, Sep 10, 2013 at 09:54:38AM +0300, Adrian Hunter wrote:
> On 09/09/13 14:17, Peter Zijlstra wrote:

> > The only reason I wanted this is so that each thread can write its own
> > data. The current one file thing is an immense bottle-neck for big
> > machines.
>
> Do you need multiple files for that? Why not just feed writes to a pool of
> threads?

Not sure I understand what you mean there. Since we have a (or multiple)
event per cpu it doesn't make sense to read that data from another cpu
and have that write it to disk. That completely destroys the locality.

Instead the suggestion was to have a thread per cpu (bound) to empty out
the per cpu mmap buffer(s) and write them to disk.

Now the only way to coherently do that into a single file is to do some
streams implementation, which I suppose is possible. Pre-allocate
large-ish sections per thread and mark them as such, then let the thread
spool data into it, when full allocate a new section.

However this would result in allocation-block-size*nr_cpus minimal file
sizes, which wouldn't be a problem per-se since they can be sparse.

It will however pretty much mess up Jolsa's max-MBs per file thing and
it is of course slightly more complex than having a file per cpu/thread.
And is more likely to create kernel lock contention; the pagecache locks
are typically per inode, bouncing those around the machine isn't nice
either.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

dsahern at gmail

Sep 10, 2013, 10:29 AM

Post #14 of 15 (1720 views)

Permalink

On 9/9/13 9:06 AM, Ingo Molnar wrote:
>> Aren't you losing potentially important events by doing that -- FORK,
>> COMM, MMAP?
>
> I suspect these could/should be tracked and emitted fully (in bulk) when a
> new data file is opened, so that each partial data file is fully
> consistent?

In my case I am not saving task events, but processing them. In Jiri's
case where events are written to a file it should be possible to stash
the unprocessed events on a list, when the exit happens move them to a
dead threads list which can be cleaned up from time to time and then on
file dump requests dump the task events followed by the sample events.

>
>> I have a flight recorder style command that address this problem
>> (long-running/daemons) by processing task events and then stashing the
>> sample events on a time-ordered list with chopping to maintain the time
>> window.
>
> Could this be used to emit currently relevant task context?

sure.

>
> Btw., I also think it would be useful to have kernel support for that -
> the 'collections' stuff I talked about a good while ago: the kernel would
> work with user-space to iterate over all MMAPs and all running COMMs at
> the opening of a tracing session.
>
> That way we could avoid racy access to /proc, we could make sure that all
> information that is emitted by FORK/COMM/MMAP is also emitted for the
> 'bulk' data, etc.

Walking the task list and emitting events would be better but wouldn't
that be a performance hit holding the task lock (tasklist_lock?)? (I
thought that one is needed when walking the task list.)

David

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Re: [PATCHv2 00/25] perf tool: Add support for multiple data file storage [ In reply to ]

dsahern at gmail

Sep 14, 2013, 11:32 AM

Post #15 of 15 (1719 views)

Permalink

On 9/9/13 9:03 AM, David Ahern wrote:
>>> I have a flight recorder style command that address this problem
>>> (long-running/daemons) by processing task events and then stashing
>>> the sample events on a time-ordered list with chopping to maintain
>>> the time window.
>>
>> so far I noticed there could be race among EXIT and remaining
>> SAMPLE events on another CPU mmap than EXIT event.. ending up
>> with EXIT being stored in the old file, while SAMPLEs will get
>> to the new one
>>
>> I was thinking about some 'perf daemon' so I dont need to run that
>> manually.. seems similar to what you did
>
> Right now I focus on scheduling events. This latest version of it can be
> easily recycled for other use cases. Some work would be needed to dump
> events to a file versus dumping processed information.
>
> I am in San Jose this week. Not sure if I will have time to finish it to
> a point of pushing out patches, but maybe I can push to github in the
> next couple of days.

https://github.com/dsahern/linux/tree/perf-sched-timehist-3.11

Take a look at tools/perf/schedmon.c. Task events are processed and
sample events are stashed on a list (daemon__process_sample). Could
easily do something similar for mmap, comm, fork events and then on exit
event flush events for terminated tasks. Then you will have both the
task and sample events which can be dumped to a file for later processing.

David
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/