Mailing List Archive

EXT3 problems?
Hi there,
I seem to remember some comments regarding problems with EXT3 and periodic
thrashing of the hard drive, yet I can't find anything in the archives.

When recording, picture is fine. When watching live TV or watching a
recording, I get a disconcerting pause in the video every couple of seconds
or so. The hard disk activity light only briefly illuminates every second or
so (hardly busy!). I'm using EXT3 for storage, and it seemed that somebody
had a similar problem some time ago.

Some details:

Recording: 480x480 RTJPEG @ quality 200
Audio: MP3 @ 44.1kHz.

1.6 GHz Athlon XP
40Gb IBM Deskstar - HD parms show DMA + 32-bit ON (~35.1Mb sustained
transfer rate) (it's a quick little bugger!). Linux and ringbuffer on same
partition. (I know, but my wife needs Windows on the other disk!)
512Mb memory.
Redhat 7.3.
TWM window manager (for testing purposes). I've also killed all other
processes that are dispensable.

While recording/playback, TOP shows CPU between 40-55% on average for
X/mythbackend/mythfrontend in total.

By reducing to 352x288, the problem all but vanishes, but is still there in
a lesser form.

Am I trying to push my machine too hard? Performance tips?

Regards,
Ralph

----------
Our communications with you matter to us. This e-mail and any attachments
are confidential and are sent on the basis of our copyright, e-mail and
security policy which can be inspected by visiting
http://www.tribaldata.co.uk/contact_index.asp. and clicking on the print
friendly link. If you are not the intended recipient, please notify the
sender and delete this message. Thank you.
----------
Re: EXT3 problems? [ In reply to ]
If you think ext3 is your problem, you can try 2 things:

1) Disable access time stamps on your storage partition (add noatime in
the fstab file for the desired partition).

2) use ext2 instead. ext3 is backwards compatible with ext2. You'll lose
journaling (whole point of using ext3), but it might result in a slightly
faster Disk IO.

You can also try using XFS instead. It works much better with large file
Disk IO. I've had problems in the past with disk corruption when using
XFS, but it's probably been fixed since then.


If you're having Disk IO problems, and are only using 40-50% of your CPU,
why not use MPEG4 instead? It produces smaller files, but taxes the CPU
more.

-Rob

> Hi there,
> I seem to remember some comments regarding problems with EXT3 and periodic
> thrashing of the hard drive, yet I can't find anything in the archives.
>
> When recording, picture is fine. When watching live TV or watching a
> recording, I get a disconcerting pause in the video every couple of seconds
> or so. The hard disk activity light only briefly illuminates every second or
> so (hardly busy!). I'm using EXT3 for storage, and it seemed that somebody
> had a similar problem some time ago.
>
> Some details:
>
> Recording: 480x480 RTJPEG @ quality 200
> Audio: MP3 @ 44.1kHz.
>
> 1.6 GHz Athlon XP
> 40Gb IBM Deskstar - HD parms show DMA + 32-bit ON (~35.1Mb sustained
> transfer rate) (it's a quick little bugger!). Linux and ringbuffer on same
> partition. (I know, but my wife needs Windows on the other disk!)
> 512Mb memory.
> Redhat 7.3.
> TWM window manager (for testing purposes). I've also killed all other
> processes that are dispensable.
>
> While recording/playback, TOP shows CPU between 40-55% on average for
> X/mythbackend/mythfrontend in total.
>
> By reducing to 352x288, the problem all but vanishes, but is still there in
> a lesser form.
>
> Am I trying to push my machine too hard? Performance tips?
>
> Regards,
> Ralph
>
> ----------
> Our communications with you matter to us. This e-mail and any attachments
> are confidential and are sent on the basis of our copyright, e-mail and
> security policy which can be inspected by visiting
> http://www.tribaldata.co.uk/contact_index.asp. and clicking on the print
> friendly link. If you are not the intended recipient, please notify the
> sender and delete this message. Thank you.
> ----------
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
>



Dege

So Many Things in Life Would Be Really Funny
.... If They Weren't Happening To Me
Re: EXT3 problems? [ In reply to ]
From recent experience, I'd suggest that you stick with ext2/ext3 instead
of moving to xfs or reiserfs, for two reasons:

1. ext2/3 seems to be the only fs that supports bad block tables.
2. if you're running into issues, best not to add another component to the
equation.

Another note: Journalizing should only add overhead when writing to disk
(at least, that's the theory). If you're doing multiple recordings, you
might take a pretty big performance hit. I've got my system on ext3 with
a XP 2500+, and I'm not seeing too big a drain, tho.

Best of luck.
Aaron

On Tue, 27 May 2003, Robert Dege wrote:

>
> If you think ext3 is your problem, you can try 2 things:
>
> 1) Disable access time stamps on your storage partition (add noatime in
> the fstab file for the desired partition).
>
> 2) use ext2 instead. ext3 is backwards compatible with ext2. You'll lose
> journaling (whole point of using ext3), but it might result in a slightly
> faster Disk IO.
>
> You can also try using XFS instead. It works much better with large file
> Disk IO. I've had problems in the past with disk corruption when using
> XFS, but it's probably been fixed since then.
>
>
> If you're having Disk IO problems, and are only using 40-50% of your CPU,
> why not use MPEG4 instead? It produces smaller files, but taxes the CPU
> more.
>
> -Rob
>
> > Hi there,
> > I seem to remember some comments regarding problems with EXT3 and periodic
> > thrashing of the hard drive, yet I can't find anything in the archives.
> >
> > When recording, picture is fine. When watching live TV or watching a
> > recording, I get a disconcerting pause in the video every couple of seconds
> > or so. The hard disk activity light only briefly illuminates every second or
> > so (hardly busy!). I'm using EXT3 for storage, and it seemed that somebody
> > had a similar problem some time ago.
> >
> > Some details:
> >
> > Recording: 480x480 RTJPEG @ quality 200
> > Audio: MP3 @ 44.1kHz.
> >
> > 1.6 GHz Athlon XP
> > 40Gb IBM Deskstar - HD parms show DMA + 32-bit ON (~35.1Mb sustained
> > transfer rate) (it's a quick little bugger!). Linux and ringbuffer on same
> > partition. (I know, but my wife needs Windows on the other disk!)
> > 512Mb memory.
> > Redhat 7.3.
> > TWM window manager (for testing purposes). I've also killed all other
> > processes that are dispensable.
> >
> > While recording/playback, TOP shows CPU between 40-55% on average for
> > X/mythbackend/mythfrontend in total.
> >
> > By reducing to 352x288, the problem all but vanishes, but is still there in
> > a lesser form.
> >
> > Am I trying to push my machine too hard? Performance tips?
> >
> > Regards,
> > Ralph
> >
> > ----------
> > Our communications with you matter to us. This e-mail and any attachments
> > are confidential and are sent on the basis of our copyright, e-mail and
> > security policy which can be inspected by visiting
> > http://www.tribaldata.co.uk/contact_index.asp. and clicking on the print
> > friendly link. If you are not the intended recipient, please notify the
> > sender and delete this message. Thank you.
> > ----------
> >
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users@snowman.net
> > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
> >
>
>
>
> Dege
>
> So Many Things in Life Would Be Really Funny
> .... If They Weren't Happening To Me
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
>
Re: EXT3 problems? [ In reply to ]
I believe that by default ext3 only journals meta-data... which means it
shouldn't add much overhead at all even with multiple writes. If you
mount the file system with the noatime option there should be even less.

On Tue, 2003-05-27 at 08:30, pixel@hourglassone.com wrote:
> >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> of moving to xfs or reiserfs, for two reasons:
>
> 1. ext2/3 seems to be the only fs that supports bad block tables.
> 2. if you're running into issues, best not to add another component to the
> equation.
>
> Another note: Journalizing should only add overhead when writing to disk
> (at least, that's the theory). If you're doing multiple recordings, you
> might take a pretty big performance hit. I've got my system on ext3 with
> a XP 2500+, and I'm not seeing too big a drain, tho.
>
> Best of luck.
> Aaron
>
> On Tue, 27 May 2003, Robert Dege wrote:
>
> >
> > If you think ext3 is your problem, you can try 2 things:
> >
> > 1) Disable access time stamps on your storage partition (add noatime in
> > the fstab file for the desired partition).
> >
> > 2) use ext2 instead. ext3 is backwards compatible with ext2. You'll lose
> > journaling (whole point of using ext3), but it might result in a slightly
> > faster Disk IO.
> >
> > You can also try using XFS instead. It works much better with large file
> > Disk IO. I've had problems in the past with disk corruption when using
> > XFS, but it's probably been fixed since then.
> >
> >
> > If you're having Disk IO problems, and are only using 40-50% of your CPU,
> > why not use MPEG4 instead? It produces smaller files, but taxes the CPU
> > more.
> >
> > -Rob
> >
> > > Hi there,
> > > I seem to remember some comments regarding problems with EXT3 and periodic
> > > thrashing of the hard drive, yet I can't find anything in the archives.
> > >
> > > When recording, picture is fine. When watching live TV or watching a
> > > recording, I get a disconcerting pause in the video every couple of seconds
> > > or so. The hard disk activity light only briefly illuminates every second or
> > > so (hardly busy!). I'm using EXT3 for storage, and it seemed that somebody
> > > had a similar problem some time ago.
> > >
> > > Some details:
> > >
> > > Recording: 480x480 RTJPEG @ quality 200
> > > Audio: MP3 @ 44.1kHz.
> > >
> > > 1.6 GHz Athlon XP
> > > 40Gb IBM Deskstar - HD parms show DMA + 32-bit ON (~35.1Mb sustained
> > > transfer rate) (it's a quick little bugger!). Linux and ringbuffer on same
> > > partition. (I know, but my wife needs Windows on the other disk!)
> > > 512Mb memory.
> > > Redhat 7.3.
> > > TWM window manager (for testing purposes). I've also killed all other
> > > processes that are dispensable.
> > >
> > > While recording/playback, TOP shows CPU between 40-55% on average for
> > > X/mythbackend/mythfrontend in total.
> > >
> > > By reducing to 352x288, the problem all but vanishes, but is still there in
> > > a lesser form.
> > >
> > > Am I trying to push my machine too hard? Performance tips?
> > >
> > > Regards,
> > > Ralph
> > >
> > > ----------
> > > Our communications with you matter to us. This e-mail and any attachments
> > > are confidential and are sent on the basis of our copyright, e-mail and
> > > security policy which can be inspected by visiting
> > > http://www.tribaldata.co.uk/contact_index.asp. and clicking on the print
> > > friendly link. If you are not the intended recipient, please notify the
> > > sender and delete this message. Thank you.
> > > ----------
> > >
> > > _______________________________________________
> > > mythtv-users mailing list
> > > mythtv-users@snowman.net
> > > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
> > >
> >
> >
> >
> > Dege
> >
> > So Many Things in Life Would Be Really Funny
> > .... If They Weren't Happening To Me
> >
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users@snowman.net
> > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
> >
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
Re: EXT3 problems? [ In reply to ]
> >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> of moving to xfs or reiserfs, for two reasons:

I second this. I was experiencing exactly the same thing when I tried
to use ext3, even after having made the adjustments suggested in the
mailing list. I am now using ext2 and am quite happy with it. From my
testing, though, I actually discovered fat32 seems to have worked best
with my specific setup. Not having seen any benchmarks that support
this, it could be just a fluke. The only reason I'm not using fat32
right now is its inability to store files over 2 gigs...which isn't so
much a problem if your system can handle using the MPEG4 codec.


Brett
Re: EXT3 problems? [ In reply to ]
On Tue, 2003-05-27 at 13:44, Brett Profitt wrote:
> > >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> > of moving to xfs or reiserfs, for two reasons:
>
> I second this. I was experiencing exactly the same thing when I tried
> to use ext3, even after having made the adjustments suggested in the
> mailing list. I am now using ext2 and am quite happy with it. From my
> testing, though, I actually discovered fat32 seems to have worked best
> with my specific setup. Not having seen any benchmarks that support
> this, it could be just a fluke. The only reason I'm not using fat32
> right now is its inability to store files over 2 gigs...which isn't so
> much a problem if your system can handle using the MPEG4 codec.

I've been using reiserfs with good results. I've yet to encounter an
issue where reiserfs was the culprit and have used it on a variety of
systems including MythTV video partitions. At the very least I would
suggest some kind of journaled filesystem, who wants to wait for an fsck
to finish on a 200GB mythtv partition?

Aran
Re: EXT3 problems? [ In reply to ]
It may be time to create a new FS for Media. There is a reason that TiVo
didn't use any of the standard FS's.

>
> On Tue, 2003-05-27 at 13:44, Brett Profitt wrote:
> > > >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> > > of moving to xfs or reiserfs, for two reasons:
> >
> > I second this. I was experiencing exactly the same thing when I tried
> > to use ext3, even after having made the adjustments suggested in the
> > mailing list. I am now using ext2 and am quite happy with it. From my
> > testing, though, I actually discovered fat32 seems to have worked best
> > with my specific setup. Not having seen any benchmarks that support
> > this, it could be just a fluke. The only reason I'm not using fat32
> > right now is its inability to store files over 2 gigs...which isn't so
> > much a problem if your system can handle using the MPEG4 codec.
>
> I've been using reiserfs with good results. I've yet to encounter an
> issue where reiserfs was the culprit and have used it on a variety of
> systems including MythTV video partitions. At the very least I would
> suggest some kind of journaled filesystem, who wants to wait for an fsck
> to finish on a 200GB mythtv partition?
>
> Aran
>
>
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
>
>
>
RE: EXT3 problems? [ In reply to ]
> It may be time to create a new FS for Media. There is a
> reason that TiVo didn't use any of the standard FS's.

I'm inclined to agree with that. One of the benefits of TiVo's MFS
filesystem is that it allocates space in 512MB clusters. With TiVo's usual
range of recording qualities, that amounts to about 30 minutes per
allocation. It really cuts down on the amout of drive space and time wasted
in allocating things in (relatively) small cluster sizes that most normal
filesystems use.

I wonder if ext[23] or reiser, jfs, or any of the other options supported by
current kernels can be formatted with huge cluster sizes? Or failing that,
could they be easily and reliably hacked to do so?

Unfortunately, I'm not even sure if the kernel could handle clusters that
large. For TiVo's MFS, they implemented everything in userspace without any
help from the kernel, so who knows...

Best regards,
Zac Bedell
Re: EXT3 problems? [ In reply to ]
Hrmm.. Seems like a Bad Idea (tm) to me. adding another filesystem that
needs supporting is nobody's idea of a cakewalk. Kernel source
maintainers may not like it too much, either.

However, that being said, if you want to tackle it :).


On Tue, 2003-05-27 at 14:22, usenet@wingert.org wrote:
> It may be time to create a new FS for Media. There is a reason that TiVo
> didn't use any of the standard FS's.
>
> >
> > On Tue, 2003-05-27 at 13:44, Brett Profitt wrote:
> > > > >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> > > > of moving to xfs or reiserfs, for two reasons:
> > >
> > > I second this. I was experiencing exactly the same thing when I tried
> > > to use ext3, even after having made the adjustments suggested in the
> > > mailing list. I am now using ext2 and am quite happy with it. From my
> > > testing, though, I actually discovered fat32 seems to have worked best
> > > with my specific setup. Not having seen any benchmarks that support
> > > this, it could be just a fluke. The only reason I'm not using fat32
> > > right now is its inability to store files over 2 gigs...which isn't so
> > > much a problem if your system can handle using the MPEG4 codec.
> >
> > I've been using reiserfs with good results. I've yet to encounter an
> > issue where reiserfs was the culprit and have used it on a variety of
> > systems including MythTV video partitions. At the very least I would
> > suggest some kind of journaled filesystem, who wants to wait for an fsck
> > to finish on a 200GB mythtv partition?
> >
> > Aran
> >
> >
> >
> >
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users@snowman.net
> > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
> >
> >
> >
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
Re: EXT3 problems? [ In reply to ]
I noticed something about xfs subvolumes in the kernel that should lend
themselves nicely to media applications - anybody tinkered with them yet?

John

On Tue, May 27, 2003 at 02:22:55PM -0700, usenet@wingert.org wrote:
> It may be time to create a new FS for Media. There is a reason that TiVo
> didn't use any of the standard FS's.
>
> >
> > On Tue, 2003-05-27 at 13:44, Brett Profitt wrote:
> > > > >From recent experience, I'd suggest that you stick with ext2/ext3 instead
> > > > of moving to xfs or reiserfs, for two reasons:
> > >
> > > I second this. I was experiencing exactly the same thing when I tried
> > > to use ext3, even after having made the adjustments suggested in the
> > > mailing list. I am now using ext2 and am quite happy with it. From my
> > > testing, though, I actually discovered fat32 seems to have worked best
> > > with my specific setup. Not having seen any benchmarks that support
> > > this, it could be just a fluke. The only reason I'm not using fat32
> > > right now is its inability to store files over 2 gigs...which isn't so
> > > much a problem if your system can handle using the MPEG4 codec.
> >
> > I've been using reiserfs with good results. I've yet to encounter an
> > issue where reiserfs was the culprit and have used it on a variety of
> > systems including MythTV video partitions. At the very least I would
> > suggest some kind of journaled filesystem, who wants to wait for an fsck
> > to finish on a 200GB mythtv partition?
> >
> > Aran
> >
> >
> >
> >
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users@snowman.net
> > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
> >
> >
> >
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
Re: EXT3 problems? [ In reply to ]
Along those lines, has anyone noticed any performance improvement by
formatting their ext2/3 partitions with '-T largefile4', which should make
one inode per 4meg? And to repeat Zac's question, do reiser/xfs/jfs do
anything similar?

Or does this not even come into play when using options like noatime for
ext3?


----- Original Message -----
From: "Zachary S. Bedell" <zaclist@adirondack.net>
To: "'Discussion about mythtv'" <mythtv-users@snowman.net>
Sent: Tuesday, May 27, 2003 5:03 PM
Subject: RE: [mythtv-users] EXT3 problems?


> > It may be time to create a new FS for Media. There is a
> > reason that TiVo didn't use any of the standard FS's.
>
> I'm inclined to agree with that. One of the benefits of TiVo's MFS
> filesystem is that it allocates space in 512MB clusters. With TiVo's
usual
> range of recording qualities, that amounts to about 30 minutes per
> allocation. It really cuts down on the amout of drive space and time
wasted
> in allocating things in (relatively) small cluster sizes that most
normal
> filesystems use.
>
> I wonder if ext[23] or reiser, jfs, or any of the other options
supported by
> current kernels can be formatted with huge cluster sizes? Or failing
that,
> could they be easily and reliably hacked to do so?
>
> Unfortunately, I'm not even sure if the kernel could handle clusters
that
> large. For TiVo's MFS, they implemented everything in userspace without
any
> help from the kernel, so who knows...
>
> Best regards,
> Zac Bedell
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
Re: EXT3 problems? [ In reply to ]
Using larger inode chunks allows you to reclaim some disk space. Since
you're assigning a single inode to every 4 megs instead of every 4 kb, you
reclaim space that would have previously been reserved for the additional
inode tables.

-Rob

> Along those lines, has anyone noticed any performance improvement by
> formatting their ext2/3 partitions with '-T largefile4', which should make
> one inode per 4meg? And to repeat Zac's question, do reiser/xfs/jfs do
> anything similar?
>
> Or does this not even come into play when using options like noatime for
> ext3?
>
>
> ----- Original Message -----
> From: "Zachary S. Bedell" <zaclist@adirondack.net>
> To: "'Discussion about mythtv'" <mythtv-users@snowman.net>
> Sent: Tuesday, May 27, 2003 5:03 PM
> Subject: RE: [mythtv-users] EXT3 problems?
>
>
> > > It may be time to create a new FS for Media. There is a
> > > reason that TiVo didn't use any of the standard FS's.
> >
> > I'm inclined to agree with that. One of the benefits of TiVo's MFS
> > filesystem is that it allocates space in 512MB clusters. With TiVo's
> usual
> > range of recording qualities, that amounts to about 30 minutes per
> > allocation. It really cuts down on the amout of drive space and time
> wasted
> > in allocating things in (relatively) small cluster sizes that most
> normal
> > filesystems use.
> >
> > I wonder if ext[23] or reiser, jfs, or any of the other options
> supported by
> > current kernels can be formatted with huge cluster sizes? Or failing
> that,
> > could they be easily and reliably hacked to do so?
> >
> > Unfortunately, I'm not even sure if the kernel could handle clusters
> that
> > large. For TiVo's MFS, they implemented everything in userspace without
> any
> > help from the kernel, so who knows...
> >
> > Best regards,
> > Zac Bedell
> >
> >
> > _______________________________________________
> > mythtv-users mailing list
> > mythtv-users@snowman.net
> > http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
>
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users@snowman.net
> http://lists.snowman.net/cgi-bin/mailman/listinfo/mythtv-users
>



Dege

So Many Things in Life Would Be Really Funny
.... If They Weren't Happening To Me
RE: EXT3 problems? [ In reply to ]
> Along those lines, has anyone noticed any performance improvement by
> formatting their ext2/3 partitions with '-T largefile4', which should make
> one inode per 4meg? And to repeat Zac's question, do reiser/xfs/jfs do
> anything similar?
>
> Or does this not even come into play when using options like noatime for
> ext3?
>

I created my "video" partition using custom mkfs options. Basically I used
options which decreased the ability to store lots of small files in favour
of large files ala Myth. Given that my current dedicated 30G partition will
not even hold 100 video files (at that's a pretty extreme case - only 300
meg each)

I pretty sure I gained something like 1-2 million blocks (i.e. 1-2GB), which
although not huge was worth the 15 minutes or so it took me to work out.

Have a look at the man page.

As for performance improvements, I can't really say since I never tried with
a normal partition. I doubt they would be that great though.

Oh and I use ext3 with the noatime option.


Christian