Mailing List Archive

for storing files, I've seen it's a lot easier simply doing linear
appending via software raid0, or using
other volume management. For linear appending, the overhead is basically
nill, and it greatly decreases any
complexity. you don't really want to worry about what hard drives have
what percent free, and what files
you need to shuffle around...etc..

for what it's worth, that's what I'm doing now, and it helps a ton. I
used to have 8 drives, each separate, and
it was really a pain to manage them.

-Mark


-----Original Message-----
From: mythtv-dev-admin@snowman.net [mailto:mythtv-dev-admin@snowman.net]
On Behalf Of Jeremy Oddo
Sent: Thursday, November 28, 2002 12:02 PM
To: mythtv-dev@snowman.net
Subject: [mythtv] Possible Future Option?


I don't know how hard this would be to do, but I was thinking of a nice
little option (don't know if it's been talked about before). I think it
would be cool to be able to define multiple recording volumes.
Something
like:

/mnt/store0
/mnt/store1
/mnt/store2
/mnt/store3

Each /mnt/store# could be a different drive. Then, in the SETTINGS
option within MythTV, you could pick which path (drive) you want to use.
It might also be cool to list stats by each path (ie: % used, % free,
etc.).
Then, when you record a program, the MySQL database could have a field
for the path being used for that recording (/mnt/store0, /mnt/store1,
etc.).
This would allow you to add drives ad hoc. You could add storage as you
need it.


jeremy


_______________________________________________
mythtv-dev mailing list
mythtv-dev@snowman.net
http://www.snowman.net/mailman/listinfo/mythtv-dev

On Thu, Nov 28, 2002 at 09:02:05AM -0800, Jeremy Oddo wrote:

> I don't know how hard this would be to do, but I was thinking of a nice
> little option (don't know if it's been talked about before). I think it
> would be cool to be able to define multiple recording volumes. Something
> like:
>
> /mnt/store0
> /mnt/store1
> /mnt/store2
> /mnt/store3
>
> Each /mnt/store# could be a different drive. Then, in the SETTINGS option
> within MythTV, you could pick which path (drive) you want to use. It
> might also be cool to list stats by each path (ie: % used, % free, etc.).
> Then, when you record a program, the MySQL database could have a field for
> the path being used for that recording (/mnt/store0, /mnt/store1, etc.).
> This would allow you to add drives ad hoc. You could add storage as you
> need it.

This is beyond the scope of myth, I think. What you want is a volume
manager like EVMS or LVM. The Linux md driver might also meet your needs if
it is capable of expanding an existing raid-linear set (I'm not sure whether
it can).

--
- mdz

> On Thu, Nov 28, 2002 at 09:02:05AM -0800, Jeremy Oddo wrote:
>
>> I don't know how hard this would be to do, but I was thinking of a
>> nice little option (don't know if it's been talked about before). I
>> think it would be cool to be able to define multiple recording
>> volumes. Something like:
>>
>> /mnt/store0
>> /mnt/store1
>> /mnt/store2
>> /mnt/store3
>>
>> Each /mnt/store# could be a different drive. Then, in the SETTINGS
>> option within MythTV, you could pick which path (drive) you want to
>> use. It might also be cool to list stats by each path (ie: % used, %
>> free, etc.). Then, when you record a program, the MySQL database
>> could have a field for the path being used for that recording
>> (/mnt/store0, /mnt/store1, etc.). This would allow you to add drives
>> ad hoc. You could add storage as you need it.
>
> This is beyond the scope of myth, I think. What you want is a volume
> manager like EVMS or LVM. The Linux md driver might also meet your
> needs if it is capable of expanding an existing raid-linear set (I'm not
> sure whether it can).
>
Perhaps you misunderstood me, because what I am talking about doesn't seem
beyond the scope of myth, seeing as how most of it is already in myth!
I'm not talking about fancy volume managment. Right now we can define
where we want our recordings to go. I was just thinking that it would be
cool to be able to define MULTIPLE paths (or an array of paths) and then
be able to select that "record to" path under the "settings" option. I'm
not much of a coder, but I know this wouldn't take to much coding. And
I'm guessing that a lot of the variables that are in the settings.txt will
end up in the settings option someday.

Just my thoughts....

Jeremy

On Thu, Nov 28, 2002 at 11:49:02PM -0800, Jeremy Oddo wrote:

> Perhaps you misunderstood me, because what I am talking about doesn't seem
> beyond the scope of myth, seeing as how most of it is already in myth!
> I'm not talking about fancy volume managment. Right now we can define
> where we want our recordings to go. I was just thinking that it would be
> cool to be able to define MULTIPLE paths (or an array of paths) and then
> be able to select that "record to" path under the "settings" option. I'm
> not much of a coder, but I know this wouldn't take to much coding. And
> I'm guessing that a lot of the variables that are in the settings.txt will
> end up in the settings option someday.

I understood fine, but I cannot think of a realistic situation where would
be useful to me. Why would you want to place different recordings under
different paths?

--
- mdz

Matt Zimmerman wrote:
> On Thu, Nov 28, 2002 at 11:49:02PM -0800, Jeremy Oddo wrote:
>> ...I was just thinking that it would be
>>cool to be able to define MULTIPLE paths (or an array of paths) and then
>>be able to select that "record to" path under the "settings" option...
>
> I understood fine, but I cannot think of a realistic situation where would
> be useful to me. Why would you want to place different recordings under
> different paths?

Matt, first off, thank you for your efforts on this project.
I first installed from your Debian packages to check things
out before compiling from source. It was very reassuring to
have everything configured correctly for me and to see it
work on the first try.

There may never be a realistic situation where storing files
in different directories would be useful to you and I believe
that there are certainly bigger fish to fry. However, I too
thought this might be useful even before seeing Jeremy's post.

First, I'm using a machine with a 30GB disk. Root and swap
are at the beginning then a large partition where I put the
video files. I want more space so I'll probably get a 60-100
Gig disk. I may not be able to stripe the mismatched disks
together and may not want to even if I could. If I can only
write to one dir on the new disk I can't take advantage of
the free space on the current disk. Myth could choose to
start a new recording in the dir with the most free space
so the user doesn't need to worry about it.

Next, with multiple tuners, the performance might be better
if each file is streamed to one dedicated disk. The fastest
way to write a large continuos files is to position the
disk head and let it rip with the heads only making track
to track seeks at the end of each cylinder. Any RAID
scheme would cause more head movement for this specific
case even though RAID is much better in typical cases of
random reads and writes. Again, myth could choose the dir
with the most free space that is not already being used
for another recording.

I've been thinking about how to have tuners on separate
machines with each writing to its local disk. To do this
the 'recorded' table might need a column for the hostname
in order to locate the recordings. The path could also be
included in this table. Even if the path is not stored
in the database, the filename for a recording could be
searched for in each of the configured directories.

So, I second Jeremy's request but please don't put off
any of the great stuff being worked on now for this
little extra.

-- bjm

On 30 Nov 2002 at 0:54, Matt Zimmerman wrote:

> I understood fine, but I cannot think of a realistic situation where would be
> useful to me. Why would you want to place different recordings under
> different paths?

I can think of a few:

1) You have an external/removable/portable hard drive - firewire, USB2.0 etc
2) In relation to the separate encoder and decoder, you could write files to
a local drive and when that's full, write to the network and/or vice versa.
3) You add a second drive and want to alternate recording between drives.,
Say all Buffy shows on drive 1 and all Angel shows on drive 2 etc

If this was a democracy, it would get my vote ;-)

Plus, setting up LVM is more reading and work and also if one f your drives
dies in an LVM setup, you could potentially lose a lot of depending on how
you are setup. In the case that started this thread, you lose the contents on
one but have the stuff on the second or thirs or whatever drive.


--
Harondel J. Sibble
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
help@pdscc.com (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax) (604) 686-2253 (pager)

On Sat, Nov 30, 2002 at 12:45:38AM -0800, Bruce Markey wrote:

> Matt Zimmerman wrote:
> >I understood fine, but I cannot think of a realistic situation where
> >would be useful to me. Why would you want to place different recordings
> >under different paths?
>
> Matt, first off, thank you for your efforts on this project. I first
> installed from your Debian packages to check things out before compiling
> from source. It was very reassuring to have everything configured
> correctly for me and to see it work on the first try.

Thanks, it's always good to hear about successful installations. In most
cases, those without problems don't provide feedback.

> There may never be a realistic situation where storing files in different
> directories would be useful to you and I believe that there are certainly
> bigger fish to fry. However, I too thought this might be useful even
> before seeing Jeremy's post.
>
> First, I'm using a machine with a 30GB disk. Root and swap are at the
> beginning then a large partition where I put the video files. I want more
> space so I'll probably get a 60-100 Gig disk. I may not be able to stripe
> the mismatched disks together and may not want to even if I could. If I
> can only write to one dir on the new disk I can't take advantage of the
> free space on the current disk. Myth could choose to start a new recording
> in the dir with the most free space so the user doesn't need to worry
> about it.

Simple concatenation (e.g., RAID-linear or EVMS drivelinking) has no problem
with differently-sized disks, and lets you place one large filesystem on the
volume. If the goal is to pool your available storage, then a lower-level
solution is a superior option, I think. It avoids the need for a decision
about which filesystem to use, and allows for files which are larger than
would fit on one disk.

> Next, with multiple tuners, the performance might be better if each file
> is streamed to one dedicated disk. The fastest way to write a large
> continuos files is to position the disk head and let it rip with the heads
> only making track to track seeks at the end of each cylinder. Any RAID
> scheme would cause more head movement for this specific case even though
> RAID is much better in typical cases of random reads and writes. Again,
> myth could choose the dir with the most free space that is not already
> being used for another recording.

This is an interesting idea. I would like to see how this works out in
practice as far as throughput. A smart driver will merge the requests for
sequential data into a single request sent to the controller, even if the
data is destined for discontiguous userland buffers (scatter/gather). I
know that Linux SCSI drivers will do this (indeed, a decent SCSI controller
will do this in hardware), but I'm not sure about the IDE subsystem. If it
can efficiently do scatter/gather I/O, I would expect performance to be
nearly equivalent in both cases, since the random-order requests for
contiguous blocks would be merged into one big sequential request per disk.

I think this probably comes down more to using a filesystem which is tuned
(or tunable) for large, contiguous file allocation. I believe XFS was
designed with this in mind.

--
- mdz

On Sat, Nov 30, 2002 at 08:07:55AM -0800, Harondel J. Sibble wrote:

> On 30 Nov 2002 at 0:54, Matt Zimmerman wrote:
>
> > I understood fine, but I cannot think of a realistic situation where would be
> > useful to me. Why would you want to place different recordings under
> > different paths?
>
> I can think of a few:
>
> 1) You have an external/removable/portable hard drive - firewire, USB2.0 etc

And? Either you want recorings to go directly to that drive, or not. If
you plan to remove it, you don't want scheduled recordings to go to it. You
would record to a fixed disk and then copy the recordings if you wanted to
take them somewhere.

> 2) In relation to the separate encoder and decoder, you could write files to
> a local drive and when that's full, write to the network and/or vice versa.

What's the benefit of this setup? If your goal is to play back content from
the local drive when possible, it would be better implemented as a cache
than letting it overflow onto another system.

> 3) You add a second drive and want to alternate recording between drives.,
> Say all Buffy shows on drive 1 and all Angel shows on drive 2 etc

Why?

> Plus, setting up LVM is more reading and work and also if one f your
> drives dies in an LVM setup, you could potentially lose a lot of depending
> on how you are setup. In the case that started this thread, you lose the
> contents on one but have the stuff on the second or thirs or whatever
> drive.

In a concatenated volume, you generally lose all files which have blocks
stored on the failed disk (the other parts of the file are recoverable, but
this is usually not interesting unless the data is critically important).
In a one-filesystem-per-disk setup, you lose all files which are stored on
the failed disk. These scenarios could be identical or different, depending
on the filesystem.

In any non-redundant setup, you are going to lose data when a disk fails.
TV recordings are not generally mission-critical anyway.

--
- mdz

It would be nice to place different recordings under different paths so you
could have recordings from one tuner card go to one hard drive and recordings
from a second tuner go to a different hard drive. This would (hopefully)
result in less hard drive thrashing and better performance.

On Saturday 30 November 2002 00:54, Matt Zimmerman wrote:
> On Thu, Nov 28, 2002 at 11:49:02PM -0800, Jeremy Oddo wrote:
> > Perhaps you misunderstood me, because what I am talking about doesn't
> > seem beyond the scope of myth, seeing as how most of it is already in
> > myth! I'm not talking about fancy volume managment. Right now we can
> > define where we want our recordings to go. I was just thinking that it
> > would be cool to be able to define MULTIPLE paths (or an array of paths)
> > and then be able to select that "record to" path under the "settings"
> > option. I'm not much of a coder, but I know this wouldn't take to much
> > coding. And I'm guessing that a lot of the variables that are in the
> > settings.txt will end up in the settings option someday.
>
> I understood fine, but I cannot think of a realistic situation where would
> be useful to me. Why would you want to place different recordings under
> different paths?