Mailing List Archive

Thoughts on encoder farms / distributed capturing and viewing
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Isaac, do you have thoughts on how you're going to proceed? I've
been thinking of "nice to have" features. Stream of thought stuff
here...

o Self-organizing encoder farms. Every 30 seconds, mythtv sends out
a broadcast packet to the local LAN that has information in it like:
hostname, disk space free, number of encoder cards and encoder card
type (or source, in case something like DirecTV is only available on
one encoder). This allows cooperative scheduling. The other way is
to simply do what Cisco's IP/TV does: at startup, the server
enumerates the encoder cards and the existing content and sends a
message back to the Content Manager. The Content Manager then
changes the status of the server to "Managed" and maintains a table.
The server and the content manager have to be manually configured to
make them aware of each other.

o A machine that doesn't have any encoder cards is still valuable,
either as a transcode machine, or as a disk space repository /
archive server.

o Totally decoupling the encoder and the viewer. Therefore, a local
node that has an encoder _might not_ be using it to watch live TV;
its "live" channel may actually be coming from a different encoder on
a different node. The ringbuffer file is getting filled using IP.

o A master scheduler is always looking a few minutes into the
future. If a program is supposed to be recorded, use the "free
encoder card" broadcast messages to determine which node should do
the recording, and the free space message to determine if we need to
purge content. If you use a master/slave relationship rather than a
cooperative one, then this might not be as big an issue, since
presumably the master will always know the status of each encoder
card.

o A "stripe" based conflict resolution scheme. I'm not sure that
I'll be able to explain this, but I'll try. One of the things that I
don't like about my Tivo is the following: "Everybody Loves Raymond"
runs on CBS from 1900 to 1930 in Chicago. Sometimes, due to clock
issues, or the network being a little late, we miss the last bit of
"zinger" before the final credit roll. So I tell Tivo to go 2
minutes long, and so now it records from 1900 to 1932. The problem
is that if there's a show on a different channel starting at 1930, it
won't get recorded at all, because the Tivo isn't allocating smaller
blocks. So even if I'm willing to miss the initial credits on a
show, I'm out of luck. My thinking is that if you split a show into
1 minute blocks and overlap based on priority, the 32 minute show
will overlap the first two minutes of the show on the other channel.
Every minute, the encoder is checking to master schedule to see what
channel it should be recording, so it will naturally "fall off" the
end of the 32 minute show and onto the 28 minute one. This scheme
also handles "early start", since a higher priority show will overlap
the end of a lower priority show. Of course, if there are multiple
encoder cards that are free then there's no conflict.

o Partial deletes. I know that you mentioned commercial cutting,
but I'm interested in "delete up to this point". Say I have a 2 hour
recording, and I've already watched 1 hour of it. I want to free up
the disk space by deleting the first hour.

o A "hint" system for shows that are on multiple times per day. On
my Tivo, if I tell it to record "The Daily Show", it's going to
record it 4 times a day because lately, TMS hasn't been putting in
show descriptions, so the Tivo doesn't know that the showing at 1400
and 1900 is a rebroadcast of the previous days' 2200 broadcast. When
I do a manual record by time, I have to go in and delete the Friday
"Comedy Central Presents" which is on at the same time because I
can't program the Tivo to record M-Th.

o Configurable "history". Keep track of shows that were recorded,
and when they were deleted. That way, if I'm going to watch an
episode of a show and I don't remember if I've seen the OSD can tell
me, either at time of recording or in the "To Do" list. The Tivo has
a 28 day memory on this sort of stuff, but most syndicated shows take
longer to cycle through than that. Allow the user to specify how
long until purge. Disk space is fairly cheap. Note: this probably
won't work unless we get the episode descriptions instead of just the
show name.

How's that for a wish list? Can you get all of that in before 0.7?
:)



-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcABXPc1NpCTlP0JEQK46ACguCX9LhiIkgXQZE/0hLmkAvYAzSEAoM2a
EwJLfyqBBIoYmAxf0Wu2KO8S
=hX7H
-----END PGP SIGNATURE-----
RE: That's interesting: WinTV PVR USB has a working linux driver [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

"
Note: The information contained in this message may be privileged
and confidential and protected from disclosure.
...
snip
"

Sorry about the disclaimer; it's auto-added by the email server.
Believe me when I say that I wish it were optional.


-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcAtM/c1NpCTlP0JEQJHmwCeJxajCJQcOq8/7XlkrUlJx4xeVV4AoKMo
LARMJrefgQhmsOKtuQY9KrPH
=WsW9
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
Robert Kulagowski wrote:

>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>Isaac, do you have thoughts on how you're going to proceed? I've
>been thinking of "nice to have" features. Stream of thought stuff
>here...
>
>o Self-organizing encoder farms. Every 30 seconds, mythtv sends out
>a broadcast packet to the local LAN that has information in it like:
>hostname, disk space free, number of encoder cards and encoder card
>type (or source, in case something like DirecTV is only available on
>one encoder). This allows cooperative scheduling. The other way is
>to simply do what Cisco's IP/TV does: at startup, the server
>enumerates the encoder cards and the existing content and sends a
>message back to the Content Manager. The Content Manager then
>changes the status of the server to "Managed" and maintains a table.
>The server and the content manager have to be manually configured to
>make them aware of each other.
>
>
I am not sure about Isaac code yet as I am build a box right now for
install and development on but based on the fact that everything is
stored in SQL already I would think that a Decated SQL server and some
naming logic in his configure code and you would have a basic master
server already so you would just have to add a program to manage that data.

>o A machine that doesn't have any encoder cards is still valuable,
>either as a transcode machine, or as a disk space repository /
>archive server.
>
>
Would be true if you broke up pices of his software. One of the Thing
that could help with recoding to VCD/SVCD/DVD/MPG* recoding.

>o Totally decoupling the encoder and the viewer. Therefore, a local
>node that has an encoder _might not_ be using it to watch live TV;
>its "live" channel may actually be coming from a different encoder on
>a different node. The ringbuffer file is getting filled using IP.
>
>
Proble no possible as that would require alot of bandwidth on you local
network I not sure most nic card could handle that much data you would
have to create some form of mess network were you have 2 or 3 Nic in
each machine to send thought all that data. Also there would be alot of
latance as the sending machine would have to begain recording start
direct connection to the sending and buffer data on both ends to deal
with packet delay/loss which would make it very unresponsive.

>o A master scheduler is always looking a few minutes into the
>future. If a program is supposed to be recorded, use the "free
>encoder card" broadcast messages to determine which node should do
>the recording, and the free space message to determine if we need to
>purge content. If you use a master/slave relationship rather than a
>cooperative one, then this might not be as big an issue, since
>presumably the master will always know the status of each encoder
>card.
>
>
Correct some form of mangment appt would have to be used and priorty
would also be useful for machines in rooms you don't use as much or
recording only machines.

>o A "stripe" based conflict resolution scheme. I'm not sure that
>I'll be able to explain this, but I'll try. One of the things that I
>don't like about my Tivo is the following: "Everybody Loves Raymond"
>runs on CBS from 1900 to 1930 in Chicago. Sometimes, due to clock
>issues, or the network being a little late, we miss the last bit of
>"zinger" before the final credit roll. So I tell Tivo to go 2
>minutes long, and so now it records from 1900 to 1932. The problem
>is that if there's a show on a different channel starting at 1930, it
>won't get recorded at all, because the Tivo isn't allocating smaller
>blocks. So even if I'm willing to miss the initial credits on a
>show, I'm out of luck. My thinking is that if you split a show into
>1 minute blocks and overlap based on priority, the 32 minute show
>will overlap the first two minutes of the show on the other channel.
>Every minute, the encoder is checking to master schedule to see what
>channel it should be recording, so it will naturally "fall off" the
>end of the 32 minute show and onto the 28 minute one. This scheme
>also handles "early start", since a higher priority show will overlap
>the end of a lower priority show. Of course, if there are multiple
>encoder cards that are free then there's no conflict.
>
>
That would be real tough to setup you are talking crazy level of AI.
Althought the conflict manger could be used to decide on thing like
that by a human who could make those kind of decisions

>o Partial deletes. I know that you mentioned commercial cutting,
>but I'm interested in "delete up to this point". Say I have a 2 hour
>recording, and I've already watched 1 hour of it. I want to free up
>the disk space by deleting the first hour.
>
>
Once commercial cutting is added that would be easy to add if someone
wanted to take the time and felt there was a reason to do it.

>o A "hint" system for shows that are on multiple times per day. On
>my Tivo, if I tell it to record "The Daily Show", it's going to
>record it 4 times a day because lately, TMS hasn't been putting in
>show descriptions, so the Tivo doesn't know that the showing at 1400
>and 1900 is a rebroadcast of the previous days' 2200 broadcast. When
>I do a manual record by time, I have to go in and delete the Friday
>"Comedy Central Presents" which is on at the same time because I
>can't program the Tivo to record M-Th.
>
>
Again to much AI there if got created it would never work the way YOU
wanted it would work the way it would be designed. Althought I think
the ID of a Manual record menu which works like a VCR would be nice for
time when program data doesn't download or time when the data is
outright wrong.

>o Configurable "history". Keep track of shows that were recorded,
>and when they were deleted. That way, if I'm going to watch an
>episode of a show and I don't remember if I've seen the OSD can tell
>me, either at time of recording or in the "To Do" list. The Tivo has
>a 28 day memory on this sort of stuff, but most syndicated shows take
>longer to cycle through than that. Allow the user to specify how
>long until purge. Disk space is fairly cheap. Note: this probably
>won't work unless we get the episode descriptions instead of just the
>show name.
>
>
That is proble the easiest overall to do and can be setup to use very
little HD space. What about something alitlle more useful like a
database of stuff that is going to be record and when at what time
moving that Describtion of the show into the database so you have a full
list by clicking on a more button on you remote control and state such
as set_to_record/recording/Unwatched/watched/deleted/or anything else
which I could think about with a setting in the settings file on when to
delete from the database. Also that could be used to setup a system to
indecate if I watched a show but my wife hasn't

>How's that for a wish list? Can you get all of that in before 0.7?
>:)
>
>
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wed, 2002-10-30 at 11:51, Robert Middleswarth wrote:
> Robert Kulagowski wrote:
> >o Totally decoupling the encoder and the viewer. Therefore, a local
> >node that has an encoder _might not_ be using it to watch live TV;
> >its "live" channel may actually be coming from a different encoder on
> >a different node. The ringbuffer file is getting filled using IP.
> >
> >
> Proble no possible as that would require alot of bandwidth on you local
> network I not sure most nic card could handle that much data you would
> have to create some form of mess network were you have 2 or 3 Nic in
> each machine to send thought all that data. Also there would be alot of
> latance as the sending machine would have to begain recording start
> direct connection to the sending and buffer data on both ends to deal
> with packet delay/loss which would make it very unresponsive.

??? Check out videolan which does *just* this (decoupling the "server" /
source from the client / viewer). The idea isn't that far-fetched...
particularly if the source is mpeg4 & the client side does the decoding
of the stream -- heck, by spec an SVCD is something like 2.6Mbps video +
224Kbps audio...

> >o A "stripe" based conflict resolution scheme. I'm not sure that
> >I'll be able to explain this, but I'll try. One of the things that I
> >don't like about my Tivo is the following: "Everybody Loves Raymond"
> >runs on CBS from 1900 to 1930 in Chicago. Sometimes, due to clock
> >issues, or the network being a little late, we miss the last bit of
> >"zinger" before the final credit roll. So I tell Tivo to go 2
> >minutes long, and so now it records from 1900 to 1932. The problem
> >is that if there's a show on a different channel starting at 1930, it
> >won't get recorded at all, because the Tivo isn't allocating smaller
> >blocks. So even if I'm willing to miss the initial credits on a
> >show, I'm out of luck. My thinking is that if you split a show into
> >1 minute blocks and overlap based on priority, the 32 minute show
> >will overlap the first two minutes of the show on the other channel.
> >Every minute, the encoder is checking to master schedule to see what
> >channel it should be recording, so it will naturally "fall off" the
> >end of the 32 minute show and onto the 28 minute one. This scheme
> >also handles "early start", since a higher priority show will overlap
> >the end of a lower priority show. Of course, if there are multiple
> >encoder cards that are free then there's no conflict.
> >
> >
> That would be real tough to setup you are talking crazy level of AI.
> Althought the conflict manger could be used to decide on thing like
> that by a human who could make those kind of decisions

Not really any "crazy" AI -- actually, the existing conflict resolution
code is probably sufficient... Hm. Think I'll look at this. Basically,
for each minute, you'd need to see: Are we recording? Is there a
conflict? If so, does the current recording or the "other" recording
have priority? If current, just keep going. If "other," close this
one, start that one.

A couple of edge cases would need to be done, too -- like moving from
the higher-priority "current" to the "new" when the higher-priority one
is done (i.e.: at 1902)... Hm. Yeah, time to get that extra system
finished...

> >o A "hint" system for shows that are on multiple times per day. On
> >my Tivo, if I tell it to record "The Daily Show", it's going to
> >record it 4 times a day because lately, TMS hasn't been putting in
> >show descriptions, so the Tivo doesn't know that the showing at 1400
> >and 1900 is a rebroadcast of the previous days' 2200 broadcast. When
> >I do a manual record by time, I have to go in and delete the Friday
> >"Comedy Central Presents" which is on at the same time because I
> >can't program the Tivo to record M-Th.
> >
> >
> Again to much AI there if got created it would never work the way YOU
> wanted it would work the way it would be designed. Althought I think
> the ID of a Manual record menu which works like a VCR would be nice for
> time when program data doesn't download or time when the data is
> outright wrong.

No offense intended, but... what's this fascination with AI? And if
he's proposing requirements, why wouldn't they at least be discussed as
part of the design?

That said, I don't see a 100% solution to this (yet), but certainly if
one is using a consistent source for descriptions, and they are fairly
robust (i.e.: the description is available for all showings of the same
"episode", then one could track that and see if we have currently
recorded that "description" in addition to a particular time.


Cheers,
--Craig
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wednesday 30 October 2002 10:57 am, Robert Kulagowski wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Isaac, do you have thoughts on how you're going to proceed? I've
> been thinking of "nice to have" features. Stream of thought stuff
> here...
>
> o Self-organizing encoder farms. Every 30 seconds, mythtv sends out
> a broadcast packet to the local LAN that has information in it like:
> hostname, disk space free, number of encoder cards and encoder card
> type (or source, in case something like DirecTV is only available on
> one encoder). This allows cooperative scheduling. The other way is
> to simply do what Cisco's IP/TV does: at startup, the server
> enumerates the encoder cards and the existing content and sends a
> message back to the Content Manager. The Content Manager then
> changes the status of the server to "Managed" and maintains a table.
> The server and the content manager have to be manually configured to
> make them aware of each other.

I'll probably follow the master/slaves model, but yeah, something like that.
Configure one box as a master node, have it run the DB and act on requests
from client set-top boxes, etc. Encoding nodes can automatically announce
themselves, etc.

> o A machine that doesn't have any encoder cards is still valuable,
> either as a transcode machine, or as a disk space repository /
> archive server.

Yup.

> o Totally decoupling the encoder and the viewer. Therefore, a local
> node that has an encoder _might not_ be using it to watch live TV;
> its "live" channel may actually be coming from a different encoder on
> a different node. The ringbuffer file is getting filled using IP.

Should be quite easy to do, actually. Someone's doing live-tv completely over
a NFS mount right now, and this'd be half the amount of network traffic.

> o A master scheduler is always looking a few minutes into the
> future. If a program is supposed to be recorded, use the "free
> encoder card" broadcast messages to determine which node should do
> the recording, and the free space message to determine if we need to
> purge content. If you use a master/slave relationship rather than a
> cooperative one, then this might not be as big an issue, since
> presumably the master will always know the status of each encoder
> card.

Right, the master box can know everything, and just tell a slave when to start
and stop, no need to query, etc.

> o A "stripe" based conflict resolution scheme. I'm not sure that
> I'll be able to explain this, but I'll try. One of the things that I
> don't like about my Tivo is the following: "Everybody Loves Raymond"
> runs on CBS from 1900 to 1930 in Chicago. Sometimes, due to clock
> issues, or the network being a little late, we miss the last bit of
> "zinger" before the final credit roll. So I tell Tivo to go 2
> minutes long, and so now it records from 1900 to 1932. The problem
> is that if there's a show on a different channel starting at 1930, it
> won't get recorded at all, because the Tivo isn't allocating smaller
> blocks. So even if I'm willing to miss the initial credits on a
> show, I'm out of luck. My thinking is that if you split a show into
> 1 minute blocks and overlap based on priority, the 32 minute show
> will overlap the first two minutes of the show on the other channel.
> Every minute, the encoder is checking to master schedule to see what
> channel it should be recording, so it will naturally "fall off" the
> end of the 32 minute show and onto the 28 minute one. This scheme
> also handles "early start", since a higher priority show will overlap
> the end of a lower priority show. Of course, if there are multiple
> encoder cards that are free then there's no conflict.

_Should_ be fairly easy to accomplish that. I've already got over-time
recording in there, it'd just need logic to decide to delay starting another
recording while the overage is finished on the first one.. The current code
doesn't handle early starts, though.

> o Partial deletes. I know that you mentioned commercial cutting,
> but I'm interested in "delete up to this point". Say I have a 2 hour
> recording, and I've already watched 1 hour of it. I want to free up
> the disk space by deleting the first hour.

Once the ability's there to remove segments of the video, this'd be easy..
The hard part's rewriting the file, really.

> o A "hint" system for shows that are on multiple times per day. On
> my Tivo, if I tell it to record "The Daily Show", it's going to
> record it 4 times a day because lately, TMS hasn't been putting in
> show descriptions, so the Tivo doesn't know that the showing at 1400
> and 1900 is a rebroadcast of the previous days' 2200 broadcast. When
> I do a manual record by time, I have to go in and delete the Friday
> "Comedy Central Presents" which is on at the same time because I
> can't program the Tivo to record M-Th.

The current timeslot recording stuff doesn't handle that? It won't record
unless the title of the program matches.

> o Configurable "history". Keep track of shows that were recorded,
> and when they were deleted. That way, if I'm going to watch an
> episode of a show and I don't remember if I've seen the OSD can tell
> me, either at time of recording or in the "To Do" list. The Tivo has
> a 28 day memory on this sort of stuff, but most syndicated shows take
> longer to cycle through than that. Allow the user to specify how
> long until purge. Disk space is fairly cheap. Note: this probably
> won't work unless we get the episode descriptions instead of just the
> show name.

It already keeps track of what you've recorded.

> How's that for a wish list? Can you get all of that in before 0.7?

Can you? =) Patches are, as always, accepted.

Isaac
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> The current code doesn't handle early starts, though.

Which is why I think the striping system would be a solution to early
starts / long runs. Since you're allocating a 1 minute block, all
you're worried about is 1440 blocks / day, and the highest priority
program in any 1 minute slot is the channel that's going to get
recorded at that time.

> > o Partial deletes. I know that you mentioned commercial
> > cutting,
> Once the ability's there to remove segments of the video, this'd be
> easy.. The hard part's rewriting the file, really.

I assumed that "delete to this point" was just a specialized case of
"cut video"; just throwing things out to see what sort of inspiration
it may cause in others.

> > o A "hint" system for shows that are on multiple times per day.
> The current timeslot recording stuff doesn't handle that? It
> won't record unless the title of the program matches.

Not sure yet; I was only able to get mythtv compiled on my laptop
fairly recently, and I don't have a grabber, so a lot of the
functionality isn't available to me. IIRC, the setup program pretty
much expects you to have at least one /dev/video device and barfs if
you don't. Have to wait until I get back to the main linux box to
see how this works.

> Can you? =) Patches are, as always, accepted.

That's why I mailed you a diff for the README for users with Mandrake
Linux 9.0...



-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcBlqvc1NpCTlP0JEQK1UwCg4Z/s6Z2AcDiCLdky2K7wFcq5Ey8AnjJb
T5v7kBlwZOA9Xi6EqpeHfB1X
=1nbn
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wed, Oct 30, 2002 at 03:49:16PM -0500, Isaac Richards wrote:

> On Wednesday 30 October 2002 10:57 am, Robert Kulagowski wrote:
> > o Partial deletes. I know that you mentioned commercial cutting,
> > but I'm interested in "delete up to this point". Say I have a 2 hour
> > recording, and I've already watched 1 hour of it. I want to free up
> > the disk space by deleting the first hour.
>
> Once the ability's there to remove segments of the video, this'd be easy..
> The hard part's rewriting the file, really.

What's the hard part about rewriting the file? I must be missing something.

--
- mdz
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wednesday 30 October 2002 06:12 pm, Matt Zimmerman wrote:
> On Wed, Oct 30, 2002 at 03:49:16PM -0500, Isaac Richards wrote:
> > On Wednesday 30 October 2002 10:57 am, Robert Kulagowski wrote:
> > > o Partial deletes. I know that you mentioned commercial cutting,
> > > but I'm interested in "delete up to this point". Say I have a 2 hour
> > > recording, and I've already watched 1 hour of it. I want to free up
> > > the disk space by deleting the first hour.
> >
> > Once the ability's there to remove segments of the video, this'd be
> > easy.. The hard part's rewriting the file, really.
>
> What's the hard part about rewriting the file? I must be missing
> something.

There isn't anything, really.. The only thing to do is a little re-encoding
to guarantee that there's keyframes in the proper spaces.

Isaac
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wednesday 30 October 2002 06:05 pm, Robert Kulagowski wrote:
> > The current code doesn't handle early starts, though.
>
> Which is why I think the striping system would be a solution to early
> starts / long runs. Since you're allocating a 1 minute block, all
> you're worried about is 1440 blocks / day, and the highest priority
> program in any 1 minute slot is the channel that's going to get
> recorded at that time.

I think that that would overcomplicate things. Starting early can be done in
a similar manner to how I did ending late.

> > > o Partial deletes. I know that you mentioned commercial
> > > cutting,
> >
> > Once the ability's there to remove segments of the video, this'd be
> > easy.. The hard part's rewriting the file, really.
>
> I assumed that "delete to this point" was just a specialized case of
> "cut video"; just throwing things out to see what sort of inspiration
> it may cause in others.

It is -- rewriting the file's the only hard part about either operation =)

> > Can you? =) Patches are, as always, accepted.
>
> That's why I mailed you a diff for the README for users with Mandrake
> Linux 9.0...

I'd prefer things like that in a separate file, actually, especially to the
extent of distribution-specific instructions that you included.

Isaac
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I think that that would overcomplicate things. Starting
> early can be done in a similar manner to how I did ending late.

Sounds good to me. Just throwing out ideas...

> > That's why I mailed you a diff for the README for users with
> > Mandrake Linux 9.0...

> I'd prefer things like that in a separate file, actually,
> especially to the extent of distribution-specific instructions that
> you included.

OK; I see that a guy named Harondel J. Sipple had a "roadmap" message
for MDK 8.2 a few days ago. You should consider using that in CVS;
he did a lot of good explaining, and some of it is platform
independant. The MDK 9.0 stuff is relatively short, since a lot of
the items in MDK 8.2 which required compiling, like qt to get the
mysql support, is now out-of-the-box for 9.0.

Harondel, want to put together a combined Readme for Mandrake
8.x/9.0?

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcB5Ofc1NpCTlP0JEQJqLQCbB7LS/Gz+gxf8O1LCu4vc/n+41eAAoL/Z
g9OoHpAWeUkNf+ltK/xjv9eg
=dOCc
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wed, Oct 30, 2002 at 03:49:16PM -0500, Isaac Richards wrote:
> On Wednesday 30 October 2002 10:57 am, Robert Kulagowski wrote:
>
> > o Partial deletes. I know that you mentioned commercial cutting,
> > but I'm interested in "delete up to this point". Say I have a 2 hour
> > recording, and I've already watched 1 hour of it. I want to free up
> > the disk space by deleting the first hour.
>
> Once the ability's there to remove segments of the video, this'd be easy..
> The hard part's rewriting the file, really.

There are already some good editing and transcoding tools out there, even a
commercial removal tool or two, if we could get the recording into a format
that standard tools understand (and back). Is there any information
availble about the format MythTV is currently using (aside from the fact
that it is modified nupple video)?

--
Ray
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On 30 Oct 2002 at 14:51, Robert Middleswarth wrote:

> Proble no possible as that would require alot of bandwidth on you local
> network I not sure most nic card could handle that much data you would
> have to create some form of mess network were you have 2 or 3 Nic in
> each machine to send thought all that data. Also there would be alot of
Sure, that's kinda what firehose is for, team it with some gigabit nics over
cat5e and it would probably be usable.

http://www.heroinewarrior.com/firehose.php3


--
Harondel J. Sibble
Sibble Computer Consulting
Creating solutions for the small business and home computer user.
help@pdscc.com (use pgp keyid 0x3AD5C11D) http://www.pdscc.com
(604) 739-3709 (voice/fax) (604) 686-2253 (pager)
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> > Proble no possible as that would require alot of bandwidth
> on you local
> > network I not sure most nic card could handle that much
> data you would
> > have to create some form of mess network were you have 2 or
> 3 Nic in
> > each machine to send thought all that data. Also there
> would be alot of
> Sure, that's kinda what firehose is for, team it with some
> gigabit nics over
> cat5e and it would probably be usable.
>
> http://www.heroinewarrior.com/firehose.php3

Which is nice, except I think we started off on the wrong foot by
assuming that we're transferring raw video over the network, which
we're not. Remember, the video coming from the encoder node, whether
it's being displayed locally or being fed over IP (either through NFS
sharing of the ringbuffer.nuv file, or directly filling it) has
already been through the compressor, so we're not nearly talking the
same amount of data.

Looking at settings.txt, the default is 3.3Mbps for mpeg4; this is
well within the capacity of even 10Mbps ethernet. Might be an issue
if it's 10Mbps shared hub, but there you go.

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcDHdfc1NpCTlP0JEQKOGQCgwKVjN03IUBTdpTUsNwjfJbcJdfMAoOUO
btDPUFm6WyfywNq4cJcGBUH9
=Krfh
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
I will try an clairfy as much as I can. Wrote most of this well at work
so my thought were alittle disjointed sometimes.

Craig McLaughlin wrote:

>On Wed, 2002-10-30 at 11:51, Robert Middleswarth wrote:
>
>
>>Robert Kulagowski wrote:
>>
>>
>>>o Totally decoupling the encoder and the viewer. Therefore, a local
>>>node that has an encoder _might not_ be using it to watch live TV;
>>>its "live" channel may actually be coming from a different encoder on
>>>a different node. The ringbuffer file is getting filled using IP.
>>>
>>>
>>>
>>>
>>Proble no possible as that would require alot of bandwidth on you local
>>network I not sure most nic card could handle that much data you would
>>have to create some form of mess network were you have 2 or 3 Nic in
>>each machine to send thought all that data. Also there would be alot of
>>latance as the sending machine would have to begain recording start
>>direct connection to the sending and buffer data on both ends to deal
>>with packet delay/loss which would make it very unresponsive.
>>
>>
>
>??? Check out videolan which does *just* this (decoupling the "server" /
>source from the client / viewer). The idea isn't that far-fetched...
>particularly if the source is mpeg4 & the client side does the decoding
>of the stream -- heck, by spec an SVCD is something like 2.6Mbps video +
>224Kbps audio...
>
You are right that 1 Well Comprised Video Signal could be sent from a
Video Sever to a Video client without to much trouble. but we aren't
talking about that. We are talking about multipule pc talking back and
forth. So 1 PC could be asked to Xfer Saved data and Xfer Live Data and
Receive Live Data using a Codec that was designed for speed not size.
Althought we are talking about 100 based network cards they do start to
slow down in the 3-4 Meg range. Granted if you are talking about 1
Machine for Encodeing and 1 Machine that only displayed then yes there
would be no problem but that isn't what Mythtv is doing at this time and
if I read right not what Robert Kulagowski was talking about.

>>>o A "stripe" based conflict resolution scheme. I'm not sure that
>>>I'll be able to explain this, but I'll try. One of the things that I
>>>don't like about my Tivo is the following: "Everybody Loves Raymond"
>>>runs on CBS from 1900 to 1930 in Chicago. Sometimes, due to clock
>>>issues, or the network being a little late, we miss the last bit of
>>>"zinger" before the final credit roll. So I tell Tivo to go 2
>>>minutes long, and so now it records from 1900 to 1932. The problem
>>>is that if there's a show on a different channel starting at 1930, it
>>>won't get recorded at all, because the Tivo isn't allocating smaller
>>>blocks. So even if I'm willing to miss the initial credits on a
>>>show, I'm out of luck. My thinking is that if you split a show into
>>>1 minute blocks and overlap based on priority, the 32 minute show
>>>will overlap the first two minutes of the show on the other channel.
>>>Every minute, the encoder is checking to master schedule to see what
>>>channel it should be recording, so it will naturally "fall off" the
>>>end of the 32 minute show and onto the 28 minute one. This scheme
>>>also handles "early start", since a higher priority show will overlap
>>>the end of a lower priority show. Of course, if there are multiple
>>>encoder cards that are free then there's no conflict.
>>>
>>>
>>>
>>>
>>That would be real tough to setup you are talking crazy level of AI.
>> Althought the conflict manger could be used to decide on thing like
>>that by a human who could make those kind of decisions
>>
>>
>
>Not really any "crazy" AI -- actually, the existing conflict resolution
>code is probably sufficient... Hm. Think I'll look at this. Basically,
>for each minute, you'd need to see: Are we recording? Is there a
>conflict? If so, does the current recording or the "other" recording
>have priority? If current, just keep going. If "other," close this
>one, start that one.
>
>A couple of edge cases would need to be done, too -- like moving from
>the higher-priority "current" to the "new" when the higher-priority one
>is done (i.e.: at 1902)... Hm. Yeah, time to get that extra system
>finished...
>
Please note I use the term AI in very generic way. I think you are
right I mis-read what he was asking . :-[ I think you are right about
that being rather easy to do althought it opens a few questions about
recording on the same channel. Example how would you handle 8:00 PM
Seventh Heaven 9:00 PM Everywood both show are one hour long on the same
channel.

>>>o A "hint" system for shows that are on multiple times per day. On
>>>my Tivo, if I tell it to record "The Daily Show", it's going to
>>>record it 4 times a day because lately, TMS hasn't been putting in
>>>show descriptions, so the Tivo doesn't know that the showing at 1400
>>>and 1900 is a rebroadcast of the previous days' 2200 broadcast. When
>>>I do a manual record by time, I have to go in and delete the Friday
>>>"Comedy Central Presents" which is on at the same time because I
>>>can't program the Tivo to record M-Th.
>>>
>>>
>>>
>>>
>>Again to much AI there if got created it would never work the way YOU
>>wanted it would work the way it would be designed. Althought I think
>>the ID of a Manual record menu which works like a VCR would be nice for
>>time when program data doesn't download or time when the data is
>>outright wrong.
>>
>>
>
>No offense intended, but... what's this fascination with AI? And if
>he's proposing requirements, why wouldn't they at least be discussed as
>part of the design?
>
>That said, I don't see a 100% solution to this (yet), but certainly if
>one is using a consistent source for descriptions, and they are fairly
>robust (i.e.: the description is available for all showings of the same
>"episode", then one could track that and see if we have currently
>recorded that "description" in addition to a particular time.
>
>
>Cheers,
>--Craig
>
>
I think I got the last two sections mixed up. That should teach me for
responding well I was at work. Don't get me wrong I think some of his
idea are great and never hurts to talk about it I was just putting in my
$.02. If there is good esponde guide info then there should be a no
problem as the system will see same episode descriptions and wont record
but when there isn't a usable description and all that is list is the
shows name 5 times a day. Figuire out which one to record and which
one not to may not be so easy. The code to try and sort that out would
require some form of comprise to make the code generic enfo to be used
in the project. Tring to make a system that would pick 1 of those 5
would require more work that to just simple enter the data into a VCR.
Try and make something that would learn that 22:00 is best but if you
can't due 22:00 then do 14:00 instead would require either alot of user
spefic code and/or make some assemptions like only record 1st/last/x
time of a certain show today which would add complexty to the interface.

Sorry I wasn't as clear as I had ment tobe. It is getting late now and
I need to get some sleep. I hope this replay is more clear on what I
was tring to say.

Robert
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
Robert Kulagowski wrote:

>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>
>
>>>Proble no possible as that would require alot of bandwidth
>>>
>>>
>>on you local
>>
>>
>>>network I not sure most nic card could handle that much
>>>
>>>
>>data you would
>>
>>
>>>have to create some form of mess network were you have 2 or
>>>
>>>
>>3 Nic in
>>
>>
>>>each machine to send thought all that data. Also there
>>>
>>>
>>would be alot of
>>Sure, that's kinda what firehose is for, team it with some
>>gigabit nics over
>>cat5e and it would probably be usable.
>>
>>http://www.heroinewarrior.com/firehose.php3
>>
>>
>
>Which is nice, except I think we started off on the wrong foot by
>assuming that we're transferring raw video over the network, which
>we're not. Remember, the video coming from the encoder node, whether
>it's being displayed locally or being fed over IP (either through NFS
>sharing of the ringbuffer.nuv file, or directly filling it) has
>already been through the compressor, so we're not nearly talking the
>same amount of data.
>
>Looking at settings.txt, the default is 3.3Mbps for mpeg4; this is
>well within the capacity of even 10Mbps ethernet. Might be an issue
>if it's 10Mbps shared hub, but there you go.
>
As some who has worked with 10 Mbps ethernet it isn't 100 Mbps yes
wouldn't a problem unless you try to also xfer a saved image at the same
time. Example I have recorded enterprice on this machine and my wife
wants to touched by an angel and the schd put this on my mothers box who
is watching Seveth Heven that is recorded on box B. So we would have
the encoded runing data to my wifes box C. NFS Xfer to My Box D and Mom
Pulling Data from Box A all three adding up to over 10 Megs way above
the limit I have seen on most signal nic card. It could be done but
would require 2 to 3 nics on separt networks and Switch adding cost and
complety in my opion.
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Pulling Data from Box A all three adding up to over 10 Megs way
> above the limit I have seen on most signal nic card. It could be
> done but would require 2 to 3 nics on separt networks and Switch
> adding cost and
> complety in my opion.

I'm not sure that separate NICs are required - it seems like you're
talking about establishing multiple point to point links. As I
stated, at 10Mbps shared, multiple data streams will be a problem,
but when a 10/100 Mbps NIC is $5, (check http://www.pricewatch.com,
or go to your local microcenter) and a GigabitE card is $45
(pricewatch again) and a 8 port 100Mbps switch is $20 (pricewatch, or
microcenter), the extra money required to get sufficient data rates
isn't "crazy" money.

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcDTJ/c1NpCTlP0JEQLluQCfefla0tKZq2oO364jvJjNfraFYo4AnRnv
PqssxFSICnWw4BqlJggJMznt
=R/ww
-----END PGP SIGNATURE-----
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Pulling Data from Box A all three adding up to over 10 Megs way
> above the limit I have seen on most signal nic card. It could be
> done but would require 2 to 3 nics on separt networks and Switch
> adding cost and
> complety in my opion.

I'm not sure that separate NICs are required - it seems like you're
talking about establishing multiple point to point links. As I
stated, at 10Mbps shared, multiple data streams will be a problem,
but when a 10/100 Mbps NIC is $5, (check http://www.pricewatch.com,
or go to your local microcenter) and a GigabitE card is $45
(pricewatch again) and a 8 port 100Mbps switch is $20 (pricewatch, or
microcenter), the extra money required to get sufficient data rates
isn't "crazy" money.

-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcDYJPc1NpCTlP0JEQJLGwCg3Iz/1R4sKAHXJH4JF/rn4fMaIY8AnRli
AwTzpclAV3X2Xy/opgQUiYr+
=cLvg
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
>>>o A "hint" system for shows that are on multiple times per day.
>
>If there is good esponde guide info then there should be a no
>problem as the system will see same episode descriptions and wont record
>but when there isn't a usable description and all that is list is the
>shows name 5 times a day. Figuire out which one to record and which
>one not to may not be so easy.

Actually, this can be a lot easier if you just try to fix the
program guide, i.e. change the entries in the database. The logic
already exists in the scheduler to deal with multiple copies of the
same episode and avoid other conflicts at the same time. So, if
you just write a small script to generate identical subtitles and
descriptions (say the date of the first showing), and run it after
mythfilldatabase, the existing code can do the work.

I've been meaning to write this for a while, but I haven't have the
time. I'll try to make it generic enough for other people to use
and send it to the list when I have a chance.

David
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
Robert Kulagowski wrote:

>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>
>
>>Pulling Data from Box A all three adding up to over 10 Megs way
>>above the limit I have seen on most signal nic card. It could be
>>done but would require 2 to 3 nics on separt networks and Switch
>>adding cost and
>>complety in my opion.
>>
>>
>
>I'm not sure that separate NICs are required - it seems like you're
>talking about establishing multiple point to point links. As I
>stated, at 10Mbps shared, multiple data streams will be a problem,
>but when a 10/100 Mbps NIC is $5, (check http://www.pricewatch.com,
>or go to your local microcenter) and a GigabitE card is $45
>(pricewatch again) and a 8 port 100Mbps switch is $20 (pricewatch, or
>microcenter), the extra money required to get sufficient data rates
>isn't "crazy" money.
>
>
I wasn't saying it can't be done I am saying that it might not work with
hooking eveything up with a single nic into a 100 Megabit switch and not
have problems. I was saying that it might require some Muilt-nic
installs to get the bandwidth require that would cause additional
complexty which is something that would be good to avoid is possible.
Yes I known that switch and Nic are cheap it wasn't the money I was
thinking about it was the complexty I was thinking about.

Robert
RE: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> Actually, this can be a lot easier if you just try to fix the
> program guide, i.e. change the entries in the database.

Makes sense; so even if there are no descriptions for an episode,
like the problems with the Daily Show where episode descriptions are
blank, the "hint" logic can just dummy up the desciption by putting
in something like "Same show as YYYY-MM-DD". I wonder how many shows
there are that have this problem?

Bob


-----BEGIN PGP SIGNATURE-----
Version: PGPfreeware 7.0.3 for non-commercial use <http://www.pgp.com>

iQA/AwUBPcE1GPc1NpCTlP0JEQKvSgCfZEeSxQqjiK559pxQUaQq2PpDrxQAoKEk
12UPxQUKGskM+IYBKdXBbun4
=z1LC
-----END PGP SIGNATURE-----
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
Robert Kulagowski wrote:

>
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>
>
>>Pulling Data from Box A all three adding up to over 10 Megs way
>>above the limit I have seen on most signal nic card. It could be
>>done but would require 2 to 3 nics on separt networks and Switch
>>adding cost and
>>complety in my opion.
>>
>>
>
>I'm not sure that separate NICs are required - it seems like you're
>talking about establishing multiple point to point links. As I
>stated, at 10Mbps shared, multiple data streams will be a problem,
>but when a 10/100 Mbps NIC is $5, (check http://www.pricewatch.com,
>or go to your local microcenter) and a GigabitE card is $45
>(pricewatch again) and a 8 port 100Mbps switch is $20 (pricewatch, or
>microcenter), the extra money required to get sufficient data rates
>isn't "crazy" money.
>
>
I don't rember using the term "crazy" money last night will need to
review my message. No Point ot Point setup would probly not what I was
talking about. I am aware that nic and switches are cheap. however
someday this project is going to be used by people who are just reading
a sheet and not 100% sure how to install linux in the 1st place or buy
pre done boxes and trying to set it up and then that complexty really
show. This is from someone who everyday has to tell someone that you
have to click on the big E under windows to get to the Inet under windows.

Robert
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
David Watson wrote:

>>>>o A "hint" system for shows that are on multiple times per day.
>>>>
>>>>
>>If there is good esponde guide info then there should be a no
>>problem as the system will see same episode descriptions and wont record
>>but when there isn't a usable description and all that is list is the
>>shows name 5 times a day. Figuire out which one to record and which
>>one not to may not be so easy.
>>
>>
>
>Actually, this can be a lot easier if you just try to fix the
>program guide, i.e. change the entries in the database. The logic
>already exists in the scheduler to deal with multiple copies of the
>same episode and avoid other conflicts at the same time. So, if
>you just write a small script to generate identical subtitles and
>descriptions (say the date of the first showing), and run it after
>mythfilldatabase, the existing code can do the work.
>
>I've been meaning to write this for a while, but I haven't have the
>time. I'll try to make it generic enough for other people to use
>and send it to the list when I have a chance.
>
>David
>
>
>
That is good and will work well but I should have made this more clear
my job is currently with tech support for an inet company. I have
worked at 2 diff. inet company for over 5 years now. Because of this
backgroud I tend to look at thing in what kind of call would I get if
this becames popular. Yes then can be done with a script that does
something like that but what if that day happen to have 3 Different
episodes and no title not an impossible idea when shows star getting
cancelled and you get close to sweep then that kind of script would
block show from being recorded that you wanted to record. I was tring
to point that out although badly that it could be done but in order to
make it generic enought to be used you would probly have to ad complexty
to the interface which is something that one wants to avoid.

Robert
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wed, 2002-10-30 at 22:29, Robert Middleswarth wrote:
> You are right that 1 Well Comprised Video Signal could be sent from a
> Video Sever to a Video client without to much trouble. but we aren't
> talking about that. We are talking about multipule pc talking back and
> forth. So 1 PC could be asked to Xfer Saved data and Xfer Live Data and
> Receive Live Data using a Codec that was designed for speed not size.
> Althought we are talking about 100 based network cards they do start to
> slow down in the 3-4 Meg range. Granted if you are talking about 1
> Machine for Encodeing and 1 Machine that only displayed then yes there
> would be no problem but that isn't what Mythtv is doing at this time and
> if I read right not what Robert Kulagowski was talking about.

100Mbps NICs slow down in the 3-4 Meg range? What kind of environment
have you seen *that* in? I've seen that kind of poor performance on
some older machines running Win9x, but my little pentium 166 firewall is
routinely asked to push upward of 30, and handles it just fine... I
suspect you may be seeing IP stack issues and not hardware issues...

I regularly toss very large datasets around my network here, and often
saturate my 100Mbps switch without too much trouble (note: my 100Mbps
switch is actually a cheapo one that only has a 200Mbps backplane).

> >A couple of edge cases would need to be done, too -- like moving from
> >the higher-priority "current" to the "new" when the higher-priority one
> >is done (i.e.: at 1902)... Hm. Yeah, time to get that extra system
> >finished...
> >
> Please note I use the term AI in very generic way. I think you are
> right I mis-read what he was asking . :-[ I think you are right about
> that being rather easy to do althought it opens a few questions about
> recording on the same channel. Example how would you handle 8:00 PM
> Seventh Heaven 9:00 PM Everywood both show are one hour long on the same
> channel.

If the channel of both shows is the same, then... how about merging the
recording? One 2-hour block? Perhaps with some arbitrary split in the
middle, if one really has to have 2 separate files.

> Sorry I wasn't as clear as I had ment tobe. It is getting late now and
> I need to get some sleep. I hope this replay is more clear on what I
> was tring to say.

I understand, I think; I just happen to disagree. No offense, of
course. I think most of what you're concerned about can be handled
quite transparently. Remember the target audience, too... I would tend
to think that there's a ways to go with Linux, Capture cards, etc,
before MythTV becomes the "barrier to entry."

--Craig
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Thu, Oct 31, 2002 at 08:49:29AM -0500, Robert Middleswarth wrote:
> >
> I wasn't saying it can't be done I am saying that it might not work with
> hooking eveything up with a single nic into a 100 Megabit switch and not
> have problems. I was saying that it might require some Muilt-nic
> installs to get the bandwidth require that would cause additional
> complexty which is something that would be good to avoid is possible.
> Yes I known that switch and Nic are cheap it wasn't the money I was
> thinking about it was the complexty I was thinking about.


Even after re-reading your other messages I still don't see how you'd come
even close to saturating a 100mb nic without a lot (like 5+) Mythtv systems
all running at once. Also consider that as things stand most systems sold
today can encode mpeg4 in real time at full or nearly full res which should
help the bandwidth problems. Computers arn't getting any slower either.

--
Ray
Re: Thoughts on encoder farms / distributed capturing and viewing [ In reply to ]
On Wed, Oct 30, 2002 at 09:16:58PM -0800, Harondel J. Sibble wrote:

> On 30 Oct 2002 at 14:51, Robert Middleswarth wrote:
>
> > Proble no possible as that would require alot of bandwidth on you local
> > network I not sure most nic card could handle that much data you would
> > have to create some form of mess network were you have 2 or 3 Nic in
> > each machine to send thought all that data. Also there would be alot of
> Sure, that's kinda what firehose is for, team it with some gigabit nics over
> cat5e and it would probably be usable.
>
> http://www.heroinewarrior.com/firehose.php3

This is what the kernel bonding driver is for.

--
- mdz

1 2  View All