Mailing List Archive

On Thu, 9 Jan 2020 23:35:04 +0100, you wrote:

>Hi everybody,
>
>I've just added a patch from Mark Spieth to mythtv-setup that enables
>getting the logical channel numbers from broadcasters such as Sky
>(US/IT/NZ/AU), Dishnet(US) and others that use OpenTV.
>Unfortunately I cannot receive an OpenTV channel but I am very
>interested in how it works so if anybody can try it and report back
>then yes please!
>
>I am also very interested in a full transport stream capture of these
>signals for analysis purposes. It is now relatively easy to capture
>the full stream, a.k.a. MPTS channel, with MythTV. One minute would be
>great!

I use MythTV to record from Sky New Zealand, and I get my OpenTV EPG
from the Optus D1 12519 transponder, which has full EPG for all the
Sky NZ channels on all transponders:

https://www.lyngsat.com/Optus-D1.html

I used dvbsnoop to capture 60 seconds from 12519 and I have put it on
my web server:

http://www.jsw.gen.nz/mythtv/SkyNZ_OpenTV_12519.ts
(233,252,565 bytes)

My web server runs on an ancient OS/2 setup, and only supports single
threaded downloads, without any ability to restart failed downloads.
And New Zealand is a long way from anywhere in terms of Internet ping
times. So I recommend using wget from a Linux box for large downloads
from my server to get the best speed.

Let me know if you would like more (or different) data captured from
Sky NZ. I am not certain that 60 seconds is enough to contain a full
cycle of the EPG data.

For my EPG grabbing, I am using a copy of tv_grab_dvb_plus that I have
heavily modified to work with the current Sky NZ OpenTV data. It
works well enough to get the EPG for the channels I use on Sky NZ, but
it is far from perfect and does not get correct data for all the SkyNZ
channels. It does see the LCNs (logical channel numbers), and I have
modified it to provide the LCNs as a "number=" field in comments in he
XML output. Let me know if you would like a copy of the source.

The original tv_grab_db_plus is here:
https://sourceforge.net/projects/tvgrabeit/

If you have managed to find proper documentation on the OpenTV EPG
format, I would love to have a copy as that should allow me to make
tv_grab_dvb_plus fully compatible. At the moment, there are a number
of descriptor types that it does not know how to decode. I only have
the basic DVB documentation to work from, nothing that is OpenTV
specific. The original authors of tv_grab_dvb_plus do seem to have
had more than that.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 1/10/2020 4:11 PM, Stephen Worthington wrote:
> On Thu, 9 Jan 2020 23:35:04 +0100, you wrote:
>
>> Hi everybody,
>>
>> I've just added a patch from Mark Spieth to mythtv-setup that enables
>> getting the logical channel numbers from broadcasters such as Sky
>> (US/IT/NZ/AU), Dishnet(US) and others that use OpenTV.
>> Unfortunately I cannot receive an OpenTV channel but I am very
>> interested in how it works so if anybody can try it and report back
>> then yes please!
>>
>> I am also very interested in a full transport stream capture of these
>> signals for analysis purposes. It is now relatively easy to capture
>> the full stream, a.k.a. MPTS channel, with MythTV. One minute would be
>> great!
> I use MythTV to record from Sky New Zealand, and I get my OpenTV EPG
> from the Optus D1 12519 transponder, which has full EPG for all the
> Sky NZ channels on all transponders:
>
> https://www.lyngsat.com/Optus-D1.html
>
> I used dvbsnoop to capture 60 seconds from 12519 and I have put it on
> my web server:
>
> http://www.jsw.gen.nz/mythtv/SkyNZ_OpenTV_12519.ts
> (233,252,565 bytes)
>
> My web server runs on an ancient OS/2 setup, and only supports single
> threaded downloads, without any ability to restart failed downloads.
> And New Zealand is a long way from anywhere in terms of Internet ping
> times. So I recommend using wget from a Linux box for large downloads
> from my server to get the best speed.
>
> Let me know if you would like more (or different) data captured from
> Sky NZ. I am not certain that 60 seconds is enough to contain a full
> cycle of the EPG data.
>
> For my EPG grabbing, I am using a copy of tv_grab_dvb_plus that I have
> heavily modified to work with the current Sky NZ OpenTV data. It
> works well enough to get the EPG for the channels I use on Sky NZ, but
> it is far from perfect and does not get correct data for all the SkyNZ
> channels. It does see the LCNs (logical channel numbers), and I have
> modified it to provide the LCNs as a "number=" field in comments in he
> XML output. Let me know if you would like a copy of the source.
>
> The original tv_grab_db_plus is here:
> https://sourceforge.net/projects/tvgrabeit/
>
> If you have managed to find proper documentation on the OpenTV EPG
> format, I would love to have a copy as that should allow me to make
> tv_grab_dvb_plus fully compatible. At the moment, there are a number
> of descriptor types that it does not know how to decode. I only have
> the basic DVB documentation to work from, nothing that is OpenTV
> specific. The original authors of tv_grab_dvb_plus do seem to have
> had more than that.

Stephen,

I am one of the original authors and there is no doco other than other
programs others wrote which we combined. Works for foxtelAU too but I
have no use for it any more. Scaping is easier with webgrab++.

I can give you commit/admin privs if you would like to
contribute/publish your changes. Some love would be appreciated for the
project.

The sourceforge path is wrong because at the time we could not figure
out how to rename it after we changed its name to tvgrabdvbplus.

I originally tried to port the opentv epg grabber to myth but lost
interest so it was never done.

As for doc I believe you can buy it from the opentv foundation (whatever
its name is) with a suitable NDA. I never did this.

Cheers

Mark

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Sat, 11 Jan 2020 00:08:54 +1100, you wrote:

>Stephen,
>
>I am one of the original authors and there is no doco other than other
>programs others wrote which we combined. Works for foxtelAU too but I
>have no use for it any more. Scaping is easier with webgrab++.
>
>I can give you commit/admin privs if you would like to
>contribute/publish your changes. Some love would be appreciated for the
>project.

The problem I have with committing changes is that mine are Sky NZ
specific, and I have no way to test for any of the other services to
see if I have broken anything for them.

>The sourceforge path is wrong because at the time we could not figure
>out how to rename it after we changed its name to tvgrabdvbplus.
>
>I originally tried to port the opentv epg grabber to myth but lost
>interest so it was never done.

It works fine externally to create XMLTV files, so in the past I did
not see any real benefit to making it part of the MythTV code. But
the ability to get extra data for channel scanning such as the LCNs
is, however, very useful in making MythTV scanning more user friendly.
That could be done using external XMLTV data also, if the XMLTV
specification was updated to add LCN data and anything else that might
be useful. When I added the LCN data to tv_grab_dvb_plus's output, I
found that there was no way to make it part of the official XMLTV
data, so I had to put it into an XML comment field. That is useful
for humans when reading the XMLTV file, but not so good for automating
things in a standard way in programs like MythTV.

>As for doc I believe you can buy it from the opentv foundation (whatever
>its name is) with a suitable NDA. I never did this.

I think that costs $1000s so I am not likely to want to spend that
much.

>Cheers
>
>Mark
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Hi Stephen,

Thanks for the stream! I've put it in my open-source stream analyzer
"DVB Inspector" and I do see at first sight a number of channels and
about one week of EPG guide data. This means that the EPG is not
encrypted and that, if it does not work with the current master, it
should be possible to fix it. There is also an original network ID so
that can be used to enable the OpenTV channel number detection
automatically for that network.

Mark's patch does also the Logical Channel Numbers; I do hope that I
did not make mistakes in merging. It would be great if you can do a
scan and see if it works . Output of "-v chanscan" and possibly also
"--loglevel=debug" would also be very useful.
Not really for the experts but for everybody who has not done a scan
in a long time there is now a "Channel Scanning How To" in the Wiki
that tries to talk the casual user through the scanning process. Note
that the "Scan OpenTV" option is not described yet....

Groetjes,
Klaas.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Hi Stephen,

I had a good look at the Optus D1 stream data and it looks remarkably
similar to the BSkyB streams on Astra-2 28E2 for
which the logical channel number parsing is already some time in mythtv master.
Looking at the stream data I think that if you enter a bouquet ID
(n.b. editable in the Video Source page) in
the range 25280 to 25283 then you get already the logical channel
numbers, also if you do not check the "Scan OpenTV" option.
It could be that these bouquets use the Sky format and that the other
bouquets, 25477 and 24656 to 24672, are OpenTV specific.
The guide data is in standard format on the standard PID, so it should
all work "out of the box".
I am interested in your test results!

Groetjes,
Klaas.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

> Looking at the stream data I think that if you enter a bouquet ID
> (n.b. editable in the Video Source page) in
> the range 25280 to 25283 then you get already the logical channel
> numbers, also if you do not check the "Scan OpenTV" option.

Correction: this should now work with the latest master, as it needs the check
on the original network ID that I have just now committed.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On Fri, 10 Jan 2020 20:46:17 +0100, you wrote:

>Hi Stephen,
>
>Thanks for the stream! I've put it in my open-source stream analyzer
>"DVB Inspector" and I do see at first sight a number of channels and
>about one week of EPG guide data. This means that the EPG is not
>encrypted and that, if it does not work with the current master, it
>should be possible to fix it. There is also an original network ID so
>that can be used to enable the OpenTV channel number detection
>automatically for that network.

Yes, the SkyNZ EPG is not encrypted, even for the encrypted channels.
There is normally a week of EPG available, although recently they have
started to provide 2 weeks ahead on their web site.

>Mark's patch does also the Logical Channel Numbers; I do hope that I
>did not make mistakes in merging. It would be great if you can do a
>scan and see if it works . Output of "-v chanscan" and possibly also
>"--loglevel=debug" would also be very useful.
>Not really for the experts but for everybody who has not done a scan
>in a long time there is now a "Channel Scanning How To" in the Wiki
>that tries to talk the casual user through the scanning process. Note
>that the "Scan OpenTV" option is not described yet....
>
>Groetjes,
>Klaas.

Is there a safe way of running master on the same PC as I am using for
my production v30 system? My satellite tuners are all on my
production box, except for one old one in my test PC. But the test PC
does not have an aerial connection to my dish normally. I can plug it
in via a 10 metre cable across the floor, but that only works if I am
at home to plug it in and at the moment it is summer in New Zealand
and I am at the beach on holiday until the 30th. So to do any testing
on master, I need to be able to run it in parallel on the production
box (Ubuntu 18.04). There are spare tuners I can use to do that - I
have 8 DVB-T2 and 8 DVB-S2 tuners and only need 5 of each for the
production system. But master would need to use a different database
and install location.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Hi Stephen,

>Is there a safe way of running master on the same PC as I am using for
>my production v30 system?

Not that I know of.
I will do more rework on this part of the code anyway.
I also want to add a "Stream server" tuner type, based on the "Demo
test recorder"
so that I can do a service scan from a recorded transport stream.
In a previous life I used to have hardware to do that....
When that is running I would appreciate a really long capture (DVD
size, 4GB or so).
Thanks and just enjoy life at the beach!

Groetjes,
Klaas.
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 11/01/2020 10:47, Klaas de Waal wrote:
> Hi Stephen,
>
>> Is there a safe way of running master on the same PC as I am using for
>> my production v30 system?
>
> Not that I know of.

there is, it's just not trivial

You have to compile each version your self, and point the installation
directory to a non standard path. I use
- /usr/local/myth-git (for master)
- /usr/local/myth-prod (for prod)

then (at least the backend) needs a wrapper script like this
(and any other binaries you plan to run. eg setup, frontend)

~~~
#!/bin/sh

PATH=/usr/local/myth-prod/bin:$PATH
LD_LIBRARY_PATH=/usr/local/myth-prod/lib:$LD_LIBRARY_PATH
MYTHCONFDIR=$HOME/.mythtv-prod/
PYTHONPATH=/usr/local/myth-prod/lib/python2.7/site-packages

export PATH LD_LIBRARY_PATH MYTHCONFDIR PYTHONPATH

exec /usr/local/myth-prod/bin/mythbackend $@
~~~

which sets up all the relevant paths for the different bits.

The key one is MYTHCONFDIR, which points the configuration
to a different directory, thus keeping them separate.

The only other thing i've done, is on my dev instance, set
the backend port from :6544 -> :7544 so i can have both running
at the same time.


Regards
Stuart
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org