Mailing List Archive

On Sat, 8 Feb 2020 at 23:35, Hans Dingemans <jpldingemans@gmail.com> wrote:
>
> Hi all,
>
> First of all big thanks to all of you for this great software; I have been using Mythtv for years, very happy with it!!! However, as an enthusiastic open-source developer I always seek possibilities to improve and to contribute.
>
> Apart from my production server I now am experimenting with frontend + backend on a Raspberry Pi, trying to get it so far that it can play HLS streams from websites like https://watchnewslive.tv .
> A problem I encountered is that when trying to load the EPG for all those stations, mythfilldatabase uses too much memory and freezes up the (4Gb-version of the) Pi.
> Turns out that the entire xml-file is kept in memory multiple times. I made a patch which reduces memory usage 6 times.

I don't have a need for mythfilldatabase via xml - but I'm assuming
you are referring to the patches in this pull request? :-

https://github.com/MythTV/mythtv/pull/189

Regards
Mark
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

:

> On Sat, 8 Feb 2020 at 23:35, Hans Dingemans <jpldingemans@gmail.com>
> wrote:
> >
> > Hi all,
> >
> > First of all big thanks to all of you for this great software; I have
> been using Mythtv for years, very happy with it!!! However, as an
> enthusiastic open-source developer I always seek possibilities to improve
> and to contribute.
> >
> > Apart from my production server I now am experimenting with frontend +
> backend on a Raspberry Pi, trying to get it so far that it can play HLS
> streams from websites like https://watchnewslive.tv .
> > A problem I encountered is that when trying to load the EPG for all
> those stations, mythfilldatabase uses too much memory and freezes up the
> (4Gb-version of the) Pi.
> > Turns out that the entire xml-file is kept in memory multiple times. I
> made a patch which reduces memory usage 6 times.
>
> I don't have a need for mythfilldatabase via xml - but I'm assuming
> you are referring to the patches in this pull request? :-
>
> https://github.com/MythTV/mythtv/pull/189
>
> Regards
> Mark
>
That is correct !

>
>

On 08/02/2020 23:34, Hans Dingemans wrote:
> Hi all,
>
> First of all big thanks to all of you for this great software; I have
> been using Mythtv for years, very happy with it!!! However, as an
> enthusiastic open-source developer I always seek possibilities to
> improve and to contribute.
>
> Apart from my production server I now am experimenting with frontend +
> backend on a Raspberry Pi, trying to get it so far that it can play
> HLS streams from websites like https://watchnewslive.tv .
> A problem I encountered is that when trying to load the EPG for all
> those stations, mythfilldatabase uses too much memory and freezes up
> the (4Gb-version of the) Pi.
> Turns out that the entire xml-file is kept in memory multiple times. I
> made a patch which reduces memory usage 6 times.
>
> The nice thing about these offline software is you can test it easily;
> I just backup my sql database, load a test xml file with "production"
> mythfilldatabase, dump the database, restore it to its old version,
> reload the same xml file with my patched mythfilldatabase, dump it
> again, and do a compare against the dumps.
>
> To make sure all the different xmltv-dialects out there are working, I
> am looking for test xml-files from allover the world; the more
> diverse, the better.
> Please send them to my gmail address, if it is too large, send me a pm
> so we can see how to transfer.
>
> Thank you!
>
> Hans.


I did some testing on a Pi4 (4GB) using external xmltv data from
Schedules Direct tvgrab_zz_sdjson a while ago see
https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702


As with any other mythbackend with 4 GB or even 8 GB of RAM, the
mythfilldatabase option of --no-allatonce  dramatically reduced memory
usage when there are lots of channels, e.g. Astra 28E2 satellite.


Mike

_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

> On 08/02/2020 23:34, Hans Dingemans wrote:
> > Hi all,
> >
> > First of all big thanks to all of you for this great software; I have
> > been using Mythtv for years, very happy with it!!! However, as an
> > enthusiastic open-source developer I always seek possibilities to
> > improve and to contribute.
> >
> > Apart from my production server I now am experimenting with frontend +
> > backend on a Raspberry Pi, trying to get it so far that it can play
> > HLS streams from websites like https://watchnewslive.tv .
> > A problem I encountered is that when trying to load the EPG for all
> > those stations, mythfilldatabase uses too much memory and freezes up
> > the (4Gb-version of the) Pi.
> > Turns out that the entire xml-file is kept in memory multiple times. I
> > made a patch which reduces memory usage 6 times.
> >
> > The nice thing about these offline software is you can test it easily;
> > I just backup my sql database, load a test xml file with "production"
> > mythfilldatabase, dump the database, restore it to its old version,
> > reload the same xml file with my patched mythfilldatabase, dump it
> > again, and do a compare against the dumps.
> >
> > To make sure all the different xmltv-dialects out there are working, I
> > am looking for test xml-files from allover the world; the more
> > diverse, the better.
> > Please send them to my gmail address, if it is too large, send me a pm
> > so we can see how to transfer.
> >
> > Thank you!
> >
> > Hans.
>
>
> I did some testing on a Pi4 (4GB) using external xmltv data from
> Schedules Direct tvgrab_zz_sdjson a while ago see
>
> https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702
>
>
> As with any other mythbackend with 4 GB or even 8 GB of RAM, the
> mythfilldatabase option of --no-allatonce dramatically reduced memory
> usage when there are lots of channels, e.g. Astra 28E2 satellite.
>
>
> Mike


What commandline are you using tot start mythfilldatabase?

I use the --xmlfile parameter, and then the --no-allatonce parameter is
totally ignored (without reporting anything).

But even when you are using that parameter, all EPG info is kept in RAM
multiple times until mythfilldatabase exits, which is totally unnecessary...

On 09/02/2020 10:47, Hans Dingemans wrote:
>
> On 08/02/2020 23:34, Hans Dingemans wrote:
> > Hi all,
> >
> > First of all big thanks to all of you for this great software; I
> have
> > been using Mythtv for years, very happy with it!!! However, as an
> > enthusiastic open-source developer I always seek possibilities to
> > improve and to contribute.
> >
> > Apart from my production server I now am experimenting with
> frontend +
> > backend on a Raspberry Pi, trying to get it so far that it can play
> > HLS streams from websites like https://watchnewslive.tv .
> > A problem I encountered is that when trying to load the EPG for all
> > those stations, mythfilldatabase uses too much memory and
> freezes up
> > the (4Gb-version of the) Pi.
> > Turns out that the entire xml-file is kept in memory multiple
> times. I
> > made a patch which reduces memory usage 6 times.
> >
> > The nice thing about these offline software is you can test it
> easily;
> > I just backup my sql database, load a test xml file with
> "production"
> > mythfilldatabase, dump the database, restore it to its old version,
> > reload the same xml file with my patched mythfilldatabase, dump it
> > again, and do a compare against the dumps.
> >
> > To make sure all the different xmltv-dialects out there are
> working, I
> > am looking for test xml-files from allover the world; the more
> > diverse, the better.
> > Please send them to my gmail address, if it is too large, send
> me a pm
> > so we can see how to transfer.
> >
> > Thank you!
> >
> > Hans.
>
>
> I did some testing on a Pi4 (4GB) using external xmltv data from
> Schedules Direct tvgrab_zz_sdjson a while ago see
> https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702
>
>
> As with any other mythbackend with 4 GB or even 8 GB of RAM, the
> mythfilldatabase option of --no-allatonce  dramatically reduced
> memory
> usage when there are lots of channels, e.g. Astra 28E2 satellite.
>
>
> Mike
>
>
> What commandline are you using tot start mythfilldatabase?
>
> I use the --xmlfile parameter, and then the --no-allatonce parameter
> is totally ignored (without reporting anything).
>
> But even when you are using that parameter, all EPG info is kept in
> RAM multiple times until mythfilldatabase exits, which is totally
> unnecessary...


Not using any command line to run mythfilldatabase.

I am using the standard built-in feature of mythbackend which runs
mythfilldatabase command automatically at the appropriate time, using
the specified grabber from VideoSource configuration, the --no-allatonce
tells mythfilldatabase to request data 1 day at a time from the grabber
(in my case tv_grab_zz_sdjson) instead of all data, which could be up to
3 weeks. Using --no-allatonce the grabber is run multiple times until no
data is returned.

To illustrate the difference, the data file size for Sky Satellite
(Astra 28E2), using grabber tv_grab_zz_sdjson for all possible channels,
results in file size of 211 MB without --no-allatonce and a file size of
14 MB with --no-allatonce.


In mythtv-setup:
1. General>Programme Schedule Downloading Options
    right arrow to Automatically update programme listings then right
arrow to Listing data arguments, this where --no-allatonce goes.


Out of interest which xmltv grabber are you using ?


Mike

On Sun, Feb 9, 2020 at 7:51 AM Mike Bibbings <mike.bibbings@gmail.com>
wrote:

> On 09/02/2020 10:47, Hans Dingemans wrote:
>
>
> On 08/02/2020 23:34, Hans Dingemans wrote:
>> > Hi all,
>> >
>> > First of all big thanks to all of you for this great software; I have
>> > been using Mythtv for years, very happy with it!!! However, as an
>> > enthusiastic open-source developer I always seek possibilities to
>> > improve and to contribute.
>> >
>> > Apart from my production server I now am experimenting with frontend +
>> > backend on a Raspberry Pi, trying to get it so far that it can play
>> > HLS streams from websites like https://watchnewslive.tv .
>> > A problem I encountered is that when trying to load the EPG for all
>> > those stations, mythfilldatabase uses too much memory and freezes up
>> > the (4Gb-version of the) Pi.
>> > Turns out that the entire xml-file is kept in memory multiple times. I
>> > made a patch which reduces memory usage 6 times.
>> >
>> > The nice thing about these offline software is you can test it easily;
>> > I just backup my sql database, load a test xml file with "production"
>> > mythfilldatabase, dump the database, restore it to its old version,
>> > reload the same xml file with my patched mythfilldatabase, dump it
>> > again, and do a compare against the dumps.
>> >
>> > To make sure all the different xmltv-dialects out there are working, I
>> > am looking for test xml-files from allover the world; the more
>> > diverse, the better.
>> > Please send them to my gmail address, if it is too large, send me a pm
>> > so we can see how to transfer.
>> >
>> > Thank you!
>> >
>> > Hans.
>>
>>
>> I did some testing on a Pi4 (4GB) using external xmltv data from
>> Schedules Direct tvgrab_zz_sdjson a while ago see
>>
>> https://forum.mythtv.org/viewtopic.php?f=46&t=3232&sid=8706e6ce203ad5b3a6d95c291add157c#p15702
>>
>>
>> As with any other mythbackend with 4 GB or even 8 GB of RAM, the
>> mythfilldatabase option of --no-allatonce dramatically reduced memory
>> usage when there are lots of channels, e.g. Astra 28E2 satellite.
>>
>>
>> Mike
>
>
> What commandline are you using tot start mythfilldatabase?
>
> I use the --xmlfile parameter, and then the --no-allatonce parameter is
> totally ignored (without reporting anything).
>
> But even when you are using that parameter, all EPG info is kept in RAM
> multiple times until mythfilldatabase exits, which is totally unnecessary...
>
>
> Not using any command line to run mythfilldatabase.
>
> I am using the standard built-in feature of mythbackend which runs
> mythfilldatabase command automatically at the appropriate time, using the
> specified grabber from VideoSource configuration, the --no-allatonce tells
> mythfilldatabase to request data 1 day at a time from the grabber (in my
> case tv_grab_zz_sdjson) instead of all data, which could be up to 3 weeks.
> Using --no-allatonce the grabber is run multiple times until no data is
> returned.
>
> To illustrate the difference, the data file size for Sky Satellite (Astra
> 28E2), using grabber tv_grab_zz_sdjson for all possible channels, results
> in file size of 211 MB without --no-allatonce and a file size of 14 MB with
> --no-allatonce.
>
>
> In mythtv-setup:
> 1. General>Programme Schedule Downloading Options
> right arrow to Automatically update programme listings then right
> arrow to Listing data arguments, this where --no-allatonce goes.
>
>
> Out of interest which xmltv grabber are you using ?
>
>
> Mike
>
Very interesting Mike, I hadn't discovered that built-in option yet!
Certainly are going to look at that!

I am using zap2xml for US channels and tvgrabpyAPI for Dutch channels.

On 2/8/20 6:34 PM, Hans Dingemans wrote:
> Hi all,
>
> First of all big thanks to all of you for this great software; I have
> been using Mythtv for years, very happy with it!!! However, as an
> enthusiastic open-source developer I always seek possibilities to
> improve and to contribute.
>
> Apart from my production server I now am experimenting with frontend +
> backend on a Raspberry Pi, trying to get it so far that it can play
> HLS streams from websites like https://watchnewslive.tv .
> A problem I encountered is that when trying to load the EPG for all
> those stations, mythfilldatabase uses too much memory and freezes up
> the (4Gb-version of the) Pi.
> Turns out that the entire xml-file is kept in memory multiple times. I
> made a patch which reduces memory usage 6 times.
>
> The nice thing about these offline software is you can test it easily;
> I just backup my sql database, load a test xml file with "production"
> mythfilldatabase, dump the database, restore it to its old version,
> reload the same xml file with my patched mythfilldatabase, dump it
> again, and do a compare against the dumps.
>
> To make sure all the different xmltv-dialects out there are working, I
> am looking for test xml-files from allover the world; the more
> diverse, the better.
> Please send them to my gmail address, if it is too large, send me a pm
> so we can see how to transfer.
>
> Thank you!
>
> Hans.
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
Did you see the solution that I use? I run 3 days at a time in a script.
If needed you can reduce it to 1 day at a time. See link below
https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage

Peter

On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv@gmail.com> wrote:

>
>
> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>
> Hi all,
>
> First of all big thanks to all of you for this great software; I have been
> using Mythtv for years, very happy with it!!! However, as an enthusiastic
> open-source developer I always seek possibilities to improve and to
> contribute.
>
> Apart from my production server I now am experimenting with frontend +
> backend on a Raspberry Pi, trying to get it so far that it can play HLS
> streams from websites like https://watchnewslive.tv .
> A problem I encountered is that when trying to load the EPG for all those
> stations, mythfilldatabase uses too much memory and freezes up the
> (4Gb-version of the) Pi.
> Turns out that the entire xml-file is kept in memory multiple times. I
> made a patch which reduces memory usage 6 times.
>
> The nice thing about these offline software is you can test it easily; I
> just backup my sql database, load a test xml file with "production"
> mythfilldatabase, dump the database, restore it to its old version, reload
> the same xml file with my patched mythfilldatabase, dump it again, and do a
> compare against the dumps.
>
> To make sure all the different xmltv-dialects out there are working, I am
> looking for test xml-files from allover the world; the more diverse, the
> better.
> Please send them to my gmail address, if it is too large, send me a pm so
> we can see how to transfer.
>
> Thank you!
>
> Hans.
>
> _______________________________________________
> mythtv-dev mailing listmythtv-dev@mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
> Did you see the solution that I use? I run 3 days at a time in a script.
> If needed you can reduce it to 1 day at a time. See link below
> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>
> Peter
>
Hi Peter,

Yes I saw the solutions that you use, that post in fact showed me the
--no-allatonce parameter. But then I found out that parameter is not
working when you use the --xmlfile parameter (this seems undocumented).
So then I looked into the code, because I didn't understand why
Mythfilldatabase was using substantially more memory than the xml-file it
was processing; turned out one of the reasons is because Mythfilldatabase
uses QDomDocument to parse and store the XML data that is read; according
to QDomDocument documentation this object is not meant to handle large XML
files; it recommends to use QXmlStreamReader in these situations. So I made
a patch that actually does this: https://code.mythtv.org/trac/ticket/13517.
For some reason it reduces memory usage by a factor 6 (in my situation),
where I was expecting "only" a factor 2. Also it reduces run-time somewhat.

I hope I don't offend you if I say that I look at "chopping up" the
epg-reading process in several days (either 1 day or 3 days chunks) as a
workaround, and that reducing the excessive memory usage of
Mythfilldatabase is solving the root cause. Everybody would profit of it,
whether you are chopping the reading process by your script or by use
-no-allatonce, or when you process your xml-files as a whole.

That is why I am requesting xml files from all over the world (already
received some good stuff!), to do excessive testing, and then ask one of
the developers to commit my patch. I saw you committed the -no-allataonce
parameter, would you be the designated developer for this?

Best regards, Hans.

On 2/9/20 10:59 AM, Hans Dingemans wrote:
> On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv@gmail.com
> <mailto:pb.mythtv@gmail.com>> wrote:
>
>
>
> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>> Hi all,
>>
>> First of all big thanks to all of you for this great software; I
>> have been using Mythtv for years, very happy with it!!! However,
>> as an enthusiastic open-source developer I always seek
>> possibilities to improve and to contribute.
>>
>> Apart from my production server I now am experimenting with
>> frontend + backend on a Raspberry Pi, trying to get it so far
>> that it can play HLS streams from websites like
>> https://watchnewslive.tv .
>> A problem I encountered is that when trying to load the EPG for
>> all those stations, mythfilldatabase uses too much memory and
>> freezes up the (4Gb-version of the) Pi.
>> Turns out that the entire xml-file is kept in memory multiple
>> times. I made a patch which reduces memory usage 6 times.
>>
>> The nice thing about these offline software is you can test it
>> easily; I just backup my sql database, load a test xml file with
>> "production" mythfilldatabase, dump the database, restore it to
>> its old version, reload the same xml file with my patched
>> mythfilldatabase, dump it again, and do a compare against the dumps.
>>
>> To make sure all the different xmltv-dialects out there are
>> working, I am looking for test xml-files from allover the world;
>> the more diverse, the better.
>> Please send them to my gmail address, if it is too large, send me
>> a pm so we can see how to transfer.
>>
>> Thank you!
>>
>> Hans.
>>
>> _______________________________________________
>> mythtv-dev mailing list
>> mythtv-dev@mythtv.org <mailto:mythtv-dev@mythtv.org>
>> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
>> http://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums:https://forum.mythtv.org
> Did you see the solution that I use? I run 3 days at a time in a
> script. If needed you can reduce it to 1 day at a time. See link below
> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>
> Peter
>
> Hi Peter,
>
> Yes I saw the solutions that you use, that post in fact showed me the
> --no-allatonce parameter. But then I found out that parameter is not
> working when you use the --xmlfile parameter (this seems undocumented).
> So then I looked into the code, because I didn't understand why
> Mythfilldatabase was using substantially more memory than the xml-file
> it was processing; turned out one of the reasons is because
> Mythfilldatabase uses QDomDocument to parse and store the XML data
> that is read; according to QDomDocument documentation this object is
> not meant to handle large XML files; it recommends to use
> QXmlStreamReader in these situations. So I made a patch that actually
> does this: https://code.mythtv.org/trac/ticket/13517. For some reason
> it reduces memory usage by a factor 6 (in my situation), where I was
> expecting "only" a factor 2. Also it reduces run-time somewhat.
>
> I hope I don't offend you if I say that I look at "chopping up" the
> epg-reading process in several days (either 1 day or 3 days chunks) as
> a workaround, and that reducing the excessive memory usage of
> Mythfilldatabase is solving the root cause. Everybody would profit of
> it, whether you are chopping the reading process by your script or by
> use -no-allatonce, or when you process your xml-files as a whole.
>
> That is why I am requesting xml files from all over the world (already
> received some good stuff!), to do excessive testing, and then ask one
> of the developers to commit my patch. I saw you committed the
> -no-allataonce parameter, would you be the designated developer for this?
>
> Best regards, Hans.
>
> _______________________________________________
> mythtv-dev mailing list
> mythtv-dev@mythtv.org
> http://lists.mythtv.org/mailman/listinfo/mythtv-dev
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org


Yes sure I can commit it for you. However I would prefer a patch, if you
attach a diff file to the ticket when you are happy with it I will take
a look and commit if it looks good. I will also get you an xml file for
testing.

Peter

Op zo 9 feb. 2020 15:05 schreef Peter Bennett <pb.mythtv@gmail.com>:

>
>
> On 2/9/20 10:59 AM, Hans Dingemans wrote:
>
> On Sun, Feb 9, 2020 at 10:20 AM Peter Bennett <pb.mythtv@gmail.com> wrote:
>
>>
>>
>> On 2/8/20 6:34 PM, Hans Dingemans wrote:
>>
>> Hi all,
>>
>> First of all big thanks to all of you for this great software; I have
>> been using Mythtv for years, very happy with it!!! However, as an
>> enthusiastic open-source developer I always seek possibilities to improve
>> and to contribute.
>>
>> Apart from my production server I now am experimenting with frontend +
>> backend on a Raspberry Pi, trying to get it so far that it can play HLS
>> streams from websites like https://watchnewslive.tv .
>> A problem I encountered is that when trying to load the EPG for all those
>> stations, mythfilldatabase uses too much memory and freezes up the
>> (4Gb-version of the) Pi.
>> Turns out that the entire xml-file is kept in memory multiple times. I
>> made a patch which reduces memory usage 6 times.
>>
>> The nice thing about these offline software is you can test it easily; I
>> just backup my sql database, load a test xml file with "production"
>> mythfilldatabase, dump the database, restore it to its old version, reload
>> the same xml file with my patched mythfilldatabase, dump it again, and do a
>> compare against the dumps.
>>
>> To make sure all the different xmltv-dialects out there are working, I am
>> looking for test xml-files from allover the world; the more diverse, the
>> better.
>> Please send them to my gmail address, if it is too large, send me a pm so
>> we can see how to transfer.
>>
>> Thank you!
>>
>> Hans.
>>
>> _______________________________________________
>> mythtv-dev mailing listmythtv-dev@mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
>> MythTV Forums: https://forum.mythtv.org
>>
>> Did you see the solution that I use? I run 3 days at a time in a script.
>> If needed you can reduce it to 1 day at a time. See link below
>> https://www.mythtv.org/wiki/Mythfilldatabase#Memory_Usage
>>
>> Peter
>>
> Hi Peter,
>
> Yes I saw the solutions that you use, that post in fact showed me the
> --no-allatonce parameter. But then I found out that parameter is not
> working when you use the --xmlfile parameter (this seems undocumented).
> So then I looked into the code, because I didn't understand why
> Mythfilldatabase was using substantially more memory than the xml-file it
> was processing; turned out one of the reasons is because Mythfilldatabase
> uses QDomDocument to parse and store the XML data that is read; according
> to QDomDocument documentation this object is not meant to handle large XML
> files; it recommends to use QXmlStreamReader in these situations. So I made
> a patch that actually does this: https://code.mythtv.org/trac/ticket/13517.
> For some reason it reduces memory usage by a factor 6 (in my situation),
> where I was expecting "only" a factor 2. Also it reduces run-time somewhat.
>
> I hope I don't offend you if I say that I look at "chopping up" the
> epg-reading process in several days (either 1 day or 3 days chunks) as a
> workaround, and that reducing the excessive memory usage of
> Mythfilldatabase is solving the root cause. Everybody would profit of it,
> whether you are chopping the reading process by your script or by use
> -no-allatonce, or when you process your xml-files as a whole.
>
> That is why I am requesting xml files from all over the world (already
> received some good stuff!), to do excessive testing, and then ask one of
> the developers to commit my patch. I saw you committed the -no-allataonce
> parameter, would you be the designated developer for this?
>
> Best regards, Hans.
>
> _______________________________________________
> mythtv-dev mailing listmythtv-dev@mythtv.orghttp://lists.mythtv.org/mailman/listinfo/mythtv-devhttp://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org
>
>
>
> Yes sure I can commit it for you. However I would prefer a patch, if you
> attach a diff file to the ticket when you are happy with it I will take a
> look and commit if it looks good. I will also get you an xml file for
> testing.
> Peter
>

Great!!

On 02/09/2020 07:57 AM, Hans Dingemans wrote:
>
> I am using zap2xml for US channels

FWIW, zap2xml is in violation of Zap2It's Terms of Use and copyright.
See https://lists.gt.net/mythtv/users/612021#612021 . Mentioning this,
as in the linked post since, due to the program's author's deception,
most users have no idea that using that program is in violation of the
ToU/copyright.

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Op ma 10 feb. 2020 07:44 schreef Michael T. Dean <mtdean@thirdcontact.com>:

> On 02/09/2020 07:57 AM, Hans Dingemans wrote:
> >
> > I am using zap2xml for US channels
>
> FWIW, zap2xml is in violation of Zap2It's Terms of Use and copyright.
> See https://lists.gt.net/mythtv/users/612021#612021 . Mentioning this,
> as in the linked post since, due to the program's author's deception,
> most users have no idea that using that program is in violation of the
> ToU/copyright.
>

I never realised this; thanks for the headsup Michael!

Hans.

>

On Mon, Feb 10, 2020 at 7:44 AM Michael T. Dean <mtdean@thirdcontact.com>
wrote:

> On 02/09/2020 07:57 AM, Hans Dingemans wrote:
> >
> > I am using zap2xml for US channels
>
> FWIW, zap2xml is in violation of Zap2It's Terms of Use and copyright.
> See https://lists.gt.net/mythtv/users/612021#612021 . Mentioning this,
> as in the linked post since, due to the program's author's deception,
> most users have no idea that using that program is in violation of the
> ToU/copyright.
>
> Mike
>

Mike, I looked into this, and tvguide.com indeed has a policy that forbids
you to download their information and store it in a database. But zap2it.com
does not seem to have an (active) terms of use policy; that is, I couldn't
find it, and neither could Google.

What is the source of your information?

Hans.

On 02/11/2020 09:40 AM, Hans Dingemans wrote:
> On Mon, Feb 10, 2020 at 7:44 AM Michael T. Dean wrote:
>> On 02/09/2020 07:57 AM, Hans Dingemans wrote:
>>> I am using zap2xml for US channels
>> FWIW, zap2xml is in violation of Zap2It's Terms of Use and copyright.
>> See https://lists.gt.net/mythtv/users/612021#612021 . Mentioning this,
>> as in the linked post since, due to the program's author's deception,
>> most users have no idea that using that program is in violation of the
>> ToU/copyright.
> Mike, I looked into this, and tvguide.com indeed has a policy that forbids
> you to download their information and store it in a database. But zap2it.com
> does not seem to have an (active) terms of use policy; that is, I couldn't
> find it, and neither could Google.
>
> What is the source of your information?

History.

At the bottom of the page is:

Contact Us Copyright 2020 Tribune Media Entertainment Powered by
Gracenote, Inc. Privacy Policy

Where the "Gracenote, Inc." is a link to their home page, which includes
a link to https://www.gracenote.com/terms-of-use/ . The "Privacy
Policy" link sends you to http://privacy.tribunemedia.com/ , which has a
link, "TERMS OF SERVICE" to
http://www.tribunemedia.com/terms-of-service/ , which gives a 404. This
all seems to be due to the "Zap2it TV listings are back"/site redesign
and is indicative that it's not fully complete, yet.

Note that Tribune Media is the owner of the listings used by Schedules
Direct. See https://www.schedulesdirect.org/faq , specifically, "What
about alternatives to licensing data?" There's also mention of Zap2It
Labs in "Why are you charging for something that you used to provide for
free?" where Zap2It Labs was something that Tribune Media set up to
provide free listings for approved open-source software usage after
requesting that "we" (XMLTV users) stop screen scraping their site
because doing so was destroying their servers--almost to the extend of
DoS'ing them. Due to changes over the years, they were unable to keep
Zap2It Labs running, but were able to provide licensed listings to
Schedules Direct at a reduced licensing rate to allow the community to
continue to get listings which are paid for by Schedules Direct
membership fees--which are significantly less expensive ($25/yr or $6
for 2 months) than those for TiVo and other services (even cable
companies charge much more for DVR listings fees). (Oh, and around the
time they set up Zap2It Labs, they actually added a ToU page to Zap2It's
TV listings page that forbid bots/scraping/...)

If you search the list archives, you can find many discussions of
zap2xml and mc2xml all pointing to the fact that they download and copy
copyrighted listings data compilations and show summaries and such.
There is an approved/legal/licensed approach to getting that data (and
much more/better quality data, actually), by using Schedules Direct (
https://schedulesdirect.org ).

But don't take my word for it. Feel free to contact Tribune Media and
ask them what they think about zap2xml's screen scraping their website
to make a copy of their copyrighted data using a method that doesn't
access any of the ads that support their site. (And after getting an
answer, you may want to tell them they really need to make their ToS/ToU
prominent--or at least findable--on their new site.)

Mike
_______________________________________________
mythtv-dev mailing list
mythtv-dev@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-dev
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org