Mailing List Archive

On 9/3/23 13:56, Jan Ceuleers wrote:
> Dear all,
>
> I am a mythweb user of many years, but clearly I will need to start
> using the new alternative on port 6544 some time soon.
>
> So I took a look and I noticed that the Video Gallery takes very long to
> appear. Meanwhile the backend is maxed out on one core.
>
> I have quite an extensive collection of movies and series, consisting of
> 5429 files that are located on 4 SSDs across multiple levels of
> directories underneath /mnt/disk?/mythtv/videos/. As I said these are
> SSDs (SATA-connected), so quite fast. The backend's CPU is a 4GHz Intel
> i3-8350K, and the box also has plenty of RAM (32GB).
>
> Should I report this as a bug?
>
> Many thanks, Jan
> _______________________________________________

Thanks for testing this. I have about 1500 videos (on hard drive) and it
takes between 1 and 2 seconds. With 5500 videos I would expect no more
than 7 seconds.

I can investigate using a lazy load for videos. That may conflict with
the directory listing scheme I have, where directories are listed and
you can drill down.

Do you use directories?

When it takes a long time, how many rows are reported (at the top level
each direcory is a row)?

Does clicking on a directory to see its contents also take a long time?

Does clicking back on Videos at the top after looking at a subdirectory
also take a long time?

Peter
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 9/3/23 17:08, Peter Bennett wrote:
>
> On 9/3/23 13:56, Jan Ceuleers wrote:
>> Dear all,
>>
>> I am a mythweb user of many years, but clearly I will need to start
>> using the new alternative on port 6544 some time soon.
>>
>> So I took a look and I noticed that the Video Gallery takes very long to
>> appear. Meanwhile the backend is maxed out on one core.
>>
>> I have quite an extensive collection of movies and series, consisting of
>> 5429 files that are located on 4 SSDs across multiple levels of
>> directories underneath /mnt/disk?/mythtv/videos/. As I said these are
>> SSDs (SATA-connected), so quite fast. The backend's CPU is a 4GHz Intel
>> i3-8350K, and the box also has plenty of RAM (32GB).
>>
>> Should I report this as a bug?
>>
>> Many thanks, Jan
>> _______________________________________________
>
> Thanks for testing this. I have about 1500 videos (on hard drive) and
> it takes between 1 and 2 seconds. With 5500 videos I would expect no
> more than 7 seconds.
>
> I can investigate using a lazy load for videos. That may conflict with
> the directory listing scheme I have, where directories are listed and
> you can drill down.
>
> Do you use directories?
>
> When it takes a long time, how many rows are reported (at the top
> level each direcory is a row)?
>
> Does clicking on a directory to see its contents also take a long time?
>
> Does clicking back on Videos at the top after looking at a
> subdirectory also take a long time?
>
> Peter

I don't know why you have this problem. I loaded 7531 videos on to my
laptop running mythbackend. Running the browser on the same laptop, it
takes 3 seconds to refresh the 242 rows (directories and files) in the
root directory of the storage group. Clicking "Flatten Directory so that
it displays all 7531 rows at once takes 7 seconds.

This laptop has a i5-9300H CPU @ 2.40GHz and 16 GB ram.

Paasmark benchmarks for the CPUs

i3-8350K: 6928

i5-9300H : 7658

Yours is slightly slower than mine, but not significantly, and I have
more files in my system.

Are you running the browser on the same machine as the backend? Maybe
the browser is on a slow machine?

Peter

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 03/09/2023 23:08, Peter Bennett wrote:
> Thanks for testing this. I have about 1500 videos (on hard drive) and
> it takes between 1 and 2 seconds. With 5500 videos I would expect no
> more than 7 seconds.

I just timed it again: it took 50 seconds.

> I can investigate using a lazy load for videos. That may conflict with
> the directory listing scheme I have, where directories are listed and
> you can drill down.
>
> Do you use directories?

I do use directories, yes. Lots of them:

* The top level is alphabetic in groups (0-9, A-B, C-D, E-F, G-H, I-J,
K-L, M, N-O-P-Q, R-S, T, U-V-W-X-Y-Z). So 12 directories at the top
level.
* The above directories contain movies, but also subdirectories (for
TV box sets).
* The TV box set directories can contain subdirectories for each
season. Single-season series don't have these subdirectories.

> When it takes a long time, how many rows are reported (at the top
> level each direcory is a row)?
Not sure what you're asking. Is there an incantation I should run to
produce the answer?
>
> Does clicking on a directory to see its contents also take a long time?
No, that is instant.
>
> Does clicking back on Videos at the top after looking at a
> subdirectory also take a long time?

No.

Many thanks, Jan

On 9/6/23 12:54, Jan Ceuleers wrote:
> On 03/09/2023 23:08, Peter Bennett wrote:
>> Thanks for testing this. I have about 1500 videos (on hard drive) and
>> it takes between 1 and 2 seconds. With 5500 videos I would expect no
>> more than 7 seconds.
>
> I just timed it again: it took 50 seconds.
>
>> I can investigate using a lazy load for videos. That may conflict
>> with the directory listing scheme I have, where directories are
>> listed and you can drill down.
>>
>> Do you use directories?
>
> I do use directories, yes. Lots of them:
>
> * The top level is alphabetic in groups (0-9, A-B, C-D, E-F, G-H,
> I-J, K-L, M, N-O-P-Q, R-S, T, U-V-W-X-Y-Z). So 12 directories at
> the top level.
> * The above directories contain movies, but also subdirectories (for
> TV box sets).
> * The TV box set directories can contain subdirectories for each
> season. Single-season series don't have these subdirectories.
>
>> When it takes a long time, how many rows are reported (at the top
>> level each direcory is a row)?
> Not sure what you're asking. Is there an incantation I should run to
> produce the answer?
>>
>> Does clicking on a directory to see its contents also take a long time?
> No, that is instant.
>>
>> Does clicking back on Videos at the top after looking at a
>> subdirectory also take a long time?
>
> No.
>
> Many thanks, Jan
>
>
Please try this, to see if the slowdown is on the backend:
curl "http://localhost:6544/Video/GetVideoList?Sort=FileName" >/dev/null

On my laptop it reports time spent 0:00:01 with over 7000 videos in my
database.

If you are running your browser on a different computer from the
backend, then try from that computer:
curl "http://backend:6544/Video/GetVideoList?Sort=FileName" >/dev/null

Look at the Time Spent. This will tell if it is a slowdown in the network.

Peter

On 06/09/2023 20:26, Peter Bennett wrote:

>
> On 9/6/23 12:54, Jan Ceuleers wrote:
>> On 03/09/2023 23:08, Peter Bennett wrote:
>>> Thanks for testing this. I have about 1500 videos (on hard drive)
>>> and it takes between 1 and 2 seconds. With 5500 videos I would
>>> expect no more than 7 seconds.
>>
>> I just timed it again: it took 50 seconds.
>>
>>> I can investigate using a lazy load for videos. That may conflict
>>> with the directory listing scheme I have, where directories are
>>> listed and you can drill down.
>>>
>>> Do you use directories?
>>
>> I do use directories, yes. Lots of them:
>>
>> * The top level is alphabetic in groups (0-9, A-B, C-D, E-F, G-H,
>> I-J, K-L, M, N-O-P-Q, R-S, T, U-V-W-X-Y-Z). So 12 directories at
>> the top level.
>> * The above directories contain movies, but also subdirectories
>> (for TV box sets).
>> * The TV box set directories can contain subdirectories for each
>> season. Single-season series don't have these subdirectories.
>>
>>> When it takes a long time, how many rows are reported (at the top
>>> level each direcory is a row)?
>> Not sure what you're asking. Is there an incantation I should run to
>> produce the answer?
>>>
>>> Does clicking on a directory to see its contents also take a long time?
>> No, that is instant.
>>>
>>> Does clicking back on Videos at the top after looking at a
>>> subdirectory also take a long time?
>>
>> No.
>>
>> Many thanks, Jan
>>
>>
> Please try this, to see if the slowdown is on the backend:
> curl "http://localhost:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> On my laptop it reports time spent 0:00:01 with over 7000 videos in my
> database.
>
> If you are running your browser on a different computer from the
> backend, then try from that computer:
> curl "http://backend:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> Look at the Time Spent. This will tell if it is a slowdown in the network.
>
> Peter
>
>

You should also be able to use your browsers debug tools to see how long
each network operation takes as well.


Big disclosure I haven't been able to test the latest WebApp but when
Stuart and I started the initial development the WebApp was very fast
but as we added more stuff it started to slow down dramatically for me
but not for Stuart! I was seeing regular delays of 10 to 15 seconds to
update the screen. We never figured out why I was seeing it and not
Stuart or what the cause was. Angular does shift most processing to the
client end ie the machine running the browser rather than what is more
traditional with stuff like php where most of the processing is done at
the server end. At the time I was using a 16 core Ryzen 7 so it can't be
lack of processing power on the client end. I was running the BE in a
ESXI virtual machine which may have contributed to the slow down since
the WebApp does sent many requests to the backends services API but I
never found any evidence to support that idea.


If Jan is seeing many of the screens showing very slow update times then
she may be seeing the same thing I was. If it's just one screen then
it's likely something else. One thing you need to know with Angular is
that it is basically one giant web page that has the entire WebApp on
it. Angular just hides and shows stuff based on what the user clicks on
once the page is loaded. It does mean that the initial loading is always
slow since the browser has to pull the several hundred megabyte? of data
for the entire WebApp before it shows the page you want. At least that
is the way I believe it works.



Paul H.

On Wed, Sep 6, 2023 at 9:06?PM Paul Harrison <mythtv@mythqml.net> wrote:

> Big disclosure I haven't been able to test the latest WebApp but when Stuart and I started the initial development the WebApp was very fast but as we added more stuff it started to slow down dramatically for me but not for Stuart! I was seeing regular delays of 10 to 15 seconds to update the screen. We never figured out why I was seeing it and not Stuart or what the cause was.

In a couple of popular browsers you can
open the devtools which can assist in
analyzing the performance of loading a
site. As with all such tools, sometimes
the information is helpful, and sometimes
not so much, but it is usually worth a try.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 06/09/2023 22:01, Paul Harrison wrote:
>
> Big disclosure I haven't been able to test the latest WebApp but when Stuart and I started the
> initial development the WebApp was very fast but as we added more stuff it started to slow down
> dramatically for me but not for Stuart! I was seeing regular delays of 10 to 15 seconds to update
> the screen. We never figured out why I was seeing it and not Stuart or what the cause was. Angular
> does shift most processing to the client end ie the machine running the browser rather than what is
> more traditional with stuff like php where most of the processing is done at the server end. At the
> time I was using a 16 core Ryzen 7 so it can't be lack of processing power on the client end. I was
> running the BE in a ESXI virtual machine which may have contributed to the slow down since the
> WebApp does sent many requests to the backends services API but I never found any evidence to
> support that idea.
>
>
> If Jan is seeing many of the screens showing very slow update times then she may be seeing the same
> thing I was. If it's just one screen then it's likely something else. One thing you need to know
> with Angular is that it is basically one giant web page that has the entire WebApp on it. Angular
> just hides and shows stuff based on what the user clicks on once the page is loaded. It does mean
> that the initial loading is always slow since the browser has to pull the several hundred megabyte?
> of data for the entire WebApp before it shows the page you want. At least that is the way I believe
> it works.
>
Anecdotal note: I have recently upgraded my main backend from Debian Buster(11) to Bookworm(12). To
my surprise I now find that loading both Firefox and Thunderbird take many seconds, in TB this can
be up to 20 or so before I see anything on screen.

In all cases my main server stays on 24/7, which means that once loaded the images should be in
cache and, previously, both loaded almost instantly. I have discovered changes to both FF and TB but
I suspect the main culprit is something in systemd, which I have neither the time nor patience to debug.

What you are seeing may be more of the same.

--

Mike Perkins


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 9/6/23 17:01, Paul Harrison wrote:
>
> You should also be able to use your browsers debug tools to see how
> long each network operation takes as well.
>
>
> Big disclosure I haven't been able to test the latest WebApp but when
> Stuart and I started the initial development the WebApp was very fast
> but as we added more stuff it started to slow down dramatically for me
> but not for Stuart! I was seeing regular delays of 10 to 15 seconds to
> update the screen. We never figured out why I was seeing it and not
> Stuart or what the cause was. Angular does shift most processing to
> the client end ie the machine running the browser rather than what is
> more traditional with stuff like php where most of the processing is
> done at the server end. At the time I was using a 16 core Ryzen 7 so
> it can't be lack of processing power on the client end. I was running
> the BE in a ESXI virtual machine which may have contributed to the
> slow down since the WebApp does sent many requests to the backends
> services API but I never found any evidence to support that idea.
>
>
> If Jan is seeing many of the screens showing very slow update times
> then she may be seeing the same thing I was. If it's just one screen
> then it's likely something else. One thing you need to know with
> Angular is that it is basically one giant web page that has the entire
> WebApp on it. Angular just hides and shows stuff based on what the
> user clicks on once the page is loaded. It does mean that the initial
> loading is always slow since the browser has to pull the several
> hundred megabyte? of data for the entire WebApp before it shows the
> page you want. At least that is the way I believe it works.
>
>
>
While developing the code I have found in some cases unacceptably long
responses, in particular with upcoming shows and recorded list. In both
cases I fixed it by switching to a virtual list with lazy load. That was
an immediate huge improvement. I have the code ready to switch videos to
a virtual list with lazy load. I want to make sure that the problem the
user is experiencing is on the angular side, not the server side. Lazy
load will not help if the server is taking a long time to produce the list.

Peter

On 06/09/2023 21:26, Peter Bennett wrote:
> Please try this, to see if the slowdown is on the backend:
>
> curl "http://localhost:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> On my laptop it reports time spent 0:00:01 with over 7000 videos in my
> database.
>
> If you are running your browser on a different computer from the
> backend, then try from that computer:
> curl "http://backend:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> Look at the Time Spent. This will tell if it is a slowdown in the network.
>
This takes 1 second on the backend and 2 seconds when run on the laptop
(the one in which the browser takes 50 seconds).

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 06/09/2023 21:26, Peter Bennett wrote:

> Please try this, to see if the slowdown is on the backend:
>
> curl "http://localhost:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> On my laptop it reports time spent 0:00:01 with over 7000 videos in my
> database.
>
> If you are running your browser on a different computer from the
> backend, then try from that computer:
> curl "http://backend:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>
> Look at the Time Spent. This will tell if it is a slowdown in the network.
>
This takes 1 second on the backend and 2 seconds when run on the laptop
(the one in which the browser takes 50 seconds).

On 9/7/23 09:20, Jan Ceuleers wrote:
> On 06/09/2023 21:26, Peter Bennett wrote:
>> Please try this, to see if the slowdown is on the backend:
>>
>> curl"http://localhost:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>>
>> On my laptop it reports time spent 0:00:01 with over 7000 videos in my
>> database.
>>
>> If you are running your browser on a different computer from the
>> backend, then try from that computer:
>> curl"http://backend:6544/Video/GetVideoList?Sort=FileName" >/dev/null
>>
>> Look at the Time Spent. This will tell if it is a slowdown in the network.
>>
> This takes 1 second on the backend and 2 seconds when run on the laptop
> (the one in which the browser takes 50 seconds).
>
>
OK Thanks, that is good news. The lazy load should fix it. I will commit
that code soon.

Peter

On 07/09/2023 16:08, Peter Bennett wrote:
>
>
> On 9/7/23 09:20, Jan Ceuleers wrote:
>>
>> This takes 1 second on the backend and 2 seconds when run on the laptop
>> (the one in which the browser takes 50 seconds).
>>
>>
> OK Thanks, that is good news. The lazy load should fix it. I will
> commit that code soon.
>
Many thanks indeed.

On 9/7/23 12:04, Jan Ceuleers wrote:
> On 07/09/2023 16:08, Peter Bennett wrote:
>>
>>
>> On 9/7/23 09:20, Jan Ceuleers wrote:
>>>
>>> This takes 1 second on the backend and 2 seconds when run on the laptop
>>> (the one in which the browser takes 50 seconds).
>>>
>>>
>> OK Thanks, that is good news. The lazy load should fix it. I will
>> commit that code soon.
>>
> Many thanks indeed.
>
I have committed the lazy load changes to master. Let me know whether it
solves the problem.

Peter

On 09/09/2023 15:36, Peter Bennett wrote:
> I have committed the lazy load changes to master. Let me know whether
> it solves the problem.

Thank you.

I'm on v33-fixes from the mythbuntu repo, so I'll wait for those changes
to trickle down.

 mythbackend --version
Please attach all output as a file in bug reports.
MythTV Version : v33.1+fixes.202308152218.21d2d9d951~ubuntu20.04.1
MythTV Branch : fixes/33
Network Protocol : 91
Library API : 33.20220913-1
QT Version : 5.12.8
Options compiled in:
 linux profile use_hidesyms using_alsa using_oss using_pulse
using_pulseoutput using_backend using_bindings_perl
using_bindings_python using_bindings_php using_dvb using_firewire
using_frontend using_hdhomerun using_satip using_vbox using_ceton
using_joystick_menu using_libcec using_libcrypto using_gnutls
using_libdns_sd using_libxml2 using_lirc using_mheg using_opengl
using_egl using_qtwebkit using_qtscript using_qtdbus using_taglib
using_v4l2 using_v4l2prime using_x11 using_system_libbluray
using_systemd_notify using_systemd_journal using_drm using_bindings_perl
using_bindings_python using_bindings_php using_freetype2
using_mythtranscode using_opengl using_egl using_drm using_vaapi
using_nvdec using_vdpau using_ffmpeg_threads using_mheg using_libass
using_libxml2 using_libmp3lame

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 9/9/23 10:04, Jan Ceuleers wrote:
> On 09/09/2023 15:36, Peter Bennett wrote:
>> I have committed the lazy load changes to master. Let me know whether
>> it solves the problem.
> Thank you.
>
> I'm on v33-fixes from the mythbuntu repo, so I'll wait for those changes
> to trickle down.
>
>  mythbackend --version
> Please attach all output as a file in bug reports.
> MythTV Version : v33.1+fixes.202308152218.21d2d9d951~ubuntu20.04.1
> MythTV Branch : fixes/33
> Network Protocol : 91
> Library API : 33.20220913-1
> QT Version : 5.12.8
> Options compiled in:
>  linux profile use_hidesyms using_alsa using_oss using_pulse
> using_pulseoutput using_backend using_bindings_perl
> using_bindings_python using_bindings_php using_dvb using_firewire
> using_frontend using_hdhomerun using_satip using_vbox using_ceton
> using_joystick_menu using_libcec using_libcrypto using_gnutls
> using_libdns_sd using_libxml2 using_lirc using_mheg using_opengl
> using_egl using_qtwebkit using_qtscript using_qtdbus using_taglib
> using_v4l2 using_v4l2prime using_x11 using_system_libbluray
> using_systemd_notify using_systemd_journal using_drm using_bindings_perl
> using_bindings_python using_bindings_php using_freetype2
> using_mythtranscode using_opengl using_egl using_drm using_vaapi
> using_nvdec using_vdpau using_ffmpeg_threads using_mheg using_libass
> using_libxml2 using_libmp3lame
>
If you are on V33 you are looking at the old code that is being
replaced. The web app is completely new and does not exist in V33.

Peter


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 07/09/2023 13:02, Peter Bennett wrote:

>
> On 9/6/23 17:01, Paul Harrison wrote:
>>
>> You should also be able to use your browsers debug tools to see how
>> long each network operation takes as well.
>>
>>
>> Big disclosure I haven't been able to test the latest WebApp but when
>> Stuart and I started the initial development the WebApp was very fast
>> but as we added more stuff it started to slow down dramatically for
>> me but not for Stuart! I was seeing regular delays of 10 to 15
>> seconds to update the screen. We never figured out why I was seeing
>> it and not Stuart or what the cause was. Angular does shift most
>> processing to the client end ie the machine running the browser
>> rather than what is more traditional with stuff like php where most
>> of the processing is done at the server end. At the time I was using
>> a 16 core Ryzen 7 so it can't be lack of processing power on the
>> client end. I was running the BE in a ESXI virtual machine which may
>> have contributed to the slow down since the WebApp does sent many
>> requests to the backends services API but I never found any evidence
>> to support that idea.
>>
>>
>> If Jan is seeing many of the screens showing very slow update times
>> then she may be seeing the same thing I was. If it's just one screen
>> then it's likely something else. One thing you need to know with
>> Angular is that it is basically one giant web page that has the
>> entire WebApp on it. Angular just hides and shows stuff based on what
>> the user clicks on once the page is loaded. It does mean that the
>> initial loading is always slow since the browser has to pull the
>> several hundred megabyte? of data for the entire WebApp before it
>> shows the page you want. At least that is the way I believe it works.
>>
>>
>>
> While developing the code I have found in some cases unacceptably long
> responses, in particular with upcoming shows and recorded list. In
> both cases I fixed it by switching to a virtual list with lazy load.
> That was an immediate huge improvement. I have the code ready to
> switch videos to a virtual list with lazy load. I want to make sure
> that the problem the user is experiencing is on the angular side, not
> the server side. Lazy load will not help if the server is taking a
> long time to produce the list.
>
> Peter
>

I made some time to at least get a working backend running again so I
could test the latest code. For me there are still some long delays like
when switching to some tabs or if a page refresh is done. I tried both
Firefox and Chromium  browsers both work the same.


Using the browser debug tools I found something interesting. More often
than not one of the requests sent to the backend will result in a 10
second wait before it receives any data. I'd be interested to know if
anyone else can see these 10 second delays.


If while viewing one of the WebApp pages you open the debug tools, 
right click and select 'Inspect' in the context menu or press
Ctrl+Shift+I, click the network tab in the debug window, then click
refresh or press F5 on the WebApp window. You should see the list of
requests update. Look for any with a tortoise symbol in the Filename
column, those are the requests that the browser thinks where slow. If
you click one of them you can see the timings. For me the slowest
requests always spend just over 10 seconds waiting. The file that is
slow isn't always the same one it seems to be random which one is slow,
although I think all so far have been in the root directory like main.js
in the linked screenshot.

https://mythqml.net/img/Screenshots/WebApp-timings.jpg


Paul H.

On 9/9/23 18:11, Paul Harrison wrote:
>
> I made some time to at least get a working backend running again so I
> could test the latest code. For me there are still some long delays
> like when switching to some tabs or if a page refresh is done. I tried
> both Firefox and Chromium  browsers both work the same.
>
>
> Using the browser debug tools I found something interesting. More
> often than not one of the requests sent to the backend will result in
> a 10 second wait before it receives any data. I'd be interested to
> know if anyone else can see these 10 second delays.
>
>
> If while viewing one of the WebApp pages you open the debug tools, 
> right click and select 'Inspect' in the context menu or press
> Ctrl+Shift+I, click the network tab in the debug window, then click
> refresh or press F5 on the WebApp window. You should see the list of
> requests update. Look for any with a tortoise symbol in the Filename
> column, those are the requests that the browser thinks where slow. If
> you click one of them you can see the timings. For me the slowest
> requests always spend just over 10 seconds waiting. The file that is
> slow isn't always the same one it seems to be random which one is
> slow, although I think all so far have been in the root directory like
> main.js in the linked screenshot.
>
> https://mythqml.net/img/Screenshots/WebApp-timings.jpg
>
>
> Paul H.
>
I have never noticed 10 second delays, and I don't think anybody else
has. Somebody would have let me know about the slowdown.

The longest timing in the debugger network tab is the GetBackendStatus
at 1.46 sec, followed by GetProgramGuide, at 743 ms. GetUpcomingList is
236 ms if you select "Show All Statuses".

I don't see a tortoise. Firefox also does not give me any slow responses.

Channel Editor takes about 5 seconds to load, but that is because it is
loading hundreds of icons, and getting a 404 error for most of them as
they are missing on my system.

I use Chrome for developing and testing, but I have tested on Firefox also.

As for Jan's report of slowness, it turned out he is running MythTV V33
so he is not using the web app, he is using the incomplete web frontend.

I suspect something else is going on in your system to cause the 10
second delays.

Are you running the browser on the same machine as the backend?

Do you get 10 second delays when running the API calls through curl or
on a browser?

Are you using SSL? I am not and probably 99.9% of users are not.

I have installed a recent master on my production system, and I use the
web app perfectly successfully on that, no delays.

Here are my timimgs for main.js: 10 ms for local machine and 51 ms for
remote over wifi. Nowhere near 10 seconds.

https://imgur.com/a/3I9iUFN

Peter

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 10/09/2023 01:48, Peter Bennett wrote:

> I have never noticed 10 second delays, and I don't think anybody else
> has. Somebody would have let me know about the slowdown.
>

I wonder how many users have actually used it though? The WebApp is
fairly slow to load so an extra 10 seconds delay here and there would
likely go unnoticed. The only reason it really stands out for me is
because I know how quick it was initially. I think it was only when we
added more stuff that causes more concurrent connections to the BE that
the problems started. It is just about usable if you are desperate but
not something I would want to use everyday which is OK with me since the
WebApp was always a means to an end, the important thing was to get all
the API in place then I can use it however I want :)


> The longest timing in the debugger network tab is the GetBackendStatus
> at 1.46 sec, followed by GetProgramGuide, at 743 ms. GetUpcomingList
> is 236 ms if you select "Show All Statuses".
>

Your system is way quicker than mine GetBackendStatus takes 3.15 sec,
GetProgramGuide 904ms. GetUpcomingList only takes 64ms but we no longer
use Myth to record stuff so the list is only 8 rows long \0/


> I don't see a tortoise. Firefox also does not give me any slow responses.
>
> Channel Editor takes about 5 seconds to load, but that is because it
> is loading hundreds of icons, and getting a 404 error for most of them
> as they are missing on my system.
>

The Channel Editor is fairly quick to load but I don't have many
channels. They nearly all have icons and I do see the occasional one
taking 10 seconds to load but that does not slow down the appearance of
the editor just the appearance of the delayed icons.


> I use Chrome for developing and testing, but I have tested on Firefox
> also.
>

Always use Firefox. Have also tested with Chromium but not Chrome, I
think they are closely related though.  I'm fairly confident it's not a
browser problem.


> As for Jan's report of slowness, it turned out he is running MythTV
> V33 so he is not using the web app, he is using the incomplete web
> frontend.
>

I did see that, I'm not laughing honestly! Dare I say it? I'm going to
say it.  We should try having regular 6 monthly releases to coincide
with Ubuntu releases. I know I'm crazy I really should not be let out in
public.


Another data point I did just try the old WebFrontend and that is quick
and responsive with no noticeable delays. Some of the API calls do take
a few seconds like the one for recordings but that's just how long those
calls take to complete. Certainly no sign of any requests waiting 10
seconds!


I did notice when trying the  Video Gallery that Jan was having problems
with it does spam the BE with lots of request to get the cover art, most
of which get blocked because the browser only allows 6 simultaneous
connections to a single server but that is different than the delays I
was seeing in the WebApp where the delay is in how long the server takes
to respond to the request not how long the request was queued up for. I
would say the UX is way better in the WebFrontend that the WebApp simply
because it's way more responsive and let be honest it just looks better.


> I suspect something else is going on in your system to cause the 10
> second delays.
>
> Are you running the browser on the same machine as the backend?


No the BE is running on my ESXi server in a VM. I don't see any problems
with my other VM's that host websites/apis like my ZoneMinder server.


>
> Do you get 10 second delays when running the API calls through curl or
> on a browser?
>

No I've never seen any delay using the API directly but I typically only
send one request at a time not the multiple concurrent requests that
browser send which I think is the problem.


> Are you using SSL? I am not and probably 99.9% of users are not.
>

No not using SSL.


Don't get my started on why we shouldn't make the same mistakes as
MythWeb which by default is completely open with no security at all.
It's slightly worse with the WebApp because of all the access to the
setup stuff. Anyone with access to the WebApp can completely trash your
system if they are so inclined. I think at the very least there should
be a way to only allow certain users to access the setup stuff to
prevent unintended meddling. I did put a prominent login button in the
header to remind everyone what the intention was but I see you removed
it :( Correct me if something has changed but I believe there is also
currently no way to configure slave backends something that needs be
addressed before completely removing mythtv-setup?


> I have installed a recent master on my production system, and I use
> the web app perfectly successfully on that, no delays.
>
> Here are my timimgs for main.js: 10 ms for local machine and 51 ms for
> remote over wifi. Nowhere near 10 seconds.
>
> https://imgur.com/a/3I9iUFN
>

Aren't you the lucky one :)


TBH I'm not too concerned about it so long as the API is still usable
I'm OK with that it was more a FYI.


As you know we no longer use MythTV to record stuff since we got TiVos
we pretty much only use those or the smart TV Apps for on demand stuff.
That will undoubtedly change again when our current contract runs out
and the heavily discounted deal reverts to normal prices but for now we
really don't need to use MythTV at all for anything so I don't bother
wasting cpu cycles on it. Plus I needed the SSD space, memory  and cores
to play with Home Assistant :) I think we could learn a lot from the way
HA handles a lot of stuff. They do have the advantage of a much larger
user base though.


> Peter
>
>

Paul H.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 9/10/23 09:59, Paul Harrison wrote:
> On 10/09/2023 01:48, Peter Bennett wrote:
>
>> I have never noticed 10 second delays, and I don't think anybody else
>> has. Somebody would have let me know about the slowdown.
>>
>
> I wonder how many users have actually used it though?

James Abernathy tested everything, including doing multiple setups from
scratch using the web app setup.

Klaas DeWaal, Bill Meek, Ken Mandelbaum, David Engel, gave feedback
after testing it. Some feedback was that it does not look pretty, but no
slowness complaints that have not been fixed.


> The WebApp is fairly slow to load so an extra 10 seconds delay here
> and there would likely go unnoticed. The only reason it really stands
> out for me is because I know how quick it was initially. I think it
> was only when we added more stuff that causes more concurrent
> connections to the BE that the problems started. It is just about
> usable if you are desperate but not something I would want to use
> everyday which is OK with me since the WebApp was always a means to an
> end, the important thing was to get all the API in place then I can
> use it however I want :)
I notice no slowness at all. I use it regularly to look at my backend
upcoming recordings, videos, etc.
>
>
>> The longest timing in the debugger network tab is the
>> GetBackendStatus at 1.46 sec, followed by GetProgramGuide, at 743 ms.
>> GetUpcomingList is 236 ms if you select "Show All Statuses".
>>
>
> Your system is way quicker than mine GetBackendStatus takes 3.15 sec,
> GetProgramGuide 904ms. GetUpcomingList only takes 64ms but we no
> longer use Myth to record stuff so the list is only 8 rows long \0/
>
>
>> I don't see a tortoise. Firefox also does not give me any slow
>> responses.
>>
>> Channel Editor takes about 5 seconds to load, but that is because it
>> is loading hundreds of icons, and getting a 404 error for most of
>> them as they are missing on my system.
>>
>
> The Channel Editor is fairly quick to load but I don't have many
> channels. They nearly all have icons and I do see the occasional one
> taking 10 seconds to load but that does not slow down the appearance
> of the editor just the appearance of the delayed icons.
>
I have loaded every possible channel including ones I do not receive,
now there are 843 channels in my laptop test system.
>
>> I use Chrome for developing and testing, but I have tested on Firefox
>> also.
>>
>
> Always use Firefox. Have also tested with Chromium but not Chrome, I
> think they are closely related though.  I'm fairly confident it's not
> a browser problem.
>
>
>> As for Jan's report of slowness, it turned out he is running MythTV
>> V33 so he is not using the web app, he is using the incomplete web
>> frontend.
>>
>
> I did see that, I'm not laughing honestly! Dare I say it? I'm going to
> say it.  We should try having regular 6 monthly releases to coincide
> with Ubuntu releases. I know I'm crazy I really should not be let out
> in public.
>
>
> Another data point I did just try the old WebFrontend and that is
> quick and responsive with no noticeable delays. Some of the API calls
> do take a few seconds like the one for recordings but that's just how
> long those calls take to complete. Certainly no sign of any requests
> waiting 10 seconds!
>
>
> I did notice when trying the  Video Gallery that Jan was having
> problems with it does spam the BE with lots of request to get the
> cover art, most of which get blocked because the browser only allows 6
> simultaneous connections to a single server but that is different than
> the delays I was seeing in the WebApp where the delay is in how long
> the server takes to respond to the request not how long the request
> was queued up for. I would say the UX is way better in the WebFrontend
> that the WebApp simply because it's way more responsive and let be
> honest it just looks better.
>
>
>> I suspect something else is going on in your system to cause the 10
>> second delays.
>>
>> Are you running the browser on the same machine as the backend?
>
>
> No the BE is running on my ESXi server in a VM. I don't see any
> problems with my other VM's that host websites/apis like my ZoneMinder
> server.
>
>
>>
>> Do you get 10 second delays when running the API calls through curl
>> or on a browser?
>>
>
> No I've never seen any delay using the API directly but I typically
> only send one request at a time not the multiple concurrent requests
> that browser send which I think is the problem.
>
>
>> Are you using SSL? I am not and probably 99.9% of users are not.
>>
>
> No not using SSL.
>
>
> Don't get my started on why we shouldn't make the same mistakes as
> MythWeb which by default is completely open with no security at all.
> It's slightly worse with the WebApp because of all the access to the
> setup stuff. Anyone with access to the WebApp can completely trash
> your system if they are so inclined. I think at the very least there
> should be a way to only allow certain users to access the setup stuff
> to prevent unintended meddling. I did put a prominent login button in
> the header to remind everyone what the intention was but I see you
> removed it :( Correct me if something has changed but I believe there
> is also currently no way to configure slave backends something that
> needs be addressed before completely removing mythtv-setup?
>
I have done nothing for security in the web app. I rely on the ip
address checks that I put in a couple of years ago. Unless the user
specifically allows connection form all ip addresses, only connections
for the local subnet are permitted.

Slave backend setup is supported. If you start the web app on a slave
backend, or a system that looks like it should be a slave, it displays
at the top of the page:

This appears to be a slave backend. If it is intended as a slave
backend, please disable scheduling on the master backend while running
slave backend setup.
If this is not intended as a slave backend, please go to Setup, General,
Host Address Backend Setup, and select "This server is the Master
Backend" or else set the correct custom identifier on the Database Setup
page. Save and Restart the backend,

In the case of master backends, the code will not save any setup pages
until you disable scheduling. In the case of slave backends, the user
must manually make sure that scheduling is disabled on the master.

The dashboard part of the web app is rather limited when running against
a slave.

>
>> I have installed a recent master on my production system, and I use
>> the web app perfectly successfully on that, no delays.
>>
>> Here are my timimgs for main.js: 10 ms for local machine and 51 ms
>> for remote over wifi. Nowhere near 10 seconds.
>>
>> https://imgur.com/a/3I9iUFN
>>
>
> Aren't you the lucky one :)
>
It is hard work, not luck. I have spent over a year working on this and
some of the time was spent on improving performance whe I found
something was slow.
>
> TBH I'm not too concerned about it so long as the API is still usable
> I'm OK with that it was more a FYI.
>
>
> As you know we no longer use MythTV to record stuff since we got TiVos
> we pretty much only use those or the smart TV Apps for on demand stuff.
I did not know that.
> That will undoubtedly change again when our current contract runs out
> and the heavily discounted deal reverts to normal prices but for now
> we really don't need to use MythTV at all for anything so I don't
> bother wasting cpu cycles on it. Plus I needed the SSD space, memory 
> and cores to play with Home Assistant :) I think we could learn a lot
> from the way HA handles a lot of stuff. They do have the advantage of
> a much larger user base though.
>
>
>> Peter
>>
>>
>
> Paul H.
>
>
I just did some testing on the slowest machine I have, a pentium laptop
from 2017. For example it takes 30 seconds on this laptop for chrome to
start after clicking it on the start menu. I am accessing a backend that
is on wifi and the slow machine with the browser is also on wifi. The
slow laptop has only 2.4 GHz wifi, the backend on a fast laptop has 5
GHz wifi, so the signal is getting translated in the router.

Channel editor (843 items) : 7 seconds to open
Program Guide (843 channels): 6 seconds to open
Videos (7531 videos) : 3 seconds to open
Upcoming (701 shows): 1 second

Is your database remote from your backend? Tests of running queries
against my slave backend are rather slower (because of the remote
database), with API response of 6.2 seconds for the call to get the
video list, for example

If my database is on a system connected to the backend via wifi, then I
get some horrible response times. Wifi is asymmetric. They advertise
speeds of a Gigabit per second, but that is download speed. Upload speed
is a fraction of that, and having the database on a wifi connected
laptop kills the speed.

Peter




_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

I haven't looked at the code and I haven't tried the setup, but one
thing that jumped out at me in following this discussion is that 10.01
second timeout in the web timing chart, plus statements like

> For me the slowest requests always spend just over 10 seconds
> waiting. The file that is slow isn't always the same one it seems
> to be random which one is slow

I'm wondering if this isn't CPU load but some sort of 10 second timeout
somewhere, or maybe two five-second timeouts in opposite directions,
etc. But I don't know where those might be coming from. Local DNS/DHCP
resolution? Obviously this is a total guess, but I'm suspicious.

Is there any possibility you could run wireshark? You'd have to have
some way of capturing the packet stream, but if you could do it on the
affected host(s), or perhaps if you have a smart switch where you could
clone traffic on one of the ports (or a really dumb hub :), a packet
trace might shed some light.
_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

Just for the record the Web App in version 34 is perfectly responsive on
an old Intel® Pentium(R) CPU G3220 @ 3.00GHz × 2. Remote to the backend.

It works acceptably on a under powered Fire HD tablet, taking a couple
of seconds to switch between the Program Guide and the list of
Recordings. Both of these lists are relatively short - about 50 channels
and 200 recordings as I don't keep stuff long term.

Tested in Amazon Silk Browser, Firefox and Chrome.

Chrome is the fastest with a one second response time.

mythtv-light_34~Pre-475-g6b2c86e26b-0_amd64_jammy

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 11/09/2023 04:07, f-myth-users@media.mit.edu wrote:

> I haven't looked at the code and I haven't tried the setup, but one
> thing that jumped out at me in following this discussion is that 10.01
> second timeout in the web timing chart, plus statements like
>
>> For me the slowest requests always spend just over 10 seconds
>> waiting. The file that is slow isn't always the same one it seems
>> to be random which one is slow
> I'm wondering if this isn't CPU load but some sort of 10 second timeout
> somewhere, or maybe two five-second timeouts in opposite directions,
> etc. But I don't know where those might be coming from. Local DNS/DHCP
> resolution? Obviously this is a total guess, but I'm suspicious.


I think you are correct the fact the slow requests all take just over 10
seconds can't be a coincidence.


It's strange the only thing that appears to have the problem is the
WebApp which makes me think the problem is in the BE server but why
aren't others seeing it. It could be like de-interlace artifacts or mpeg
blocking on dark backgrounds most uses can blissfully go through life
and never see them but once someone points them out to you that is all
you can see. Maybe I'm just seeing the delays  because I know they are
there.


Another thing I just thought of is it depends on what request is being
delayed if it is one of the js files or css files etc then that is bad
because the page has to wait for the data to arrive before it can start
to render. If it's an image or API call then the page can usually start
to render more quickly you just see a space where the image should be or
maybe the loading spinner continues for a little longer than normal.


> Is there any possibility you could run wireshark? You'd have to have
> some way of capturing the packet stream, but if you could do it on the
> affected host(s), or perhaps if you have a smart switch where you could
> clone traffic on one of the ports (or a really dumb hub :), a packet
> trace might shed some light.


Stuart A. and I did start to debug this back 18 months? or so ago when I
first noticed it but circumstances changed and we never got very far. It
was Jen's message that reminded me about the problem and I was just
wondering if it was the same problem but it is clearly not. One rainy
day when I'm bored I may try to debug this further.


Paul H.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 11/09/2023 10:17, John wrote:

> Just for the record the Web App in version 34 is perfectly responsive
> on an old Intel® Pentium(R) CPU G3220 @ 3.00GHz × 2. Remote to the
> backend.
>
> It works acceptably on a under powered Fire HD tablet, taking a couple
> of seconds to switch between the Program Guide and the list of
> Recordings. Both of these lists are relatively short - about 50
> channels and 200 recordings as I don't keep stuff long term.
>
> Tested in Amazon Silk Browser, Firefox and Chrome.
>
> Chrome is the fastest with a one second response time.
>
> mythtv-light_34~Pre-475-g6b2c86e26b-0_amd64_jammy
>
>

OK thanks for testing.


Did you look at the Firefox or Chrome debug timings just to confirm
there was no slow requests? For me that would be the acid test since
these things can be a little subjective so using the available data just
to make sure would be helpful.


I'm beginning to suspect something weird with my setup is causing this
but why it only affects the WebApp is a mystery.


Paul H.

_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org

On 11/09/2023 01:48, Peter Bennett wrote:

>
> On 9/10/23 09:59, Paul Harrison wrote:
>> I wonder how many users have actually used it though?
>
> James Abernathy tested everything, including doing multiple setups
> from scratch using the web app setup.
>
> Klaas DeWaal, Bill Meek, Ken Mandelbaum, David Engel, gave feedback
> after testing it. Some feedback was that it does not look pretty, but
> no slowness complaints that have not been fixed.
>
>

OK so just me then :(


> I have done nothing for security in the web app. I rely on the ip
> address checks that I put in a couple of years ago. Unless the user
> specifically allows connection form all ip addresses, only connections
> for the local subnet are permitted.
>

Yeah you definitely don't want to expose the API or WebApp to the
internet in it's current state. The initial aim was the WebApp would be
secure out of the box although admittedly we never came up with a plan
to do it. You can password protect some or all of the API but unless you
added it there is no support in the WebApp to use it that way.


The other thing we would have liked to add was to follow up on Stuart
M's user support and to be able to restrict parts of the webApp to
certain users. For example admin users could use the setup stuff but
other users can't. Or the kids can only see video's or recordings
appropriate for their age that sort of thing.


> Slave backend setup is supported. If you start the web app on a slave
> backend, or a system that looks like it should be a slave, it displays
> at the top of the page:
>
> This appears to be a slave backend. If it is intended as a slave
> backend, please disable scheduling on the master backend while running
> slave backend setup.
> If this is not intended as a slave backend, please go to Setup,
> General, Host Address Backend Setup, and select "This server is the
> Master Backend" or else set the correct custom identifier on the
> Database Setup page. Save and Restart the backend,
>
> In the case of master backends, the code will not save any setup pages
> until you disable scheduling. In the case of slave backends, the user
> must manually make sure that scheduling is disabled on the master.
>
> The dashboard part of the web app is rather limited when running
> against a slave.
>

Cool. I missed you added support for slave BE's. Is that support built
in to the API or just the WebApp? I kind of envisioned a drop down in
the header where you could select the BE you wanted to configure.


>>
>>> I have installed a recent master on my production system, and I use
>>> the web app perfectly successfully on that, no delays.
>>>
>>> Here are my timimgs for main.js: 10 ms for local machine and 51 ms
>>> for remote over wifi. Nowhere near 10 seconds.
>>>
>>> https://imgur.com/a/3I9iUFN
>>>
>>
>> Aren't you the lucky one :)
>>
> It is hard work, not luck. I have spent over a year working on this
> and some of the time was spent on improving performance whe I found
> something was slow.
>>

Oh I'm not ungrateful you took on the task. I appreciate the time and
effort you have put in to this because I know how much work is involved.


>>
>>> Peter
>>>
>>>

>> I just did some testing on the slowest machine I have, a pentium
>> laptop from 2017. For example it takes 30 seconds on this laptop for
>> chrome to start after clicking it on the start menu. I am accessing a
>> backend that is on wifi and the slow machine with the browser is also
>> on wifi. The slow laptop has only 2.4 GHz wifi, the backend on a fast
>> laptop has 5 GHz wifi, so the signal is getting translated in the
>> router.
>
> Channel editor (843 items) : 7 seconds to open
> Program Guide (843 channels): 6 seconds to open
> Videos (7531 videos) : 3 seconds to open
> Upcoming (701 shows): 1 second
>
> Is your database remote from your backend? Tests of running queries
> against my slave backend are rather slower (because of the remote
> database), with API response of 6.2 seconds for the call to get the
> video list, for example
>

My MySql server runs in it's own VM because it is shared by several VM's
in my ESXI server, it just makes maintenance easier if there is just one
MySql server running that is shared.  I haven't observed any problems
with the other users of the database server. Most of the queries are
from VM to VM so never leave the virtual environment so everything is
really quick from what I've seen.


> If my database is on a system connected to the backend via wifi, then
> I get some horrible response times. Wifi is asymmetric. They advertise
> speeds of a Gigabit per second, but that is download speed. Upload
> speed is a fraction of that, and having the database on a wifi
> connected laptop kills the speed.
>

I have a really good UniFi smart switch and everything is hard wired
using Gigabit connections where appropriate.


> Peter
>
>

Paul H.


_______________________________________________
mythtv-users mailing list
mythtv-users@mythtv.org
http://lists.mythtv.org/mailman/listinfo/mythtv-users
http://wiki.mythtv.org/Mailing_List_etiquette
MythTV Forums: https://forum.mythtv.org