Mailing List Archive

Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend
#13479: mythbackend deadlock when editing schedule metadata on frontend
--------------------------------------+--------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: Unspecified
Severity: medium | Keywords:
Ticket locked: 0 |
--------------------------------------+--------------------------
This has happened a few times, but only since V30.

After performing a search for artwork in the schedule editor, the backend
sometimes gets into a state where the frontends lose the connection and no
frontend can get connected again.

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+------------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: Unspecified
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+------------------------------
Changes (by Peter Bennett):

* Attachment
"gdb.core.1567104223.200.1464.mythbackend_v30.0-66-g83e27017203" added.

Full backtrace

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+------------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: Unspecified
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+------------------------------
Changes (by Peter Bennett):

* Attachment
"ver.gdb.core.1567104223.200.1464.mythbackend_v30.0-66-g83e27017203"
added.

Version

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+------------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+------------------------------
Changes (by Peter Bennett):

* version: Unspecified => v30-fixes


--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:1>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+------------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+------------------------------

Comment (by Peter Bennett):

This happened again, this time with no schedule or unusual actions on the
frontend, just connecting the frontend and navigating to the recordings
list.

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:2>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+------------------------------
Reporter: Peter Bennett | Owner: (none)
Type: Bug Report - General | Status: new
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+------------------------------

Comment (by Peter Bennett):

This may be related to the reposrt in the forum at
https://forum.mythtv.org/viewtopic.php?f=36&t=3545

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:3>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* owner: (none) => Peter Bennett
* status: new => assigned


Comment:

See the attached file thread_analysis.txt

Thread 34 and Thread 6 are both doing this
{{{
MainServer::connectionClosed calls sockListLock.lockForWrite
MainServer::connectionClosed calls MainServer::UpdateSystemdStatus
MainServer::UpdateSystemdStatus calls m_sched->GetAllPending(recordings)
Scheduler::GetAllPending calls QMutexLocker lockit(&schedLock)
}}}
This causes
1. lock on sockListLock
2. lock on schedLock

Thread 22 is doing this
{{{
Scheduler::run calls QMutexLocker lockit(&schedLock); (l2070)
Scheduler::run calls HandleIdleShutdown; (l2234)
Scheduler::HandleIdleShutdown calls MainServer::isClientConnected :8358
MainServer::isClientConnected calls sockListLock.lockForRead
}}}
This causes
1. lock on schedLock
2. lock on sockListLock

*** DEADLOCK ***

If you are setting two or more locks, you need to always set them in the
same order in every thread, otherwise deadlocks can happen.

There are many places where mainserver calls scheduler and fewer than 10
places where scheduler calls mainserver. All are potential deadlock
scenarios. Perhaps whenever scheduler calls mainserver it should first
unlock its mutex and lock it again upon return.

ANY COMMENTS FROM DEVELOPERS?

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:4>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "thread_analysis.txt" added.

Thread Analysis

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------

Comment (by Peter Bennett):

scheduler.cpp already has 9 places where calls to other routines are
surrounded by schedLock.unlock() and schedLock.lock(). It looks like I
only need to add those to the two places in scheduler.cpp where
m_mainServer->isClientConnected() is called.

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:5>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------

Comment (by Peter Bennett):

Regarding m_recListChanged - this can be used to check if the list
changed. If the list changes during the call to
m_mainServer->isClientConnected, that will not be a problem. In
!HandleIdleShutdown it does look through the list, and the list cannot
change while it is looking through the list since it is locked at that
time. I prefer to make as few changes as possible to reduce risk, so I
plan to only unlock and relock at the two places
m_mainServer->isClientConnected is called inside !HandleIdleShutdown.

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:6>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200203_1729_deadlock_v30.patch" added.

Deadlock patch for V30

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1137_deadlock_v31.patch" added.

Deadlock patch for V31 (master)

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------

Comment (by gigem):

Replying to [comment:6 Peter Bennett]:
> Regarding m_recListChanged - this can be used to check if the list
changed. If the list changes during the call to
m_mainServer->isClientConnected, that will not be a problem. In
!HandleIdleShutdown it does look through the list, and the list cannot
change while it is looking through the list since it is locked at that
time. I prefer to make as few changes as possible to reduce risk, so I
plan to only unlock and relock at the two places
m_mainServer->isClientConnected is called inside !HandleIdleShutdown.

Please do not do this. When m_recListChanged is true, stop and return.
Always. It's not just that iterators might have been invalidated, but
other assumptions like list ordering might have been invalidated too. You
might be able to convince yourself that in this case at this time, it
won't hurt anything. That might not always be the case, though, as an
unrelated, future change might invalidate it. To avoid that and keep
things simple, just stop and return.

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:7>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1248_deadlock_v30.patch" added.

Updated patch for v30

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1207_deadlock_v31.patch" added.

Updated patch for v31 (master)

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1528_deadlock_v30.patch" added.

Updated V30 patch

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1521_deadlock_v31.patch" added.

Updated V31 (master) patch

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200203_1729_deadlock_v30.patch" removed.

Deadlock patch for V30

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1137_deadlock_v31.patch" removed.

Deadlock patch for V31 (master)

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1248_deadlock_v30.patch" removed.

Updated patch for v30

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: assigned
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution:
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett):

* Attachment "20200204_1207_deadlock_v31.patch" removed.

Updated patch for v31 (master)

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: closed
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution: fixed
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Peter Bennett <pbennett@…>):

* status: assigned => closed
* resolution: => fixed


Comment:

In [changeset:"3e9e8353aa8b5626c0b2e2727e5ed39097bc8c98/mythtv"
3e9e8353a/mythtv]:
{{{
#!CommitTicketReference repository="mythtv"
revision="3e9e8353aa8b5626c0b2e2727e5ed39097bc8c98"
Scheduler: Fix deadlock in mythbackend

Fixes deadlock caused by conflicting Mutex locks in HandleIdleShutdown

Fixes #13479
}}}

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:8>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: closed
Priority: minor | Milestone: needs_triage
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution: fixed
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------

Comment (by Peter Bennett <pbennett@…>):

In [changeset:"47814b6d618f5d395073a5a570cae82cca73812d/mythtv"
47814b6d6/mythtv]:
{{{
#!CommitTicketReference repository="mythtv"
revision="47814b6d618f5d395073a5a570cae82cca73812d"
Scheduler: Fix deadlock in mythbackend

Fixes deadlock caused by conflicting Mutex locks in HandleIdleShutdown

Fixes #13479
}}}

--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:9>
MythTV <http://www.mythtv.org>
MythTV Media Center
Re: Ticket #13479: mythbackend deadlock when editing schedule metadata on frontend [ In reply to ]
#13479: mythbackend deadlock when editing schedule metadata on frontend
----------------------------------+-------------------------------
Reporter: Peter Bennett | Owner: Peter Bennett
Type: Bug Report - General | Status: closed
Priority: minor | Milestone: 31.0
Component: MythTV - General | Version: v30-fixes
Severity: medium | Resolution: fixed
Keywords: | Ticket locked: 0
----------------------------------+-------------------------------
Changes (by Stuart Auchterlonie):

* milestone: needs_triage => 31.0


--
Ticket URL: <https://code.mythtv.org/trac/ticket/13479#comment:10>
MythTV <http://www.mythtv.org>
MythTV Media Center