Mailing List Archive

What's the rule for issuing line tracing numbers when using a tracing function?
Hi all,

I have a weird case where I thought line events would be issued and yet
they aren't even though they're in the instructions in the bytecode (both
in 3.9 and 3.10).

i.e.:

Given the code:

def check_backtrack(x): # line 1
if not (x == 'a' # line 2
or x == 'c'): # line 3
pass # line 4

it has dis.dis such as:

2 0 LOAD_FAST 0 (x)
2 LOAD_CONST 1 ('a')
4 COMPARE_OP 2 (==)
6 POP_JUMP_IF_TRUE 12 (to 24)

3 8 LOAD_FAST 0 (x)
10 LOAD_CONST 2 ('c')
12 COMPARE_OP 2 (==)

2 14 POP_JUMP_IF_TRUE 10 (to 20)

4 16 LOAD_CONST 0 (None)
18 RETURN_VALUE

2 >> 20 LOAD_CONST 0 (None)
22 RETURN_VALUE
>> 24 LOAD_CONST 0 (None)
26 RETURN_VALUE

So, by just following the instructions/line numbers, I'd say that when the
instruction:

2 14 POP_JUMP_IF_TRUE 10 (to 20)

is executed, a line event would take place, yet, this isn't true, but if
that offset is changed to include more instructions then such a line event
is issued.

i.e.: something as:

def tracer(frame, event, arg):
print(frame, event)
return tracer

import sys
sys.settrace(tracer)
check_backtrack('f')

prints:

1 call
2 line
3 line
4 line
4 return

when I expected it to print:

1 call
2 line
3 line
2 line |<-- this is not being issued
4 line
4 return

So, I have some questions related to this:

Does anyone know why this happens?
What's the rule to identify this?
Why is that line number assigned to that instruction (i.e.: it seems a bit
odd that this is set up like that in the first place)?

Thanks,

Fabio

p.s.: I'm asking because in a debugger which changes bytecode I want to
keep the same semantics and it appears that if I add more bytecode at that
instruction offset, those semantics aren't kept (but I don't really know
what are the semantics to keep here since it seems like that instruction
should issue a line event even though it doesn't).
Re: What's the rule for issuing line tracing numbers when using a tracing function? [ In reply to ]
Em qui., 16 de set. de 2021 às 11:49, Fabio Zadrozny <fabiofz@gmail.com>
escreveu:

> Hi all,
>
> I have a weird case where I thought line events would be issued and yet
> they aren't even though they're in the instructions in the bytecode (both
> in 3.9 and 3.10).
>
> i.e.:
>
> Given the code:
>
> def check_backtrack(x): # line 1
> if not (x == 'a' # line 2
> or x == 'c'): # line 3
> pass # line 4
>
> it has dis.dis such as:
>
> 2 0 LOAD_FAST 0 (x)
> 2 LOAD_CONST 1 ('a')
> 4 COMPARE_OP 2 (==)
> 6 POP_JUMP_IF_TRUE 12 (to 24)
>
> 3 8 LOAD_FAST 0 (x)
> 10 LOAD_CONST 2 ('c')
> 12 COMPARE_OP 2 (==)
>
> 2 14 POP_JUMP_IF_TRUE 10 (to 20)
>
> 4 16 LOAD_CONST 0 (None)
> 18 RETURN_VALUE
>
> 2 >> 20 LOAD_CONST 0 (None)
> 22 RETURN_VALUE
> >> 24 LOAD_CONST 0 (None)
> 26 RETURN_VALUE
>
> So, by just following the instructions/line numbers, I'd say that when the
> instruction:
>
> 2 14 POP_JUMP_IF_TRUE 10 (to 20)
>
> is executed, a line event would take place, yet, this isn't true, but if
> that offset is changed to include more instructions then such a line event
> is issued.
>
> i.e.: something as:
>
> def tracer(frame, event, arg):
> print(frame, event)
> return tracer
>
> import sys
> sys.settrace(tracer)
> check_backtrack('f')
>
> prints:
>
> 1 call
> 2 line
> 3 line
> 4 line
> 4 return
>
> when I expected it to print:
>
> 1 call
> 2 line
> 3 line
> 2 line |<-- this is not being issued
> 4 line
> 4 return
>
> So, I have some questions related to this:
>
> Does anyone know why this happens?
> What's the rule to identify this?
> Why is that line number assigned to that instruction (i.e.: it seems a bit
> odd that this is set up like that in the first place)?
>
> Thanks,
>
> Fabio
>
> p.s.: I'm asking because in a debugger which changes bytecode I want to
> keep the same semantics and it appears that if I add more bytecode at that
> instruction offset, those semantics aren't kept (but I don't really know
> what are the semantics to keep here since it seems like that instruction
> should issue a line event even though it doesn't).
>

Answering my own question after investigating:

It boils down to the way that ceval.c does prediction of bytecodes which
makes it miss the line.

i.e.: the compare is something as:

TARGET(COMPARE_OP): {
assert(oparg <= Py_GE);
PyObject *right = POP();
PyObject *left = TOP();
PyObject *res = PyObject_RichCompare(left, right, oparg);
SET_TOP(res);
Py_DECREF(left);
Py_DECREF(right);
if (res == NULL)
goto error;
PREDICT(POP_JUMP_IF_FALSE);
PREDICT(POP_JUMP_IF_TRUE);
DISPATCH();
}

Given that, PREDICT makes the "POP_JUMP_IF_FALSE" / "POP_JUMP_IF_TRUE"
line be ignored and its line becomes merged with the "COMPARE_OP" in the
tracing and when the bytecode is manipulated this is no longer true, so
those spurious line events start to be generated in the tracing.

So, in the end it boils down the the eval loop not respecting what's
written in the bytecode under some conditions -- in this particular case
it's probably good as the jump line seems to be off and I'd say it's a bug
in the bytecode generation which is then fixed by a bug in the PREDICT
ignoring that line that's off, so, the bugs nullify each other, but then,
my rewriting of bytecode makes the PREDICT fail and then the issue of
having the line off in the bytecode becomes apparent while tracing.

As a note, this is the 2nd time I've been bitten by PREDICT (
https://mail.python.org/archives/list/python-dev@python.org/message/HYTVMG4YTU3QLRGEE4LMLGFANQPO4PD3/)...


Does someone know if with the introduction of the new
optimizations/quickening the PREDICT will still be used in 3.11? If there
are 2 interpretation modes now, the PREDICT probably makes it hard to have
both do the same things since the eval loop doesn't really match what the
bytecode says due to the PREDICT.

Thanks,

Fabio
Re: What's the rule for issuing line tracing numbers when using a tracing function? [ In reply to ]
> Does someone know if with the introduction of the new
optimizations/quickening the PREDICT will still be used in 3.11? If there
are 2 interpretation modes now, the PREDICT probably makes it hard to have
both do the same things since the eval loop doesn't really match what the
bytecode says due to the PREDICT.

For your use case (tracing), the behavior should be the same as 3.10. PEP 659 and its current implementation doesn't run specialized bytecode when tracing and falls back to normal bytecode. The behavior of normal bytecode hasn't changed either (other than what's currently listed in the changelog).

-KJ
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/QHNPOVCQNCXN33QVCKSQUTPQJZAJCCR7/
Code of Conduct: http://python.org/psf/codeofconduct/