Mailing List Archive: Changing unittest verbose output.

Changing unittest verbose output.

Mar 26, 2022, 5:48 PM

Post #1 of 10 (805 views)

When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word.
For example:

./python.exe -m unittest -v Lib.test_verbose_error.randomTest
test_one (Lib.test_verbose_error.randomTest)
this is a doc string with helpful info ... ERROR

======================================================================
ERROR: test_one (Lib.test_verbose_error.randomTest)
this is a doc string with helpful info
----------------------------------------------------------------------
Traceback (most recent call last):
File "/workspaces/cpython/Lib/test_verbose_error.py", line 11, in test_one
assert l[1]
~^^^
IndexError: list index out of range

----------------------------------------------------------------------
Ran 1 test in 0.004s

This cause for some confusion and we would like to fix it, the possible options are:

1. print the name and docstring, and ERROR, all on one line.
2. provide a command-line switch to select that behavior.

The first option has a minor backwards compatibility breakage, since the default output would change and those lines would get longer.

Any thoughts on the mentioned solutions?

https://bugs.python.org/issue47133
_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/KVHWPAVKIMYME6OBMK5CZKBA2C2HZD6F/
Code of Conduct: http://python.org/psf/codeofconduct/

Re: Changing unittest verbose output. [ In reply to ]

jelle.zijlstra at gmail

Mar 27, 2022, 7:51 AM

Post #2 of 10 (803 views)

Permalink

El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya (<itay.yeshaya@gmail.com>)
escribió:

>
> Any thoughts on the mentioned solutions?
>

This is a bit more out there, but can anyone explain why we show the
docstring at all? As far as I can tell it causes a bunch of confusion and
helps little. If there's a failing test, I want the test name so I can
quickly find it in the code; I don't want the docstring which may not even
be unique.

>
> https://bugs.python.org/issue47133
>

Re: Changing unittest verbose output. [ In reply to ]

eric at trueblade

Mar 27, 2022, 8:20 AM

Post #3 of 10 (803 views)

Permalink

On 3/27/2022 10:51 AM, Jelle Zijlstra wrote:
>
>
> El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya
> (<itay.yeshaya@gmail.com>) escribió:
>
>
> Any thoughts on the mentioned solutions?
>
>
> This is a bit more out there, but can anyone explain why we show the
> docstring at all? As far as I can tell it causes a bunch of confusion
> and helps little. If there's a failing test, I want the test name so I
> can quickly find it in the code; I don't want the docstring which may
> not even be unique.

I think this is a good idea. The target audience for a docstring isn't
the people trying to track down failing tests.

Eric

Re: Changing unittest verbose output. [ In reply to ]

guido at python

Mar 27, 2022, 8:28 AM

Post #4 of 10 (803 views)

Permalink

I recall actually really liking the docstring. I think there are situations
where the test name just can't be sufficiently explanatory. IIRC in the
past we didn't print the docstring at all and it felt right when it was
added.

ATM I'm not using unittest for anything so I don't have a strong opinion.

On Sun, Mar 27, 2022 at 8:21 AM Eric V. Smith <eric@trueblade.com> wrote:

> On 3/27/2022 10:51 AM, Jelle Zijlstra wrote:
>
>
>
> El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya (<itay.yeshaya@gmail.com>)
> escribió:
>
>>
>> Any thoughts on the mentioned solutions?
>>
>
> This is a bit more out there, but can anyone explain why we show the
> docstring at all? As far as I can tell it causes a bunch of confusion and
> helps little. If there's a failing test, I want the test name so I can
> quickly find it in the code; I don't want the docstring which may not even
> be unique.
>
>
> I think this is a good idea. The target audience for a docstring isn't the
> people trying to track down failing tests.
>
> Eric
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/BTF4LZWVTHNHBJVNHXNWSPB2WS4324TO/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Re: Changing unittest verbose output. [ In reply to ]

eric at trueblade

Mar 27, 2022, 10:44 AM

Post #5 of 10 (803 views)

Permalink

> On Mar 27, 2022, at 11:20 AM, Eric V. Smith <eric@trueblade.com> wrote:
>
> ?
> On 3/27/2022 10:51 AM, Jelle Zijlstra wrote:
>>
>>
>> El sáb, 26 mar 2022 a las 17:51, Itay Yeshaya (<itay.yeshaya@gmail.com>) escribió:
>>>
>>> Any thoughts on the mentioned solutions?
>>
>> This is a bit more out there, but can anyone explain why we show the docstring at all? As far as I can tell it causes a bunch of confusion and helps little. If there's a failing test, I want the test name so I can quickly find it in the code; I don't want the docstring which may not even be unique.
>>
> I think this is a good idea. The target audience for a docstring isn't the people trying to track down failing tests.
>
I’m wrong about the docstrings having a different audience. So I’m going to retract my support here.

Eric

Re: Changing unittest verbose output. [ In reply to ]

barry at python

Mar 27, 2022, 12:46 PM

Post #6 of 10 (803 views)

Permalink

On Mar 26, 2022, at 17:48, Itay Yeshaya <itay.yeshaya@gmail.com> wrote:
>
> When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word.

This has been a long-standing nuisance, but I’m like Guido. I pretty much use pytest for everything these days, except for maybe unittest.mock.patch.

-Barry

Re: Changing unittest verbose output. [ In reply to ]

greg at krypto

Mar 27, 2022, 2:40 PM

Post #7 of 10 (803 views)

Permalink

For many of us, this isn't a nuisance. It is a desirable feature.

test_xxx functions and methods typically don't have docstrings at all as
the majority of tests tend to be concise and obvious with the function name
itself describes what its purpose is. When we added printing the
docstring, it was useful, as you can expand upon the purpose of the test
more than you can in a reasonable name within the docstring. We do this
all the many times in our own test suite. When running the tests, you see
the docstring and are given more context as to what the test is about.
This can be useful when triaging a failure before you've even loaded the
source.

I don't doubt that someone writes thesis defenses and stores them in their
TestCase.test_method docstrings. I'm just saying that is not the norm.

I'd accept a PR adding another command line switch for unittest to disable
docstring printing in verbose mode.

-gps

On Sun, Mar 27, 2022 at 12:59 PM Barry Warsaw <barry@python.org> wrote:

> On Mar 26, 2022, at 17:48, Itay Yeshaya <itay.yeshaya@gmail.com> wrote:
> >
> > When running unittest with the -v flag, if a test has errors, and has a
> docstring, the test name is shown on one line, and the docstring is shown
> on the next line with the `ERROR` word.
>
> This has been a long-standing nuisance, but I’m like Guido. I pretty much
> use pytest for everything these days, except for maybe unittest.mock.patch.
>
> -Barry
>
>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/ZIMXSRQMWFOE4U3C3MBK6SG5TH26PDRL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

Re: Changing unittest verbose output. [ In reply to ]

guido at python

Mar 27, 2022, 2:49 PM

Post #8 of 10 (803 views)

Permalink

Hopefully it only prints the first line of the docstring?

On Sun, Mar 27, 2022 at 2:42 PM Gregory P. Smith <greg@krypto.org> wrote:

> For many of us, this isn't a nuisance. It is a desirable feature.
>
> test_xxx functions and methods typically don't have docstrings at all as
> the majority of tests tend to be concise and obvious with the function name
> itself describes what its purpose is. When we added printing the
> docstring, it was useful, as you can expand upon the purpose of the test
> more than you can in a reasonable name within the docstring. We do this
> all the many times in our own test suite. When running the tests, you see
> the docstring and are given more context as to what the test is about.
> This can be useful when triaging a failure before you've even loaded the
> source.
>
> I don't doubt that someone writes thesis defenses and stores them in their
> TestCase.test_method docstrings. I'm just saying that is not the norm.
>
> I'd accept a PR adding another command line switch for unittest to disable
> docstring printing in verbose mode.
>
> -gps
>
>
> On Sun, Mar 27, 2022 at 12:59 PM Barry Warsaw <barry@python.org> wrote:
>
>> On Mar 26, 2022, at 17:48, Itay Yeshaya <itay.yeshaya@gmail.com> wrote:
>> >
>> > When running unittest with the -v flag, if a test has errors, and has a
>> docstring, the test name is shown on one line, and the docstring is shown
>> on the next line with the `ERROR` word.
>>
>> This has been a long-standing nuisance, but I’m like Guido. I pretty
>> much use pytest for everything these days, except for maybe
>> unittest.mock.patch.
>>
>> -Barry
>>
>>
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/ZIMXSRQMWFOE4U3C3MBK6SG5TH26PDRL/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
> _______________________________________________
> Python-Dev mailing list -- python-dev@python.org
> To unsubscribe send an email to python-dev-leave@python.org
> https://mail.python.org/mailman3/lists/python-dev.python.org/
> Message archived at
> https://mail.python.org/archives/list/python-dev@python.org/message/RJOWQWMMUMBJAZZTWXWEAJSRDVARA2XL/
> Code of Conduct: http://python.org/psf/codeofconduct/
>

--
--Guido van Rossum (python.org/~guido)
*Pronouns: he/him **(why is my pronoun here?)*
<http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>

Re: Changing unittest verbose output. [ In reply to ]

greg at krypto

Mar 27, 2022, 2:53 PM

Post #9 of 10 (803 views)

Permalink

On Sun, Mar 27, 2022 at 2:50 PM Guido van Rossum <guido@python.org> wrote:

> Hopefully it only prints the first line of the docstring?
>

Indeed, it's just the first line (traditionally a one line summary per
style).

https://github.com/python/cpython/blob/f6b3a07b7df60dc04d0260169ffef6e9796a2124/Lib/unittest/runner.py#L46

>
> On Sun, Mar 27, 2022 at 2:42 PM Gregory P. Smith <greg@krypto.org> wrote:
>
>> For many of us, this isn't a nuisance. It is a desirable feature.
>>
>> test_xxx functions and methods typically don't have docstrings at all as
>> the majority of tests tend to be concise and obvious with the function name
>> itself describes what its purpose is. When we added printing the
>> docstring, it was useful, as you can expand upon the purpose of the test
>> more than you can in a reasonable name within the docstring. We do this
>> all the many times in our own test suite. When running the tests, you see
>> the docstring and are given more context as to what the test is about.
>> This can be useful when triaging a failure before you've even loaded the
>> source.
>>
>> I don't doubt that someone writes thesis defenses and stores them in
>> their TestCase.test_method docstrings. I'm just saying that is not the norm.
>>
>> I'd accept a PR adding another command line switch for unittest to
>> disable docstring printing in verbose mode.
>>
>> -gps
>>
>>
>> On Sun, Mar 27, 2022 at 12:59 PM Barry Warsaw <barry@python.org> wrote:
>>
>>> On Mar 26, 2022, at 17:48, Itay Yeshaya <itay.yeshaya@gmail.com> wrote:
>>> >
>>> > When running unittest with the -v flag, if a test has errors, and has
>>> a docstring, the test name is shown on one line, and the docstring is shown
>>> on the next line with the `ERROR` word.
>>>
>>> This has been a long-standing nuisance, but I’m like Guido. I pretty
>>> much use pytest for everything these days, except for maybe
>>> unittest.mock.patch.
>>>
>>> -Barry
>>>
>>>
>>> _______________________________________________
>>> Python-Dev mailing list -- python-dev@python.org
>>> To unsubscribe send an email to python-dev-leave@python.org
>>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>>> Message archived at
>>> https://mail.python.org/archives/list/python-dev@python.org/message/ZIMXSRQMWFOE4U3C3MBK6SG5TH26PDRL/
>>> Code of Conduct: http://python.org/psf/codeofconduct/
>>>
>> _______________________________________________
>> Python-Dev mailing list -- python-dev@python.org
>> To unsubscribe send an email to python-dev-leave@python.org
>> https://mail.python.org/mailman3/lists/python-dev.python.org/
>> Message archived at
>> https://mail.python.org/archives/list/python-dev@python.org/message/RJOWQWMMUMBJAZZTWXWEAJSRDVARA2XL/
>> Code of Conduct: http://python.org/psf/codeofconduct/
>>
>
>
> --
> --Guido van Rossum (python.org/~guido)
> *Pronouns: he/him **(why is my pronoun here?)*
> <http://feministing.com/2015/02/03/how-using-they-as-a-singular-pronoun-can-change-the-world/>
>

Re: Changing unittest verbose output. [ In reply to ]

antoine at python

Mar 29, 2022, 10:21 AM

Post #10 of 10 (803 views)

Permalink

On Sun, 27 Mar 2022 00:48:04 -0000
"Itay Yeshaya" <itay.yeshaya@gmail.com> wrote:
> When running unittest with the -v flag, if a test has errors, and has a docstring, the test name is shown on one line, and the docstring is shown on the next line with the `ERROR` word.
> For example:
>
> ./python.exe -m unittest -v Lib.test_verbose_error.randomTest
> test_one (Lib.test_verbose_error.randomTest)
> this is a doc string with helpful info ... ERROR
>
> ======================================================================
> ERROR: test_one (Lib.test_verbose_error.randomTest)
> this is a doc string with helpful info
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/workspaces/cpython/Lib/test_verbose_error.py", line 11, in test_one
> assert l[1]
> ~^^^
> IndexError: list index out of range
>
> ----------------------------------------------------------------------
> Ran 1 test in 0.004s
>
> This cause for some confusion and we would like to fix it, the possible options are:

Which confusion is this about?

Unit tests are not APIs, so the only reason to write a docstring is if
you want it to appear in test output. If you don't want it to appear
in test output, write a comment instead.

Regards

Antoine.

_______________________________________________
Python-Dev mailing list -- python-dev@python.org
To unsubscribe send an email to python-dev-leave@python.org
https://mail.python.org/mailman3/lists/python-dev.python.org/
Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/WIWR4F3M4RD6CPA5QTN2VGTPRXEQ3VDT/
Code of Conduct: http://python.org/psf/codeofconduct/

Mailing List Archive

Mailing List Archive

Attached Files: