For the edification of all involved, this post summarizes a somewhat surprising behavior in unittest around docstrings.
In bpo-46126<https://bugs.python.org/issue46126>, I reported an issue where I’d observed that CPython developers were avoiding the use of docstrings in unittests due to what was perceived as a masking of crucial information.
On further investigation, it turns out that with the “descriptions” setting of the TextTestRunner is set to True (the default), unittest will emit the first line of a test’s docstring (if present) in addition to the location of the test. As a result, for tests that have docstrings, the output varies depending on this setting. Verbose output with docstrings and descriptions enabled:
test_entry_points_unique_packages (test.test_importlib.test_metadata_api.APITests)
Entry points should only be exposed for the first package ... ERROR
Without docstrings or with descriptions disabled:
test_entry_points_unique_packages (test.test_importlib.test_metadata_api.APITests) ... ERROR
The output with the docstrings is more descriptive, providing more context and detail about the intention of the failed test. Because of the additional detail, however, unittest has elected to use a newline between the test location and the description, meaning the test result no longer appears on the same line as the test location, and thus if one were to grep for the test name, the result would be omitted, and if one were to grep for ERROR, the test name would be omitted.
As part of the investigation, I published In Python, use docstrings or comments?<https://blog.jaraco.com/why-docstrings-are-preferable-to-comments/> exploring the motivations for and value added by allowing docstrings in test functions.
It’s still an open consideration<https://bugs.python.org/issue47133> whether the unittest UX should format descriptions in the output differently to provide more consistent output while still allowing docstrings.
Based on this information, CPython will most likely (pending GH-32128<https://github.com/python/cpython/pull/32128>) continue to use the default behavior (descriptions enabled) but will also allow for docstrings in its own tests.
In bpo-46126<https://bugs.python.org/issue46126>, I reported an issue where I’d observed that CPython developers were avoiding the use of docstrings in unittests due to what was perceived as a masking of crucial information.
On further investigation, it turns out that with the “descriptions” setting of the TextTestRunner is set to True (the default), unittest will emit the first line of a test’s docstring (if present) in addition to the location of the test. As a result, for tests that have docstrings, the output varies depending on this setting. Verbose output with docstrings and descriptions enabled:
test_entry_points_unique_packages (test.test_importlib.test_metadata_api.APITests)
Entry points should only be exposed for the first package ... ERROR
Without docstrings or with descriptions disabled:
test_entry_points_unique_packages (test.test_importlib.test_metadata_api.APITests) ... ERROR
The output with the docstrings is more descriptive, providing more context and detail about the intention of the failed test. Because of the additional detail, however, unittest has elected to use a newline between the test location and the description, meaning the test result no longer appears on the same line as the test location, and thus if one were to grep for the test name, the result would be omitted, and if one were to grep for ERROR, the test name would be omitted.
As part of the investigation, I published In Python, use docstrings or comments?<https://blog.jaraco.com/why-docstrings-are-preferable-to-comments/> exploring the motivations for and value added by allowing docstrings in test functions.
It’s still an open consideration<https://bugs.python.org/issue47133> whether the unittest UX should format descriptions in the output differently to provide more consistent output while still allowing docstrings.
Based on this information, CPython will most likely (pending GH-32128<https://github.com/python/cpython/pull/32128>) continue to use the default behavior (descriptions enabled) but will also allow for docstrings in its own tests.