Mailing List Archive

on the popularity of loops while and for
I'd like get a statistic of how often each loop is used in practice.

I was trying to take a look at the Python's standard libraries --- those
included in a standard installation of Python 3.9.6, say --- to see
which loops are more often used among while and for loops. Of course,
since English use the preposition ``for'' a lot, that makes my life
harder. Removing comments is easy, but removing strings is harder. So
I don't know yet what I'll do.

Have you guys ever measured something like that in a casual or serious
way? I'd love to know. Thank you!
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
ram@zedat.fu-berlin.de (Stefan Ram) writes:

> Hope Rouselle <hrouselle@jevedi.com> writes:
>>Have you guys ever measured something like that in a casual or serious
>
> import ast
> import pathlib
> rootname=r'''<ABSOLUTE PATH TO A DIRECTORY WITH SOURCE CODE>'''
> rootpath=pathlib.Path(rootname)
> rootiterable=rootpath.glob('**/*.py')
> first = True
> WhileCount = 0
> ForCount = 0
> for filepath in rootiterable:
> try:
> with filepath.open(encoding="utf-8") as file:
> source=file.read()
> parse=ast.parse(source, filename=str(file))
> for entry in ast.walk(parse):
> if isinstance(entry, ast.While):
> WhileCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> elif isinstance(entry, ast.For):
> ForCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> except SyntaxError as inst:
> if first:
> print( f"{sys.exc_info()[ 0 ] =}" )
> print( f"{type( inst ) =}" )
> print( f"{inst.args =}" )
> print( f"{inst =}" )
> print( f"skipping {filepath}." )
> first=False

You are so wonderful! Thanks quite a lot. Here's what I got:

ForCount=18703, WhileCount=2505

I have pretty much just the standard libraries and a couple more ---
sympy and some xlsxwriter library.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
On Sun, Aug 29, 2021 at 7:40 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>
> I'd like get a statistic of how often each loop is used in practice.
>
> I was trying to take a look at the Python's standard libraries --- those
> included in a standard installation of Python 3.9.6, say --- to see
> which loops are more often used among while and for loops. Of course,
> since English use the preposition ``for'' a lot, that makes my life
> harder. Removing comments is easy, but removing strings is harder. So
> I don't know yet what I'll do.
>
> Have you guys ever measured something like that in a casual or serious
> way? I'd love to know. Thank you!

For analysis like this, I recommend using the Abstract Syntax Tree:

https://docs.python.org/3/library/ast.html

You can take a Python source file, parse it to the AST, and then walk
that tree to see what it's using. That will avoid any false positives
from the word "for" coming up in the wrong places.

ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
On 8/28/2021 9:31 AM, Hope Rouselle wrote:
> I'd like get a statistic of how often each loop is used in practice.

My guess is that for loops are at least twice as common as while loops.

> I was trying to take a look at the Python's standard libraries --- those
> included in a standard installation of Python 3.9.6, say --- to see
> which loops are more often used among while and for loops. Of course,
> since English use the preposition ``for'' a lot, that makes my life
> harder. Removing comments is easy, but removing strings is harder. So
> I don't know yet what I'll do.

Try something like

fors = 0
for file in files:
for line in file:
if re.match(r"/s*for .* in ", line):
fors += 1

This excludes comprehensions, which have replaced some for loops.

--
Terry Jan Reedy

--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
If you're doing this analysis, I'd be pretty interested in how many of
those while loops where 'while True:'

I'd wager 75%. But may be completely off

Steve


On Sat, 28 Aug 2021 at 23:03, Hope Rouselle <hrouselle@jevedi.com> wrote:

> ram@zedat.fu-berlin.de (Stefan Ram) writes:
>
> > Hope Rouselle <hrouselle@jevedi.com> writes:
> >>Have you guys ever measured something like that in a casual or serious
> >
> > import ast
> > import pathlib
> > rootname=r'''<ABSOLUTE PATH TO A DIRECTORY WITH SOURCE CODE>'''
> > rootpath=pathlib.Path(rootname)
> > rootiterable=rootpath.glob('**/*.py')
> > first = True
> > WhileCount = 0
> > ForCount = 0
> > for filepath in rootiterable:
> > try:
> > with filepath.open(encoding="utf-8") as file:
> > source=file.read()
> > parse=ast.parse(source, filename=str(file))
> > for entry in ast.walk(parse):
> > if isinstance(entry, ast.While):
> > WhileCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > elif isinstance(entry, ast.For):
> > ForCount+=1
> > print( f"{ForCount=}, {WhileCount=}" )
> > except SyntaxError as inst:
> > if first:
> > print( f"{sys.exc_info()[ 0 ] =}" )
> > print( f"{type( inst ) =}" )
> > print( f"{inst.args =}" )
> > print( f"{inst =}" )
> > print( f"skipping {filepath}." )
> > first=False
>
> You are so wonderful! Thanks quite a lot. Here's what I got:
>
> ForCount=18703, WhileCount=2505
>
> I have pretty much just the standard libraries and a couple more ---
> sympy and some xlsxwriter library.
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
On Sun, 29 Aug 2021 at 00:04, Stefan Ram <ram@zedat.fu-berlin.de> wrote:

> Stestagg <stestagg@gmail.com> writes:
> >If you're doing this analysis, I'd be pretty interested in how many of
> >those while loops where 'while True:'
>
> Here, about 40 %.



Thanks! That's an interesting stat. I might have to look more into how
while loops are actually used in other people's code.


>
> Below is a new version that is supposed to also count "if True:"
> and "if 1:" and also removes a bug with the exception handling
> (missing "import sys" and missing "UnicodeDecodeError"). I'm not
> experienced in using the AST module, so there might still be bugs
> in the program!
>
> import ast
> import pathlib
> import sys
> rootname=r'''PATH TO SOURCE DIR (WITHOUT A BACKQUOTE AT THE END)'''
> rootpath=pathlib.Path(rootname)
> rootiterable=rootpath.glob('**/*.py')
> WhileCount = 0
> TrueCount = 0
> ForCount = 0
> for filepath in rootiterable:
> try:
> with filepath.open(encoding="utf-8") as file:
> source=file.read()
> parse=ast.parse(source, filename=str(file))
> for entry in ast.walk(parse):
> if isinstance(entry, ast.While):
> WhileCount+=1
> if ( isinstance( entry.test, ast.Constant ) and
> hasattr( entry.test, "value") and
> ( isinstance( entry.test.value, bool )
> and entry.test.value == True or
> isinstance( entry.test.value, int )
> and entry.test.value != 0 )):
> TrueCount+=1
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
> elif isinstance(entry, ast.For):
> ForCount+=1
> print( f"{ForCount=}, {WhileCount=}" )
> except ( SyntaxError, UnicodeDecodeError )as inst:
> print( f"skipping {filepath}." )
> print( f"{sys.exc_info()[ 0 ] =}" )
> print( f"{type( inst ) =}" )
> print( f"{inst.args =}" )
> print( f"{inst =}\n" )
> print( f"{ForCount=}, {WhileCount=}, {TrueCount=}" )
>
>
> --
> https://mail.python.org/mailman/listinfo/python-list
>
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
?

> On 28 Aug 2021, at 22:42, Hope Rouselle <hrouselle@jevedi.com> wrote:
>
> ?I'd like get a statistic of how often each loop is used in practice.
>
> I was trying to take a look at the Python's standard libraries --- those
> included in a standard installation of Python 3.9.6, say --- to see
> which loops are more often used among while and for loops. Of course,
> since English use the preposition ``for'' a lot, that makes my life
> harder. Removing comments is easy, but removing strings is harder. So
> I don't know yet what I'll do.
>
> Have you guys ever measured something like that in a casual or serious
> way? I'd love to know. Thank you!

I am interesting in why you think that choice of while vs. for is about popularity?

Surely the choice is made in most cases by the algorithm?
If you have an iterator then you use for.
Of course you can use while instead of for but in code review that will get queried.

Barry



> --
> https://mail.python.org/mailman/listinfo/python-list

--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
Chris Angelico <rosuav@gmail.com> writes:

> On Sun, Aug 29, 2021 at 7:40 AM Hope Rouselle <hrouselle@jevedi.com> wrote:
>>
>> I'd like get a statistic of how often each loop is used in practice.
>>
>> I was trying to take a look at the Python's standard libraries --- those
>> included in a standard installation of Python 3.9.6, say --- to see
>> which loops are more often used among while and for loops. Of course,
>> since English use the preposition ``for'' a lot, that makes my life
>> harder. Removing comments is easy, but removing strings is harder. So
>> I don't know yet what I'll do.
>>
>> Have you guys ever measured something like that in a casual or serious
>> way? I'd love to know. Thank you!
>
> For analysis like this, I recommend using the Abstract Syntax Tree:
>
> https://docs.python.org/3/library/ast.html
>
> You can take a Python source file, parse it to the AST, and then walk
> that tree to see what it's using. That will avoid any false positives
> from the word "for" coming up in the wrong places.

Wonderful. Someone here actually posted a nice ``routine'' that does
the job marvelously well. I had the impression the poster posted only
to solve my request --- impressive and appreciated.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
Terry Reedy <tjreedy@udel.edu> writes:

> On 8/28/2021 9:31 AM, Hope Rouselle wrote:
>> I'd like get a statistic of how often each loop is used in practice.
>
> My guess is that for loops are at least twice as common as while loops.

Scanning just the Python 3.9.6's Lib/ directory --- using the ast module
and the program someone kindly posted here ---, I found for loop
appearing 598% more times than while loops. The ratio was something
like that 6.98..., excluding files that due to encoding errors (or
whatever) could not be parsed. (I reported these numbers elsewhere in
this thread.)

>> I was trying to take a look at the Python's standard libraries --- those
>> included in a standard installation of Python 3.9.6, say --- to see
>> which loops are more often used among while and for loops. Of course,
>> since English use the preposition ``for'' a lot, that makes my life
>> harder. Removing comments is easy, but removing strings is harder. So
>> I don't know yet what I'll do.
>
> Try something like
>
> fors = 0
> for file in files:
> for line in file:
> if re.match(r"/s*for .* in ", line):
> fors += 1
>
> This excludes comprehensions, which have replaced some for loops.

That's nice too. Thanks for the alternative.
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
Barry <barry@barrys-emacs.org> writes:

>> On 28 Aug 2021, at 22:42, Hope Rouselle <hrouselle@jevedi.com> wrote:
>>
>> ?I'd like get a statistic of how often each loop is used in practice.
>>
>> I was trying to take a look at the Python's standard libraries --- those
>> included in a standard installation of Python 3.9.6, say --- to see
>> which loops are more often used among while and for loops. Of course,
>> since English use the preposition ``for'' a lot, that makes my life
>> harder. Removing comments is easy, but removing strings is harder. So
>> I don't know yet what I'll do.
>>
>> Have you guys ever measured something like that in a casual or serious
>> way? I'd love to know. Thank you!
>
> I am interesting in why you think that choice of while vs. for is
> about popularity?

Perhaps you think my choice of "popular" had a special judgement about
it. Let's define "popular" in the way that I used it.

--8<---------------cut here---------------start------------->8---
Definition. We say x is more popular than y if x appears more times
relative to y when considered in a sample.

Example. For loops are more popular than while loops in a certain
sample because they appeared 17 times compared to 5 times the while loop
appeared.
--8<---------------cut here---------------end--------------->8---

So it's just a numeric comparison.

[...]
--
https://mail.python.org/mailman/listinfo/python-list
Re: on the popularity of loops while and for [ In reply to ]
On 2021-08-29 10:04:47 +0100, Barry wrote:
> > I'd like get a statistic of how often each loop is used in practice.
> >
> > I was trying to take a look at the Python's standard libraries --- those
> > included in a standard installation of Python 3.9.6, say --- to see
> > which loops are more often used among while and for loops. Of course,
> > since English use the preposition ``for'' a lot, that makes my life
> > harder. Removing comments is easy, but removing strings is harder. So
> > I don't know yet what I'll do.
> >
> > Have you guys ever measured something like that in a casual or serious
> > way? I'd love to know. Thank you!
>
> I am interesting in why you think that choice of while vs. for is
> about popularity?
>
> Surely the choice is made in most cases by the algorithm?

The context here is an introductory course for Python. So there is not
"the algorithm", there are all the algorithms that a novice is likely
to encounter.

For me it makes absolute sense to base the contents of the course on
popularity. Constructs which a novice programmer is very likely to use
or encounter in other people's code should be covered more thoroughly
than constructs that will be used only rarely. Some are so rare that
they can be safely omitted. The while loop is certainly not in that
category, but it probably makes sense to spend less time on it than on
the for loop.

hp

--
_ | Peter J. Holzer | Story must make more sense than reality.
|_|_) | |
| | | hjp@hjp.at | -- Charles Stross, "Creative writing
__/ | http://www.hjp.at/ | challenge!"
Re: on the popularity of loops while and for [ In reply to ]
"Peter J. Holzer" <hjp-python@hjp.at> writes:

> On 2021-08-29 10:04:47 +0100, Barry wrote:
>> > I'd like get a statistic of how often each loop is used in practice.
>> >
>> > I was trying to take a look at the Python's standard libraries --- those
>> > included in a standard installation of Python 3.9.6, say --- to see
>> > which loops are more often used among while and for loops. Of course,
>> > since English use the preposition ``for'' a lot, that makes my life
>> > harder. Removing comments is easy, but removing strings is harder. So
>> > I don't know yet what I'll do.
>> >
>> > Have you guys ever measured something like that in a casual or serious
>> > way? I'd love to know. Thank you!
>>
>> I am interesting in why you think that choice of while vs. for is
>> about popularity?
>>
>> Surely the choice is made in most cases by the algorithm?
>
> The context here is an introductory course for Python. So there is not
> "the algorithm", there are all the algorithms that a novice is likely
> to encounter.
>
> For me it makes absolute sense to base the contents of the course on
> popularity. Constructs which a novice programmer is very likely to use
> or encounter in other people's code should be covered more thoroughly
> than constructs that will be used only rarely. Some are so rare that
> they can be safely omitted. The while loop is certainly not in that
> category, but it probably makes sense to spend less time on it than on
> the for loop.

``A man after my own heart.''
--
https://mail.python.org/mailman/listinfo/python-list