Mailing List Archive: Getting at line numbers with parsermodule

Getting at line numbers with parsermodule

Aug 10, 1999, 2:29 PM

Post #1 of 5 (398 views)

I'm trying to work up some syntax- and consistency-checking tools, and
rather than creating my own Python parser (a la "pylint") I'm trying
to use the parser module.

It's coming along OK (modulo not having enough free time to work on
it), except I've run into one snag. I can find certain "bad" patterns
in the AST - one thing in particular I'm looking for is code like
"%s %s %s" % (1,2) - but once I've identified this code I have trouble
printing out the line number in the source where the offending code
occurred. I tried to simply go throught the AST object (converted to
a tuple) and look for NEWLINE tokens, count these as I go throught the
AST to get a current line-count. This doesn't seem to work correctly
since there are fewer NEWLINE tokens than there were lines in the
original file.

What I'd like to be able to do is, given any node in an AST object
returned by compile.suite, is to know the line number in the input
that corresponds to this node. Is this doable? Any suggestions?

Getting at line numbers with parsermodule [ In reply to ]

<fdrake at acm

Aug 10, 1999, 2:54 PM

Post #2 of 5 (397 views)

Permalink

Charles G Waldman writes:
> What I'd like to be able to do is, given any node in an AST object
> returned by compile.suite, is to know the line number in the input
> that corresponds to this node. Is this doable? Any suggestions?

Charles,
To get at the nodes, you're using either the totuple() or tolist()
methods of the AST objects (or ast2tuple(), ast2list()). There is an
optional parameter which may be set to true (1 works nicely) to get
line numbers for each terminal token. One issue: the numbers are
those stored in the "struct node" structure, which is the line number
of the end of token. Getting the line number of the token start is
simple:

>>> import parser, pprint, string
>>> ast = parser.expr('"""foo\nbar"""')
>>> pprint(ast.totuple(1))
(258,
(308,
(288,
(289,
(290,
(291,
(293,
(294,
(295,
(296,
(297,
(298, (299, (300, (301, (3, '"""foo\012bar"""', 2))))))))))))))),
(4, '', 2),
(0, '', 2))
>>> # this is bolluxed -----^
... 2 - string.count('"""foo\nar"""', '\n')
1
>>>

I hope this helps!

-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives

Getting at line numbers with parsermodule [ In reply to ]

billtut at microsoft

Aug 10, 1999, 5:00 PM

Post #3 of 5 (396 views)

Permalink

> From: Charles G Waldman [mailto:cgw@fnal.gov]
>

> What I'd like to be able to do is, given any node in an AST object
> returned by compile.suite, is to know the line number in the input
> that corresponds to this node. Is this doable? Any suggestions?
>

You could just use parser.ast2tuple(ast, 1)
The extra parameter adds the line # as the last element of each AST node
tuple.

Bill

Getting at line numbers with parsermodule [ In reply to ]

tim_one at email

Aug 10, 1999, 8:23 PM

Post #4 of 5 (396 views)

Permalink

[Charles G Waldman]
> ...
> I'm trying to use the parser module.
> ...
> I have trouble printing out the line number in the source where
> the offending code occurred.

RTFM for this one: ast2list and ast2tuple each take an optional line_info
argument, defaulting to false.

> I tried to simply go throught the AST object (converted to a tuple) and
> look for NEWLINE tokens, count these as I go throught the AST to get a
> current line-count. This doesn't seem to work correctly since there are
> fewer NEWLINE tokens than there were lines in the original file.

Right, a NEWLINE in Python's grammar occurs at the end of a statement, and
not every physical \n ends a statement. For example, you won't get NEWLINE
at the end of any line that's continued (via string, backslash, or
open-brace/bracket/paren continuation). So far as Python's parsing is
concerned the statement hasn't ended yet.

> What I'd like to be able to do is, given any node in an AST object
> returned by compile.suite, is to know the line number in the input
> that corresponds to this node. Is this doable?

No. Specifying a true line_info argument will get you line numbers for
terminal tokens, though.

would-the-manual-lie?-ly y'rs - tim

Getting at line numbers with parsermodule [ In reply to ]

<fdrake at acm

Aug 11, 1999, 5:56 AM

Post #5 of 5 (398 views)

Permalink

Tim Peters writes:
> would-the-manual-lie?-ly y'rs - tim

Hee hee hee...

-Fred

--
Fred L. Drake, Jr. <fdrake@acm.org>
Corporation for National Research Initiatives