Mailing List Archive

Python equivalent of #line?
C has a preprocessor command
#line n "filename"
Can I do this in Python?

Ed Jones
Python equivalent of #line? [ In reply to ]
Sorry, there is no way to influence the compilers notion of the current
line, but you can patch the generated code.

HTH
Gerald

Edward C. Jones wrote:
> C has a preprocessor command
> #line n "filename"
> Can I do this in Python?
>
> Ed Jones
>
Python equivalent of #line? [ In reply to ]
Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> C has a preprocessor command
> #line n "filename"
> Can I do this in Python?

Why do you want that?

You can get the current linenumber:
14:49:50:2:gerrit@stopcontact:/tmp$ cat test.py
#!/usr/bin/python

import sys

print sys._getframe().f_lineno; print sys._getframe().f_lineno

for i in xrange(3):
print sys._getframe().f_lineno
14:49:53:2:gerrit@stopcontact:/tmp$ python test.py
5
5
8
8
8

yours,
Gerrit.

--
203. If a free-born man strike the body of another free-born man or
equal rank, he shall pay one gold mina.
-- Hammurabi, Code of Law
--
Asperger Syndroom - een persoonlijke benadering:
http://people.nl.linux.org/~gerrit/
Het zijn tijden om je zelf met politiek te bemoeien:
http://www.sp.nl/
Python equivalent of #line? [ In reply to ]
[Gerrit Holl]

> Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> > C has a preprocessor command
> > #line n "filename"
> > Can I do this in Python?

> Why do you want that?

For the case you drive generation of Python code from lengthy data files.
You may want tracebacks to refer to your data files instead of generated
Python code, which is not your real sources, nor where you will point your
editor for making the real corrections.

It would be nice if the Python compiler was supporting such directives.
Even if I agree many people would not need it, it would be extremely useful
to those who do. It would be also useful for developers being able to ask
the Python generator to disregard those `#line' directives when the time
comes to debug the Python generators themselves.

--
François Pinard http://www.iro.umontreal.ca/~pinard
Python equivalent of #line? [ In reply to ]
François Pinard wrote:
> [Gerrit Holl]
>
>
>>Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
>>
>>>C has a preprocessor command
>>> #line n "filename"
>>>Can I do this in Python?
>
>
>>Why do you want that?
>
>
> For the case you drive generation of Python code from lengthy data files.
> You may want tracebacks to refer to your data files instead of generated
> Python code, which is not your real sources, nor where you will point your
> editor for making the real corrections.

That is exactly what I am doing.

> It would be nice if the Python compiler was supporting such directives.
> Even if I agree many people would not need it, it would be extremely useful
> to those who do. It would be also useful for developers being able to ask
> the Python generator to disregard those `#line' directives when the time
> comes to debug the Python generators themselves.

Agreed.
Python equivalent of #line? [ In reply to ]
On 25 May 2003 09:37:31 -0400, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:

>[Gerrit Holl]
>
>> Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
>> > C has a preprocessor command
>> > #line n "filename"
>> > Can I do this in Python?
>
>> Why do you want that?
>
>For the case you drive generation of Python code from lengthy data files.
>You may want tracebacks to refer to your data files instead of generated
>Python code, which is not your real sources, nor where you will point your
>editor for making the real corrections.
>
>It would be nice if the Python compiler was supporting such directives.
>Even if I agree many people would not need it, it would be extremely useful
>to those who do. It would be also useful for developers being able to ask
>the Python generator to disregard those `#line' directives when the time
>comes to debug the Python generators themselves.
>
If you were e.g., generating gen.py from gdata.txt, you could at the same time
generate a vim tag file to translate key line locations of gen.py to gdata.txt
locations (since you would know both line numbers from your reading and writing).
UIAM, a line in a tag file of the form (note tab delimiters mandatory by default):

gen123\tgdata.txt\t456

would let you start vim and type :ta /gen123 and get right to tgdata.txt at line 456

If you made the tags file "gen.tags" instead of the default "tags" you could start vim
at the first tagged line something like

vim -c ":set tags=gen.tags|ta /gen"

and then step through them all with :tn or go to a particular one with :ta /genXXX

Regards,
Bengt Richter
Python equivalent of #line? [ In reply to ]
[Bengt Richter]

> On 25 May 2003 09:37:31 -0400, pinard@iro.umontreal.ca (=?iso-8859-1?q?Fran=E7ois_Pinard?=) wrote:

> >[Gerrit Holl]
> >
> >> Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> >> > C has a preprocessor command
> >> > #line n "filename"
> >> > Can I do this in Python?
> >
> >> Why do you want that?
> >
> >For the case you drive generation of Python code from lengthy data files.
> >You may want tracebacks to refer to your data files instead of generated
> >Python code, which is not your real sources, nor where you will point your
> >editor for making the real corrections.

> If you were e.g., generating gen.py from gdata.txt, you could at the same
> time generate a vim tag file to translate key line locations of gen.py to
> gdata.txt locations [...]

Indeed, this might cleverly help at edit time. On the other hand, it
creates some burden on the shoulders of developers for generating these tag
files, keeping them in sync, distributing them, explaining other users how
to use them, etc. We would need to generate `tags' for `vim' and a
differently formatted `TAGS' for Emacs; while many other editors just do not
support such files. Finally, the ability of tracebacks to quote one source
line per traceback level would not necessarily be served as we would expect.

It might be more functional and useful if supported by the Python compiler.

--
François Pinard http://www.iro.umontreal.ca/~pinard
Python equivalent of #line? [ In reply to ]
On Sun, May 25, 2003 at 09:37:31AM -0400, François Pinard wrote:
> [Gerrit Holl]
>
> > Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> > > C has a preprocessor command
> > > #line n "filename"
> > > Can I do this in Python?
>
> > Why do you want that?
>
> For the case you drive generation of Python code from lengthy data files ...

I think automatically creating python code is a bad idea. I once had
to deal with an old project which stored data in the syntax of a
programming language. It is hell. Data should be in a format suitable
for data.

I hope this warning does not apply to your project.

thomas

--
Thomas Guettler <guettli@thomas-guettler.de>
http://www.thomas-guettler.de
Python equivalent of #line? [ In reply to ]
[Thomas Guettler]
> [Fran=E7ois Pinard]
> > [Gerrit Holl]
> > > [Edward C. Jones]

> > > > C has a preprocessor command
> > > > #line n "filename"
> > > > Can I do this in Python?
> > > Why do you want that?
> > For the case you drive generation of Python code from lengthy data files

> I think automatically creating python code is a bad idea. I once had to
> deal with an old project which stored data in the syntax of a programming
> language. It is hell. Data should be in a format suitable for data.

Python source is very suitable for expressing data, you know. I have many
cases where this is far more like a blessing than like hell. I guess, from
the mere words we are using, that this may become a religious debate! :-)

For the original case under discussion, however, I think the point was about
Python executable code generated from data files that could hold, for
example, either Python snippets, or algorithms expressed in other languages.
In such cases, `#line'-like directives would be especially useful.

--=20
Fran=C3=A7ois Pinard http://www.iro.umontreal.ca/~pinard
Python equivalent of #line? [ In reply to ]
Thomas Guettler wrote:
> I once had
> to deal with an old project which stored data in the syntax of a
> programming language. It is hell. Data should be in a format suitable
> for data.

Generating Python code from another programming language
wouldn't be such a bad idea, though. One of my potential
future projects, for example, involves taking a file of
SQL statements and generating Python functions which
wrap them.

It occurs to me that one might be able to hack the
line number table in the bytecode, if one were feeling
brave enough -- it's stored in a rather funky format.

It would be nice to have a nicer way, though. Maybe
something like

__line__ = <const>

--
Greg Ewing, Computer Science Dept,
University of Canterbury,
Christchurch, New Zealand
http://www.cosc.canterbury.ac.nz/~greg
Python equivalent of #line? [ In reply to ]
Quoth Thomas Guettler:
[...]
> I think automatically creating python code is a bad idea. I once had
> to deal with an old project which stored data in the syntax of a
> programming language. It is hell. Data should be in a format suitable
> for data.

Code generation makes sense in some situations; Knuth's "literate
programming" methodology comes to mind. (And having line numbers
refer to the original source makes good sense there.)

--
Steven Taschuk staschuk@telusplanet.net
Every public frenzy produces legislation purporting to address it.
(Kinsley's Law)
Python equivalent of #line? [ In reply to ]
Greg Ewing wrote:
>
> It would be nice to have a nicer way, though. Maybe
> something like
>
> __line__ = <const>

Agreed.

For the next Capon version I'm doing this: Filenames and line numbers
are encoded in function names, and exceptions are intercepted and
patched. This works well; the only drawback is that patched
tracebacks (presumably, haven't tried it) don't play well with pdb.

I'll see if I can factor this functionality out into a separate module
for reuse.

- Anders
Python equivalent of #line? [ In reply to ]
pinard@iro.umontreal.ca (François Pinard) writes:

> > Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> > > C has a preprocessor command
> > > #line n "filename"
> > > Can I do this in Python?

> For the case you drive generation of Python code from lengthy data files.
> You may want tracebacks to refer to your data files instead of generated
> Python code, which is not your real sources, nor where you will point your
> editor for making the real corrections.

I've used a hack to achieve this when compiling pieces of Python code
interspersed through an XML file: first of all, use compile() which
can be passed a filename, and then simply pad the start of the source
being compiled with blank lines, to align it with the real line
numbers in the original file.


--
===============================================================
<erwin@andreasen.org> Herlev, Denmark
<URL:http://www.andreasen.org/> <*>
===============================================================
Python equivalent of #line? [ In reply to ]
[Erwin S. Andreasen]

> pinard@iro.umontreal.ca (Fran=E7ois Pinard) writes:

> > > Edward C. Jones schreef op zondag 25 mei om 14:27:34 +0000:
> > > > C has a preprocessor command
> > > > #line n "filename"
> > > > Can I do this in Python?

> > For the case you drive generation of Python code from lengthy data file=
s.
> > You may want tracebacks to refer to your data files instead of generated
> > Python code, which is not your real sources, nor where you will point y=
our
> > editor for making the real corrections.

> I've used a hack to achieve this when compiling pieces of Python code
> interspersed through an XML file: first of all, use compile() which
> can be passed a filename, and then simply pad the start of the source
> being compiled with blank lines, to align it with the real line
> numbers in the original file.

This idea of using blank lines for aligning the source is cleverly used by
Peter Deutsch's `ansi2knr' utility. It uses a single `#line' directive
right at the start of the output it generates. This works well for the C
language, as we can freely use blank lines (except in strings), and we can
also merge many lines into one if we need to prevent the line counter from
advancing. In case of multi-file sources, because the C pre-processor is
called on `ansi2knr' output, changes of file names are also transmitted
correctly to the C compiler.

I once implemented a similar feature in Ren=C3=A9 Seindal's nice GNU m4, but
could not reuse the blank line trick, as m4 users might want to precisely
control the generation of newlines in their output. Despite this is not
usual in most m4 applications, I tried to do the thing correctly. This is
more complex that one might think given include facilities, diverted
streams, and the capability of assembling a single line from many files.

In the case of Python, the hack you describe might not work so easily, at
least when the Python source is generated from many files -- something which
I had to deal with a few times already.

We could surely do a lot of hackery that would work, more or less in special
cases (and less than more in the general case!), maybe relying on special
editors, hacking bytecode, making the generated code uglier than required,
sometimes making the release engineering more complex that it really has to
be. Here, having Python supporting `#line' directives might be both the
most simple and the most proper thing to do.

--=20
Fran=C3=A7ois Pinard http://www.iro.umontreal.ca/~pinard
Python equivalent of #line? [ In reply to ]
Steven Taschuk <staschuk@telusplanet.net> wrote in message news:<mailman.1053933749.3956.python-list@python.org>...
>
> Code generation makes sense in some situations; Knuth's "literate
> programming" methodology comes to mind. (And having line numbers
> refer to the original source makes good sense there.)

+1 for #line

Very few people have a need for it, but those of us who write
application generators appreciate it very much.

And a real world example. About a decade ago, I was a member of a
team of 4 people who had to implement the central host software for a
statewide vehicle emissions program. Due to drawn out negotiations we
had a bit more than 6 months to implement it and a hard deadline with
significant $ penalties. The approach I chose was to use an
application generator that generated C code from templates controlled
by a template language very much like Velocity with added support for
objects with a very rich typing system. It functiond very much like
JSP or Velocity. The key piece of leverage was generating huge
amounts of SQL and C embedded SQL from object declarations. It took me
3 months to implement the application generator. (I know, if I had
known about Python, I could have done it in a weekend. :-) Even so, we
made the deadline and were inspecting vehicles without a hitch on
opening day.

I compared the generated application to the input templates and found
a 5 to 1 difference in lines of code. Another metric: our team put in
about 3 man-years and met our deadline. A competitor in another state
had a team 2 or 3 times our size. They started a year before their
deadline, missed it by a year, and took another year before the state
would accept the system. Their effort may have been 10 times ours.

Although my project was in C and we could have achieved similar
improvements had we developed the system in Python, the real point is
that you can get huge savings in implementation effort on a large
system by using a domain specific language and generating code in a
lower level target language.

One problem I had in generating C was getting the indentation correct.
The natural thing to do was to use reasonable indentation in writing
the C and the template language in the templates, which produced some
pretty horrible indentation in the output and took a fair amount of
effort in both the generator and the templates to get it right.
Python's use of indentation makes the formatting issue critical in a
template driven system. I suppose using #begin and #end of #{ and #}
and a reformatter would do the trick.

Any effort to support Python-like domain specific languages gets my
support.

Russ
Python equivalent of #line? [ In reply to ]
If you are printing your own tracebacks, you could add your own machinery
to interpret '#line' directives in the .py file. This would be possible
because '#line' will obviously be treated as a comment by Python itself.
See traceback.py for how this is currently done. It seems that in each
place (filename,lineno) is used, you'll need to put a call to your new
function:

# You write this function
def find_origin_line(filename, lineno):
"""return the originating filename and line-number for this
possibly-preprocessed Python file"""
...

# and change these functions
def print_tb(tb, limit=None, file=None):
"""Print up to 'limit' stack trace entries from the traceback 'tb'.

If 'limit' is omitted or None, all entries are printed. If 'file'
is omitted or None, the output goes to sys.stderr; otherwise
'file' should be an open file or file-like object with a write()
method.
"""
if not file:
file = sys.stderr
if limit is None:
if hasattr(sys, 'tracebacklimit'):
limit = sys.tracebacklimit
n = 0
while tb is not None and (limit is None or n < limit):
f = tb.tb_frame
lineno = tb_lineno(tb)
co = f.f_code
filename = co.co_filename
filename, lineno = find_origin_line(filename, lineno) # XXX and similarly elsewhere
name = co.co_name
_print(file,
' File "%s", line %d, in %s' % (filename,lineno,name))
line = linecache.getline(filename, lineno)
if line: _print(file, ' ' + line.strip())
tb = tb.tb_next
n = n+1

Jeff
Python equivalent of #line? [ In reply to ]
Russ Salsbury:
> +1 for #line
>
> Very few people have a need for it, but those of us who write
> application generators appreciate it very much.

Well, I did this by setting a Python variable, 'lineno', in the
generated code, then putting a try/except around the whole
thing so I could get the lineno for reporting.

Also used this trick when parsing input files.

Andrew
dalke@dalkescientific.com