Mailing List Archive

Macro preprocessor.
Hi folks. I wrote a macro preprocessor that can be used with a language that
uses indendation for statement or declaration grouping, because it is
intelligent about dealing with leading whitespace.

http://users.footprints.net/~kaz/mpp.html

I haven't done work on it in a long while and I need some motivation to to add
features to it to make it more useful. (It needs conditionals, numeric
variables and operations, iteration constructs). It would help, like, to
have some users. :)

I know jack about Python, but I was partially thinking of you guys when I added
whitespace intelligence to the list of requirements. (That and the desire to
have a preprocessor tool that doesn't leave obvious artifacts usually caused by
preprocessing, such as broken formatting, and gives you great control over
whitespace).

In this macro preprocessor you can do something like this:

#def snippet1 {
line1
line2
line3
}

#def snippet2 {
line4
#snippet1
line5
}

def func(x)
#snippet2

and the output will be:

def func(x)
line4
line1
line2
line3
line5

as you would intuitively expect, rather than coming out in some horrible mess
as would happen with a traditional C preprocessor utility, or something like
m4, either of which would make dog's breakfast out of, say, a Python program.

There are other features im MPP. One noteworthy feature is that it supports C++
style nested namespaces:

#namespace html {
#def url(link, text) {<a href="http://#link">#text</a>}
}

#using html.url
#url(www.slashdot.org, slashdot)

#comment { or full scope resolution }
#html.url(www.freshmeat.net, freshmeat)

#comment { create alternate name for something }
#alias web_stuff html
#web_stuff.url(www.kernel.org, kernel)

#comment { and other cheap symbol-table tricks. :) }

With namespaces you can create macro libraries that don't clash. Speaking of
which, there is a fully blown #include directive, with "..." and <...> header
searching logic that can be influenced with command-line options. There is a
#once directive to suppress multiple inclusions of same header. Great
diagnostics that pinpoint the line number and file where an error was caused,
accompanied by a dump of the whole #include chain that led up to the error, if
it was in a header. Oh, and did I mention that you can dynamically modify the
lexical character mapping, similar to the way codes can be redefined in TeX? If
you don't want the # character to introduce directives, you can change it to
something else on the fly. (If you change it within a header, the change won't
automatically be propagated to the host file that included it.)

#code leftbrace [
#code rightbrace ]
#code escape %

%def foo [ bar ]

Okay, enough. :) I'll probably get hate mail now about being off topic.
Macro preprocessor. [ In reply to ]
Kaz Kylheku schrieb in Nachricht ...
>Hi folks. I wrote a macro preprocessor that can be used with a language
that
>uses indendation for statement or declaration grouping, because it is
>intelligent about dealing with leading whitespace.
>
>http://users.footprints.net/~kaz/mpp.html

Looks exactly like one of the features i missed in Python...

<duck>

Dirk
Macro preprocessor. [ In reply to ]
Sounds useful, although there are a lot of C-like things
in there (#include, braces, etc.) that would look somewhat
out of place in a Python program.

A truly Pythonic macro processor would use indentation for
delimiting macro definitions, and would use the existing
module system for managing namespaces, etc.

Also, what about line numbers? As far as I know, Python
has no equivalent of a #line directive, so keeping error
messages in sync with the source file is going to be
rather difficult.

--
Greg Ewing, Computer Science Dept, | The address below is not spam-
University of Canterbury, | protected, so as not to waste
Christchurch, New Zealand | the time of Guido van Rossum.
greg@cosc.canterbury.ac.nz
Macro preprocessor. [ In reply to ]
In article <slrn7r291c.137.kaz@ashi.FootPrints.net>
kaz@ashi.FootPrints.net "Kaz Kylheku" writes:
> Hi folks. I wrote a macro preprocessor that can be used with a language that
> uses indendation for statement or declaration grouping, because it is
> intelligent about dealing with leading whitespace.

Sounds interesting. What license is it released under?

> http://users.footprints.net/~kaz/mpp.html
>
> I haven't done work on it in a long while and I need some motivation to to add
> features to it to make it more useful. (It needs conditionals, numeric
> variables and operations, iteration constructs). It would help, like, to
> have some users. :)

Parrot (a gui builder system I'm developing) needs a preprocessor, I
might give it a go for that.

> [...]
> If
> you don't want the # character to introduce directives, you can change it to
> something else on the fly. (If you change it within a header, the change won't
> automatically be propagated to the host file that included it.)
>
> #code leftbrace [
> #code rightbrace ]
> #code escape %
>
> %def foo [ bar ]

Python uses # for comments, so this would come in useful.

--
Phil Hunt....philh@vision25.demon.co.uk
Macro preprocessor. [ In reply to ]
Kaz Kylheku wrote:
[description of a cute macro processor skipped]

> Okay, enough. :) I'll probably get hate mail now about being off topic.

No hate mail, but some thoughts:

I can think of macro processors being useful for verbose, clumsy
languages. For sure it makes sense to use them for ASM, C(++),
and sendmail, for instance.

But why for Python?
Python is short, concise, readable, has a cute class system
and introspection.
Would any of the above be improved by introducing macroes, new
syntax, namespaces, and would this cost be justified?

There are a few cases where I admit that I generate scriptlets
on-the-fly using the formatting operator. I can also imagine
that macroes could be used to make debugging easier than with
conditionals in rare cases. Maybe I would like to splice
parameterized function bodies into a caller, maybe there is
a cleaner compromise somewhere between textual replacement
and the bytecodehacks approach.

But: Plain textual replacement is covered good enough by
the % operator, if this is needed. Macroes on the token
level are fine for languages with limited namespaces, no
inheritance features and so on. Python has these to a very
high extent, so what would macroes help what cannot be done
by classes and inheritance, which justifies another level
of indirection, new syntax, and all of that?

If I would introduce macroes into Python at all, then
probably on the parser level, in order to replace nodes
by equivalent constructs which are faster or would be
otherwise less readable.

Conclusion:
I consider it a virtue of a language if it is powerful
enough to go without macroes. Macroes are needed for
languages which have a lack of expressiveness, except for
Scheme, where macroes are part of the language with the goal
to use them to define language constructs. If introducing
macroes for Python at all, then they should use the existing
Python structures and namespaces, macro features should be
bound to syntactic language elements and parts of expressions,
should not introduce another level of indirection but work
at runtime. And I'd wish to see an example where a macrofied
source file is more readable than a plain one.

IMHO, this all does not apply for Python, and it the last
language where I would want to use macroes.

This is no complaint against your implementation which I think
is very useful for languages which really need it.

please-keep-the-good-work - ly y'rs - chris

--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
and we keep the good language :-)
Macro preprocessor. [ In reply to ]
Christian Tismer <tismer@appliedbiometrics.com> writes:

> Kaz Kylheku wrote:
> [description of a cute macro processor skipped]
>
> > Okay, enough. :) I'll probably get hate mail now about being off topic.
>
> No hate mail, but some thoughts:
[...]

> There are a few cases where I admit that I generate scriptlets
> on-the-fly using the formatting operator.

Now you've piqued my interest... How do you do this, and in which
cases (for what purposes)? Do you create strings that you execute, or
what? (Sounds like a cool thing to do ;)

--

Magnus Making no sound / Yet smouldering with passion
Lie The firefly is still sadder / Than the moaning insect
Hetland : Minamoto Shigeyuki
Macro preprocessor. [ In reply to ]
"Magnus L. Hetland" wrote:
>
> Christian Tismer <tismer@appliedbiometrics.com> writes:
>
> > Kaz Kylheku wrote:
> > [description of a cute macro processor skipped]
> >
> > > Okay, enough. :) I'll probably get hate mail now about being off topic.
> >
> > No hate mail, but some thoughts:
> [...]
>
> > There are a few cases where I admit that I generate scriptlets
> > on-the-fly using the formatting operator.
>
> Now you've piqued my interest... How do you do this, and in which
> cases (for what purposes)? Do you create strings that you execute, or
> what? (Sounds like a cool thing to do ;)

I wrote a helper class which fakes dynamic classes which
can be pickled and stored in database fields.
This was used for a glass fiber network simulation project, where
networks of nodes with lots of properties and direct or symbolic
references to other nodes have to be stored, edited and updated.

The first version of this used exec and formatting to generate
classes dynamically and provide attributes and methods.

Later I could replace almost everything with calls to the "new"
module, and dynnamic assignment of functions as methods.

This is a case where classes and instances are not so sharply
distinguished, it falls into the class of problems where
metaclasses are handy. One simulation defines a special
configuration, gathers data and is saved as an object in
the database, this makes it like an instance. But other
simulations can refer to it, use it as a building block
and inherit from it, which makes it look like a class.

This is a proprietary project, so I can show just some
excerpts from an outdated version of the source files.
System design has been completely redefined, so this is
history.

Here is a dynamic module generator:

----------------------------------------------------------------
#######################################################

# provide a virtual module which has no true file

DYN_MODULE_NAME = "dynamic node classes"

def __virtual_setup():
import new, sys
vm = new.module(DYN_MODULE_NAME)
sys.modules[DYN_MODULE_NAME] = vm

__virtual_setup()
del __virtual_setup

DYN_MODULE = __import__(DYN_MODULE_NAME)

# an alternative could be to replace the standard __import__
# function with one that can behave as expected.
# Anyway, this works.

class NodeGenerator: # Generates *no* code to declare a new class
on-the-fly
def __init__(self, base, tag):
import new
self.nodeLabel = cname = base + "_" + tag # name of the class to
generate
self.classDef = cl = new.classobj(cname, (globals()[base],), {})
def __init__(self, id):
self.__class__.__bases__[0].__init__(self,id)
# which has a slight semantical difference. Guess about it!
cl.__init__ = __init__ # function becomes an unbound method
cl.name = tag

# Initialize empty parameter and port defs
cl.parameterDefs = {}
cl.portDefs = {}

# provide a virtual module name
cl.__module__ = DYN_MODULE_NAME

def address (self, addr):
# Record the persistent address that the class has
self.classDef.address = addr


def defParameter (self, name, type, default, unit, range, desc):
# Record a parameter definition
param = {}
param[ "type" ] = type
param[ "default" ] = default
param[ "unit" ] = unit
param[ "range" ] = range
param[ "desc" ] = desc

self.classDef.parameterDefs[ name ] = param


def defPort (self, name, type, direction, cardinality, desc):
# Record a port definition
port = {}
port[ "type" ] = type
port[ "direction" ] = direction
port[ "cardinality" ] = cardinality
port[ "desc" ] = desc

self.classDef.portDefs[ name ] = port


def declareNode(self):
# just return the ready thing
return self.classDef

#######################################################
----------------------------------------------------------------

and here a factory for abstract nodes:

----------------------------------------------------------------

class AbstractNodeFactory:
def __init__(self):
self.knownNodes = {} # The known types of AbstractNodes keyed by
addresses

def getNode(self, address):
# Is the desired node known?
if address not in self.knownNodes.keys():
# No => generate on-the-fly
generator = NodeGenerator( "AbstractNode", "SomeShortLabel_"
+ address )

# Some dummy stuff
generator.address( address )

generator.defParameter( "Weight", "float", "42.0", "lbs",
"41.9-42.1", "The weight of the brain" )

generator.defPort( "Eye", "optical", "input", "2", "Optical
input" )
generator.defPort( "Ear", "acustical", "input", "2",
"Acustical input" )
generator.defPort( "Mouth", "acustical", "output", "1",
"Acustical output" )

# Declare the class and get to know it
generator.declareNode()
self.knownNodes[ address ] = generator.classDef # was
generatedClass
# store in the dynamic module as well
node = self.knownNodes[ address ]
setattr(DYN_MODULE, node.__name__, node)
# keep a reference to the factory instance as well
node.Factory = self

# Return the class object
return self.knownNodes[ address ]

# pickling support.
# we don't store the generated classes, but the parameters
# needed to restore them.
# therefore, the __getstate__/__setstate__ method
# avoid to make use of the __dict__
def __getstate__(self):
return self.knownNodes.keys()

def __setstate__(self, state):
self.__init__()
map(self.getNode, state)

class persistency:
"""wrap a node instance suitable for pickling"""
def __init__(self, node_inst):
factories = {}
instances = {} # maybe we must use id() in the future
todo = [node_inst]
while todo:
lookat = todo[0]
del todo[0]
for port in lookat.ports.values():
for conn in port.values():
inst = conn[0]
if not instances.has_key(inst):
todo.append(inst)
instances[inst] = 1
factories[inst.Factory] = 1
self.factories = factories.keys()
self.instances = instances.keys()
self.saveset = (self.factories, node_inst)

def __getstate__(self):
return self.saveset

def __setstate__(self, state):
factories, instance = state
self.__init__(instance)

#######################################################


--
Christian Tismer :^) <mailto:tismer@appliedbiometrics.com>
Applied Biometrics GmbH : Have a break! Take a ride on Python's
Kaiserin-Augusta-Allee 101 : *Starship* http://starship.python.net
10553 Berlin : PGP key -> http://wwwkeys.pgp.net
PGP Fingerprint E182 71C7 1A9D 66E9 9D15 D3CC D4D7 93E2 1FAE F6DF
we're tired of banana software - shipped green, ripens at home