Mailing List Archive

Tokenizer problems
Hi,

is someone currently working on the code in CVS? The way it is now, it
doesn't seem to work.

Specifically, the following input text:

''Test''

generates this output:

<p><font color="#FF0000"><b>'</b></font><font
color="#FFFF00"><b></b></font><font color="#FF0000"><b>'</b></font><font
color="#FFFF00"><b></b></font>'Test''

It seems that the Tokenizer thinks each apostrophe is an individual
token, but it should consider '' as a single token.

Could someone fix this please?
Thanks,
Timwi