Mailing List Archive

> On Nov 12, 2021, at 08:54, Neil Bowers <neilb@neilb.org> wrote:
>
> Markdown has long since won the battle of simple text-based documentation formats. People, not just developers, are used to writing it in lots of different places. Odds are that developers trying out Perl, coming from other language experience, will be familiar with markdown, and pod will just seem weird.
>
> I regularly find myself wanting to write markdown instead of pod, particularly when writing modules. Something like:
>
> =format markdown
>
> # NAME
>
> ...
>
> ## Functions
>
> ...
>
> =cut

This already works, right? What kind of support do you envision?

Markdown targets HTML, not text. I’ve had difficulty before in finding Markdown-to-text renderers; that said, I guess folks who want to read Markdown as text can probably just read the MD source code without too much difficulty.

Native RST support might also be nice if the desire is to *replace* POD.

Generally I find folks inadvertently misuse POD by thinking indentation is structural (as it is in RST and Markdown) and by ignoring the need for extra newlines to delimit sections. I do agree that, for neophytes, POD is a bit of a shin-bumper. I’m not sure how MD would be a replacement for it, though; that seems more of a job for RST.

-F

On Fri, 12 Nov 2021 13:54:02 +0000
Neil Bowers <neilb@neilb.org> wrote:

> Markdown has long since won the battle of simple text-based
> documentation formats. People, not just developers, are used to
> writing it in lots of different places. Odds are that developers
> trying out Perl, coming from other language experience, will be
> familiar with markdown, and pod will just seem weird.

Even when I'm writing POD I often end up doing `this`.. ohwait I meant
C<this>.

> There are some things that pod provides that markdown doesn’t, and
> there’s the whole table issue. But on the very rare occasion I’ve
> wanted a table in documentation I’ve used a manually formatted ASCII
> table. And most modules have fairly simple documentation, in terms of
> the pod syntax currently used.

Tables++

(really I want tables)

> The biggest problem I’m aware of is that this would require
> coordination of changes in multiple places, and I don’t even know
> what all of those places are.

> And there are different markdown formats, so we’d have to pick one.

Yes, that's always an issue.

On a somewhat-tangential note, I lately have been maintaining a tool
designed to be "a better man(1)", able to view POD, Markdown, nroff
etc... with a real native formatter that understands terminal
bold/underline/italics, colour, etc...

https://twitter.com/cpan_pevans/status/1405938689050656774

https://metacpan.org/dist/App-sdview

As well as display, it also has some code to output to various formats,
so it can act as a decent cross-converter between these formats. If you
are looking for more inspiration on how to handle these things, that
might be another good place to look.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Fri, 12 Nov 2021 at 22:04, Felipe Gasper <felipe@felipegasper.com> wrote:

> > On Nov 12, 2021, at 08:54, Neil Bowers <neilb@neilb.org> wrote:
> >
> > Markdown has long since won the battle of simple text-based
> documentation formats. People, not just developers, are used to writing it
> in lots of different places. Odds are that developers trying out Perl,
> coming from other language experience, will be familiar with markdown, and
> pod will just seem weird.
> >
> > I regularly find myself wanting to write markdown instead of pod,
> particularly when writing modules. Something like:
> >
> > =format markdown
> >
> > # NAME
> >
> > ...
> >
> > ## Functions
> >
> > ...
> >
> > =cut
>
> This already works, right? What kind of support do you envision?
>

Indeed, the `=for` syntax is already there - some people (maybe just me)
embed Markdown in Perl files already - so this feels more like a
toolchain+MetaCPAN discussion than a core Perl feature?

One issue with the pre-RFC here is that it preserves the non-semantic
convention of `# NAME` (following `=head1 NAME` from pod) as above...
that's not great, everyone else writing markdown gets to put their module
name right after the `# ` so Perl markdown is still going to look a bit
primitive if it's `# NAME\nSome::Module`. "Not great" because this ends up
with the important detail - the name - indistinguishable from the rest of
the meändering text around it.

Put another way: semantic markup!

If we actually had specific `=module Some::Module` markup (or `=class
Some::Class\n\n=method some_method`, to go with our shiny new ~Object::Pad~
core OO system), we'd gain something that's machine-readable... and not
subject to the complexity of a thousand different minor variations. I can't
even reliably link to a method/function name in another module, because
some people like to use the `=head2 method($example, $parameters)` form.
Justifiably so - you get to see at a glance what the parameter structure is
just by skimming the index. Even if you get the L<> syntax right for that
version, it's fragile.

That would, of course, be an entirely different pre-RFC.

* Neil Bowers <neilb@neilb.org> [2021-11-12 13:54:02 +0000]:

> Markdown has long since won the battle of simple text-based documentation formats. People, not just developers, are used to writing it in lots of different places. Odds are that developers trying out Perl, coming from other language experience, will be familiar with markdown, and pod will just seem weird.

Peform any reader loses interest in my email, I want to point out
an amazing example of a document/rendering system written in Perl:

https://metacpan.org/dist/LaTeXML/view/bin/latexml
https://dlmf.nist.gov/LaTeXML/
https://dlmf.nist.gov/
In the process of developing the Digital Library
of Mathematical Functions, we needed a means of
transforming the LaTeX sources of our material into
XML which would be used for further manipulations,
rearrangements and construction of the web site.
In particular, a true ?Digital Library? should
focus on the semantics of the material, and so we
should convert the mathematical material into both
content and presentation MathML. At the time, we
found no software suitable to our needs, so we
began development of LaTeXML in-house.

It should be no surprise that I literally never think about what people
not familiar with Perl think and I don't care about them unless they want
to make an effort to *learn* Perl/perl. I recommend we all eschew the temptation
to do so. We do so at our own per[i]l. I imply this a lot, but here it is stated
very clearly.

We have almost 40 years of POD and the best text processing
environment in the universe. We can do better than jump over
to something that has yet to prove any longevity or stability.

>
> I regularly find myself wanting to write markdown instead of pod, particularly when writing modules. Something like:

That's you. I like our dog food more than I hate it.

What I find myself wanting to do more intuitively in POD are:

* lists, sublists
* tables (mentioned below)
* inlined <code>...</code>
* verbatim/<pre/> sections
* better internal linking
* alt. to =headN for sections (maybe adopting MD's '#' behind
the '=' sentinal, (e.g., "=#, =###, etc")

So rather than propose we embed a formatting syntax in a formatting
syntax, I recommend the approach to extend the POD "standard" to
allow (in a "safe" way) the above capabilies.

Alternative: better tooling, e.g., so you can maintain your POD/MD
chimera, and run it through POD pre-processor (e.g., exposed via
something like a Dist::Zill plugin since that's equally gross).

Ultimately, POD's limitations are also its strength. If not done
properly this could open a pandora's box of all kinds of abominations
that pass for "POD".

>
> =format markdown
>

Not a terrible thought, but what happens when we want,

=format SQL

=format asciidoc

etc?

> # NAME
>
> ...
>
> ## Functions
>
> ...
>
> =cut
>
> There are some things that pod provides that markdown doesn?t, and there?s the whole table issue. But on the very rare occasion I?ve wanted a table in documentation I?ve used a manually formatted ASCII table. And most modules have fairly simple documentation, in terms of the pod syntax currently used.
>
> I think this would provide one less huh?/wtf? barrier to people trying Perl, and would let people use the broad range of markdown tools out there.
>
> The biggest problem I?m aware of is that this would require coordination of changes in multiple places, and I don?t even know what all of those places are. And there are different markdown formats, so we?d have to pick one.

Well, that is a big problem. I personally would prefer to see a
limited extension to POD be created that would make it easier
to do things like, idk - maintain RFC documentation in something
that POD or PODng. Maybe even a Perl-centric solution that resulted
in the tooling surrounding something like docbook (e.g.,).

With this in mind, a missing "arm" of the documentation effort in
Perl has to do with tooling. I've expressed interest in the past of
helping out with this aspect (maintence, refactor, extending) - so
the idea of improving the joy, accuracy, and tooling around *our*
documentation DSL, the "docs" effort would encompass:

* actual documentation content
* tools writers/maintainers *love* to use
* managing POD improvements via tooling

Finally, I'd love to use POD to do thing like write white papers
or RFCs, actual technical document. MD can't do those things well
either. Both activities require actually enjoying the act of writing
the text, which I get out of 2 things: plain text and latex. I'd
like to add POD to this mix - injecting MD into POD will not accomplish
that for me, not even close. In otherwords, anything we accept as
a defacto alternative to POD will not give us the same joy most of
us get in writing actual Perl (or provide the immediate gratification);
but I think with some thought and proper tooling, we get get there
with POD itself.

Cheers,
Brett

>
> Neil

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

On Sat, 13 Nov 2021 at 00:20, Oodler 577 via perl5-porters <
perl5-porters@perl.org> wrote:

> * Neil Bowers <neilb@neilb.org> [2021-11-12 13:54:02 +0000]:
> It should be no surprise that I literally never think about what people
> not familiar with Perl think and I don't care about them unless they want
> to make an effort to *learn* Perl/perl. I recommend we all eschew the
> temptation
> to do so. We do so at our own per[i]l. I imply this a lot, but here it is
> stated
> very clearly.
>

That's... remarkably insular. Other ecosystems are great sources of
creative inspiration, with a much broader range of ideas we could be
applying. Not too clear what temptation exactly you're saying to eschew
here - caring about what people think? learning perl??

I'm surprised and disappointed to see a comment like this on a language's
core development discussion thread, and I really hope it's not a widespread
view.

Not a terrible thought, but what happens when we want,
>
> =format SQL
>
> =format asciidoc
>

Again, we already have this feature, and it's actively used - for example,
HTML sections are how people add images to documentation:

https://metacpan.org/dist/Tickit-Widget-Layout-Relative/source/lib/Tickit/Widget/Layout/Relative.pm#L52

and has been used for a range of purposes, e.g. embedding other language
examples or as a richer form of the __DATA__ section.

See perldoc perlpod:

A command "=begin *formatname*", some paragraphs, and a command
"=end *formatname*", mean that the text/data in between is meant for
formatters that understand the special format called *formatname*.
...
The command "=for *formatname* *text...*" specifies that the
remainder of just this paragraph (starting right after *formatname*)
is in that special format.

Markdown also includes something similar, it's just much more concise (as
always):

```sql
select * from some_table
```

On Fri, Nov 12, 2021 at 8:54 AM Neil Bowers <neilb@neilb.org> wrote:

> Markdown has long since won the battle of simple text-based documentation
> formats. People, not just developers, are used to writing it in lots of
> different places. Odds are that developers trying out Perl, coming from
> other language experience, will be familiar with markdown, and pod will
> just seem weird.
>
> I regularly find myself wanting to write markdown instead of pod,
> particularly when writing modules. Something like:
>
> =format markdown
>
> # NAME
>
> ...
>
> ## Functions
>
> ...
>
> =cut
>
> There are some things that pod provides that markdown doesn’t, and there’s
> the whole table issue. But on the very rare occasion I’ve wanted a table in
> documentation I’ve used a manually formatted ASCII table. And most modules
> have fairly simple documentation, in terms of the pod syntax currently used.
>
> I think this would provide one less huh?/wtf? barrier to people trying
> Perl, and would let people use the broad range of markdown tools out there.
>
> The biggest problem I’m aware of is that this would require coordination
> of changes in multiple places, and I don’t even know what all of those
> places are. And there are different markdown formats, so we’d have to pick
> one.
>

I would echo other comments in saying that the ability to notate this is
already present, and add that it should not be the job of Pod parsers to
parse markdown - the Pod spec is intentionally simple. So in my view the
bigger question is: what toolchain do you propose should handle displaying
Pod and markdown within the same file? I don't have a good answer for this.

-Dan

* Dan Book <grinnz@gmail.com> [2021-11-13 10:54:22 -0500]:

> On Fri, Nov 12, 2021 at 8:54 AM Neil Bowers <neilb@neilb.org> wrote:
>
> > Markdown has long since won the battle of simple text-based documentation
> > formats. People, not just developers, are used to writing it in lots of
> > different places. Odds are that developers trying out Perl, coming from
> > other language experience, will be familiar with markdown, and pod will
> > just seem weird.
> >
> > I regularly find myself wanting to write markdown instead of pod,
> > particularly when writing modules. Something like:
> >
> > =format markdown
> >
> > # NAME
> >
> > ...
> >
> > ## Functions
> >
> > ...
> >
> > =cut
> >
> > There are some things that pod provides that markdown doesn?t, and there?s
> > the whole table issue. But on the very rare occasion I?ve wanted a table in
> > documentation I?ve used a manually formatted ASCII table. And most modules
> > have fairly simple documentation, in terms of the pod syntax currently used.
> >
> > I think this would provide one less huh?/wtf? barrier to people trying
> > Perl, and would let people use the broad range of markdown tools out there.
> >
> > The biggest problem I?m aware of is that this would require coordination
> > of changes in multiple places, and I don?t even know what all of those
> > places are. And there are different markdown formats, so we?d have to pick
> > one.
> >
>
> I would echo other comments in saying that the ability to notate this is
> already present, and add that it should not be the job of Pod parsers to
> parse markdown - the Pod spec is intentionally simple. So in my view the
> bigger question is: what toolchain do you propose should handle displaying
> Pod and markdown within the same file? I don't have a good answer for this.

I do not have a good answer either, other than it should be possible to do
with POD what we do with Perl, make some incremental improvements that get
to the heart of what is wanted from markdown. It might be as simple as just
extending POD, such that:

# NAME -> =head1 NAME

Though, that would necessarily need to be differentiated from comments. The
idea of semantic based tags is a good one and would allow POD to "get out of
the way". Add support for MD (or mediawiki) style tables and backticks, then
it would meet a lot of my needs (because I don't know what the others' wants
are). E.g.,

=name -> =head1 NAME
=synopsis -> =head1 SYNOPSIS
=description -> =head1 DESCRIPTION
=subroutines ..
=options ...
=author ...
=copyright ...

Other:

1. allow "=head1" auto capitalize via perldoc
2. all other "=headN" should capitalize based on convention

Code/verbatim:

1. `inline code` -> C<inline code>
2. generalize "=over N, =back" to be a block quote
2. add, "=code [language], =back" + plus some plug-in for rendering "language"
if supported (could be highlighting, could be rendering)

Regarding, "getting POD out of the way", the biggest pain I find is the need
to type a string of capitalized letters to stay consistent with the style
(e.g., "=head1 NAME").

Just some thoughts. I am interested in helping with any tooling surrounding
documentation, but I have no idea where to start. Any help or information
in that regard would be helpful to me. Also, it's hard for me to jump in
when I don't see an official effort or something sort of sanctioned working
group that gives me the confidence that any of my (or anyone's time) spent
on this would actually benefit Perl/perl (eventually).

Cheers,
Brett


>
> -Dan

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

On Fri, 12 Nov 2021 13:54:02 +0000
Neil Bowers <neilb@neilb.org> wrote:

> Markdown has long since won the battle of simple text-based documentation formats. People, not just developers, are used to writing it in lots of different places. Odds are that developers trying out Perl, coming from other language experience, will be familiar with markdown, and pod will just seem weird.
>
> I regularly find myself wanting to write markdown instead of pod, particularly when writing modules. Something like:
>
> =format markdown
>
> # NAME
>
> ...
>
> ## Functions
>
> ...
>
> =cut
>
> There are some things that pod provides that markdown doesn?t, and there?s the whole table issue. But on the very rare occasion I?ve wanted a table in documentation I?ve used a manually formatted ASCII table. And most modules have fairly simple documentation, in terms of the pod syntax currently used.
>
> I think this would provide one less huh?/wtf? barrier to people trying Perl, and would let people use the broad range of markdown tools out there.
>
> The biggest problem I?m aware of is that this would require coordination of changes in multiple places, and I don?t even know what all of those places are. And there are different markdown formats, so we?d have to pick one.
>
> Neil

As the others have already said, =for already exists. However, it's
mainly used for small, isolated fragments of the documentation, *not* to
write everything using a different markup language.

In my opinion, allowing to write a whole documentation in Markdown would
be too drastic of a change and I don't support it. Also, keep in mind
that Markdown isn't really standarized and there are several competing
syntaxes for e.g. tables. If we allowed "=for markdown", we would have
to write a complete specification for our Markdown dialect.

I think we should fix POD instead. There are two big problems with it:
the lack of images and the lack of tables. Both are fixable. I will
probably submit pre-RFCs soon.

On Mon, 15 Nov 2021 15:45:15 +0100
Tomasz Konojacki <me@xenu.pl> wrote:

> I think we should fix POD instead. There are two big problems with it:
> the lack of images and the lack of tables. Both are fixable. I will
> probably submit pre-RFCs soon.

+1

Please do :) Tables and images would be a great addition IMHO.

--
Paul "LeoNerd" Evans

leonerd@leonerd.org.uk | https://metacpan.org/author/PEVANS
http://www.leonerd.org.uk/ | https://www.tindie.com/stores/leonerd/

On Fri, Nov 12, 2021 at 01:54:02PM +0000, Neil Bowers wrote:
On Mon, Nov 15, 2021 at 03:45:15PM +0100, Tomasz Konojacki wrote:
> On Fri, 12 Nov 2021 13:54:02 +0000
> Neil Bowers <neilb@neilb.org> wrote:
>
> > Markdown has long since won the battle of simple text-based documentation formats. People, not just developers, are used to writing it in lots of different places. Odds are that developers trying out Perl, coming from other language experience, will be familiar with markdown, and pod will just seem weird.
> >
> > I regularly find myself wanting to write markdown instead of pod, particularly when writing modules. Something like:
> >
> > =format markdown
> >
> > # NAME
> >
> > ...
> >
> > ## Functions
> >
> > ...
> >
> > =cut

In core we use Pod to generate man pages, text documentation and HTML.
We seem to mention `perldoc` [or should I say C<perldoc>? :-)] more than
the other two.

(If I understand =for correctly) then using exactly the syntax you're
suggesting would mean that perldoc would render module documentation as a
blank page.

Hence this *exact* syntax likely doesn't work, but that doesn't rule the
idea out. Moreso - my rough understanding of (common) markdown is that it
basically offers the same formatted output styles as Pod can already do,
so implementing "markdown" as an equal alternative is *more* about parsing
than outputting. The solution seems to be more about implementing a markdown
parser that generates the same internal format as a Pod parser parses to,
and sharing the same text, troff and HTML backends.


> > There are some things that pod provides that markdown doesn't, and there's the whole table issue. But on the very rare occasion I've wanted a table in documentation I've used a manually formatted ASCII table. And most modules have fairly simple documentation, in terms of the pod syntax currently used.
> >
> > I think this would provide one less huh?/wtf? barrier to people trying Perl, and would let people use the broad range of markdown tools out there.
> >
> > The biggest problem I'm aware of is that this would require coordination of changes in multiple places, and I don't even know what all of those places are. And there are different markdown formats, so we'd have to pick one.

[description of =for that I expanded above]

> In my opinion, allowing to write a whole documentation in Markdown would
> be too drastic of a change and I don't support it. Also, keep in mind
> that Markdown isn't really standarized and there are several competing
> syntaxes for e.g. tables. If we allowed "=for markdown", we would have
> to write a complete specification for our Markdown dialect.

Yes, this is my biggest concern. I'm not sure if there *is* an obvious
Markdown spec to implement. The "obvious" choice is CommonMark:

https://spec.commonmark.org/0.30/

But *that* doesn't have tables.

Implement CommonMark and what do we get that Pod doesn't have?

Images.

Yay. Not really worth it.

I'm not sure if there *is* a third party spec that also has tables, but then
doesn't also have a bunch of cruft we don't want to (or can't) implement.

As best I can tell, all of GitHub, GitLab and Bitbucket extend Markdown
with basically the same table syntax:

https://github.github.com/gfm/#tables-extension-
https://docs.gitlab.com/ee/user/markdown.html#tables
https://confluence.atlassian.com/bitbucketserver/markdown-syntax-guide-776639995.html#Markdownsyntaxguide-Tables

(note that Bitbucket markdown doesn't have a published spec. That page lies
by linking only to CommonMark)


but there doesn't seem to be a common ancestor of the three that has just
what we might need.

Whereas all those three

1) offer things that we likely can't support, such as ~~strikethrough text~~
(useful, but not something that can be done in troff or plaintext?)
2) specify a bunch of domain specific "autolinking" rules that we simply
don't want to offer


And I guess there's also the elephant in the room - even CommonMark is no
good as-is, because *all* markdown permits you to inline HTML, which makes
sense for it. We need to specify "Markdown, but without inline HTML", else
how do we target perldoc?

Again, this doesn't rule the idea out, but it looks likely that to be useful
we want images, tables and no HTML escape hatch, meaning that we'd need to
create "Perl flavoured markdown", rather than being able to document that
we implement standard X.

> I think we should fix POD instead. There are two big problems with it:
> the lack of images and the lack of tables. Both are fixable. I will
> probably submit pre-RFCs soon.

I'm of an open mind on "instead". But I'd welcome Pod-like solutions for
images and tables. If I understand it correctly, markdown's image syntax
restricts you to declaring

* image URL
* image title
* image ALT text

and the rest is under its control, not yours.

Nicholas Clark

On Mon, Nov 15, 2021 at 2:59 PM Nicholas Clark <nick@ccl4.org> wrote:

> On Fri, Nov 12, 2021 at 01:54:02PM +0000, Neil Bowers wrote:
> On Mon, Nov 15, 2021 at 03:45:15PM +0100, Tomasz Konojacki wrote:
> > On Fri, 12 Nov 2021 13:54:02 +0000
> > Neil Bowers <neilb@neilb.org> wrote:
> >
> > > Markdown has long since won the battle of simple text-based
> documentation formats. People, not just developers, are used to writing it
> in lots of different places. Odds are that developers trying out Perl,
> coming from other language experience, will be familiar with markdown, and
> pod will just seem weird.
> > >
> > > I regularly find myself wanting to write markdown instead of pod,
> particularly when writing modules. Something like:
> > >
> > > =format markdown
> > >
> > > # NAME
> > >
> > > ...
> > >
> > > ## Functions
> > >
> > > ...
> > >
> > > =cut
>
> In core we use Pod to generate man pages, text documentation and HTML.
> We seem to mention `perldoc` [or should I say C<perldoc>? :-)] more than
> the other two.
>
> (If I understand =for correctly) then using exactly the syntax you're
> suggesting would mean that perldoc would render module documentation as a
> blank page.
>

Currently yes, but this doesn't rule out that Pod::Perldoc could be
extended to render markdown sections, nor that metacpan could (the most
commonly viewed HTML rendering). Either way this isn't the job of the pod
*specification* or even Pod::Simple, but of specific pod renderers, unless
we want to extend the specification to support some custom markdown
variant, which is IMO not a good idea.

-Dan

On Mon, Nov 15, 2021 at 3:20 PM Dan Book <grinnz@gmail.com> wrote:

> On Mon, Nov 15, 2021 at 2:59 PM Nicholas Clark <nick@ccl4.org> wrote:
>
>> On Fri, Nov 12, 2021 at 01:54:02PM +0000, Neil Bowers wrote:
>> On Mon, Nov 15, 2021 at 03:45:15PM +0100, Tomasz Konojacki wrote:
>> > On Fri, 12 Nov 2021 13:54:02 +0000
>> > Neil Bowers <neilb@neilb.org> wrote:
>> >
>> > > Markdown has long since won the battle of simple text-based
>> documentation formats. People, not just developers, are used to writing it
>> in lots of different places. Odds are that developers trying out Perl,
>> coming from other language experience, will be familiar with markdown, and
>> pod will just seem weird.
>> > >
>> > > I regularly find myself wanting to write markdown instead of pod,
>> particularly when writing modules. Something like:
>> > >
>> > > =format markdown
>> > >
>> > > # NAME
>> > >
>> > > ...
>> > >
>> > > ## Functions
>> > >
>> > > ...
>> > >
>> > > =cut
>>
>> In core we use Pod to generate man pages, text documentation and HTML.
>> We seem to mention `perldoc` [or should I say C<perldoc>? :-)] more than
>> the other two.
>>
>> (If I understand =for correctly) then using exactly the syntax you're
>> suggesting would mean that perldoc would render module documentation as a
>> blank page.
>>
>
> Currently yes, but this doesn't rule out that Pod::Perldoc could be
> extended to render markdown sections, nor that metacpan could (the most
> commonly viewed HTML rendering). Either way this isn't the job of the pod
> *specification* or even Pod::Simple, but of specific pod renderers, unless
> we want to extend the specification to support some custom markdown
> variant, which is IMO not a good idea.
>

Forgot to mention: this has precedent of a sort as metacpan currently
renders (stripped) =for html sections, while non-HTML renderers of course
ignore them.

-Dan

On Mon, Nov 15, 2021 at 4:43 PM Paul "LeoNerd" Evans <leonerd@leonerd.org.uk>
wrote:

> On Mon, 15 Nov 2021 15:45:15 +0100
> Tomasz Konojacki <me@xenu.pl> wrote:
>
> > I think we should fix POD instead. There are two big problems with it:
> > the lack of images and the lack of tables. Both are fixable. I will
> > probably submit pre-RFCs soon.
>
> Please do :) Tables and images would be a great addition IMHO.
>

I would like to point out that you're not the first one to try get this
done, there is much prior art.
See for instance https://github.com/Perl/perl5/issues/18169

It would be great if POD gets this feature though! (or tables)

Also I would like to take the opportunity to mention that Markdown is more
or less a Perl success story as the first implementation was Markdown.pl,
written by John Gruber.
And the argument of '40 years of POD history' -- that is great & true, but
there are a _lot_ more people today who write Markdown than there have ever
been writing POD.
Markdown _is_ kind of sucky in that it's not standardized and CommonMark
misses tables though!!
--
Michiel

2021-11-15 23:45 Tomasz Konojacki <me@xenu.pl> wrote:

>
> I think we should fix POD instead. There are two big problems with it:
> the lack of images and the lack of tables. Both are fixable. I will
> probably submit pre-RFCs soon.
>
>
Currently I write images and tables in POD using "=begin html" and "=end
html" syntax.

If these are implemented, I'm glad.

* Yuki Kimoto <kimoto.yuki@gmail.com> [2021-11-16 09:44:36 +0900]:

> 2021-11-15 23:45 Tomasz Konojacki <me@xenu.pl> wrote:
>
> >
> > I think we should fix POD instead. There are two big problems with it:
> > the lack of images and the lack of tables. Both are fixable. I will
> > probably submit pre-RFCs soon.
> >
> >
> Currently I write images and tables in POD using "=begin html" and "=end
> html" syntax.
>
> If these are implemented, I'm glad.

I don't see how we're going to be able to do tables in a tabular
way and not short the value POD has as a readable format in its
own right.

=html
<table>
<tr>
<th>Column 1</th><th>Column 2</th><th>Column 3</th>
</tr>
<tr>
<td>...... 1</td><td>...... 2</td><td>...... 3</td>
</tr>
<tr>
<td>...... 1</td><td>...... 2</td><td>...... 3</td>
</tr>
<tr>
<td>...... 1</td><td>...... 2</td><td>...... 3</td>
</tr>
</table>
=end

To this end, this problem will not be solved by syntax because
the problem "space" is literaly 2D. What I mean there is that
POD has the constraint that is not solvable by "markup" - the
plain text view (e.g., source code) view of the POD must make
sense and look good. Markup/down is meant as an intermediate
format to get to a rendered state. For us the intermediate state
is also a final state.

The solution will be a combination of:

* tool(s)

- a "podtidy" (several exist on CPAN)
- a "parser" renderers may use to take an ASCII table into
an abstract form that can then be used to output whatever
(e.g., HTML, groff, PDF, etc)

* syntax

- the syntax must also look like a table (2D)
- the official "syntax" being some form of an ASCII table
(e.g., see Text::ASCIITable)
- drop table into a "=table"/"=end" block

* workflow adoption (same as w/perltidy)

The following seems to be a reasonable and familiar workflow:

1) in POD, create table by hand using best effort "ASCIITable"
format (focus on content and partions, but not alignment)

2) run "podtidy" (e.g., via vim using the '%!podtidy` format,
which will take a valid NxM table and line things up

3) maintaining the table consists of modifying the table in
question, then re-running "podtidy"

Challenges:

* yes there are some, but more familiarity with the tools needs
to be acheived

* using 2 tools (perltidy, podtidy) seems cumbersome; maybe perltidy
needs to adopt this role once it's vetted

Other things this could support:

* =verbatim
* =blockquote

Thoughts?

Brett

Note: Tables are not the same problem as images or inline monospaced
code.

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

Hi Neil!

On Fri, 12 Nov 2021 13:54:02 +0000
Neil Bowers <neilb@neilb.org> wrote:

> Markdown has long since won the battle of simple text-based documentation
> formats. People, not just developers, are used to writing it in lots of
> different places. Odds are that developers trying out Perl, coming from other
> language experience, will be familiar with markdown, and pod will just seem
> weird.
>

Quoting my page at
https://www.shlomifish.org/philosophy/computers/web/choice-of-docs-formats/ :

'''
Why the Markdown Dialects Should be Avoided as much as Possible [ #markdowns ]

There are too many Markdown dialects (e.g: GitHub's, reddit's, Stack
Exchange's) each one with its own army and navy (= fragmentation and
incompatibilities). Moreover, they can only be converted to XHTML. '''

So I am opposed for making one of the markdowns an integral part of POD. "Only
dead fish go with the flow".

Regards,

Shlomi

On 11/18/21 3:23 AM, Shlomi Fish wrote:
> Quoting my page at
> https://www.shlomifish.org/philosophy/computers/web/choice-of-docs-formats/ :
> Why the Markdown Dialects Should be Avoided as much as Possible [ #markdowns ]...
>
> So I am opposed for making one of the markdowns an integral part of POD. "Only
> dead fish go with the flow".

I agree it is better to partly solve known problems rather than create
new ones.

POD's strength lies in its being a self-contained all-text format,
easily viewable, editable, and maintainable with git or other version
control systems.

In contrast, inserting in-line hrefs to external resources is
problematic, not only because they cannot easily be viewed off-line, but
because there is no guarantee of synchronization between versions of the
documentation and the separate resource.  External binary resources in
particular, even if meticulously mirrored, cannot easily be checked to
see what changed over time in a version-control system.

1. Images:

The problem of mixing images and diagrams with documentation were mostly
solved long ago, albeit in a somewhat different age, by NetPBM and by
troff-accompanying tools like pic or the modern pikchr.

NetPBM, http://netpbm.sourceforge.net/ , includes the definition of
image formats pbm, pgm, and ppm for binary, grayscale, and color images
stored in a plain text format (for these purposes we dismiss the 'raw'
binary varieties). Such images, though loquacious in their storage,
would be embedded directly into a POD file (perhaps as addenda),
satisfying the requirement to be self-contained, human-readable, and
easily trackable through git et al.

Pikchr (https://pikchr.org/home/doc/trunk/homepage.md) can produce
charts (such as bit-fields or Perl Reference pointers) and diagrams
(e.g.,  state, flow, or network communication protocol) in a pleasing
vector format easily comprehended on screen or paper, and whose
originals are maintainable in a text format; for example as in
https://metacpan.org/dist/perl/view/pod/perlreftut.pod#Solution the
"%table" diagram, in Pikchr, would be written:

move down 5mm
line 0mm "%table" above
B1: box "Germany" height 7mm width 2cm
box width 5mm height 7mm with .w at last box.e
dot at last box.center
arrow right
box "Frankfurt" fit with .w at last arrow.e
box "Berlin" fit with .w at last box.e
B2: box "Finland" height 7mm width 2cm with .n at B1.s
box width 5mm height 7mm with .w at last box.e
dot at last box.center
arrow right
box "Helsinki" fit with .w at last arrow.e
B3: box "USA" height 7mm width 2cm with .n at B2.s
box width 5mm height 7mm with .w at last box.e
dot at last box.center
arrow right
box "Chicago" fit with .w at last arrow.e
box "Washington" fit with .w at last box.e
box "New York" fit with .w at last box.e

You may see the end result, as SVG or PNG, by pasting the above at
https://pikchr.org/home/pikchrshow

Regrettably that does not produce ASCII style output, but Emacs at least
can display SVG inline while viewing or editing text. Perhaps other
terminal pagers (like Less) comprehend inline svg? Vector formats surely
would be preferable than bitmaps for most things Perl documentation
would encompass.  In any case, the PNG is easily convertible, even
scriptable as part of the cpan upload process, with the pngtopbm program
of NetPBM and then embedded into the POD alongside the text definition
as above.

2. Tables. 

Emacs Org-mode has a built-in ASCII-mode table editor, producing very
much in the vein of Markdown; see
https://orgmode.org/manual/Built_002din-Table-Editor.html the following
page to which,
https://orgmode.org/manual/Column-Width-and-Alignment.html explains how
columns may have width and left/right/center alignment defined.  While
that is somewhat Emacs-specific, it is highly convenient to have the
table realigned each time you hit TAB while editing the table right in
the editor: What you see is truly what you get.

Perhaps other editors have similar features which POD editors could
leverage, if the new specification resembles something already existing. 

Conclusion.

It would be regrettable to define yet another syntax for images,
diagrams, and tables when long-standing, well-used solutions already
exist, especially when they are already compatible with Perl's and
CPAN's existing workflows.

\\/
William Lindley
Shelbyville, KY

What if we punted on much of this and allow:

    =poddown

    This is basic C<markdown> plus `pod`.

    =cut

I was thinking about this because I saw the Playwright documentation (https://metacpan.org/pod/Playwright) which clearly has some accidental Markdown that's unrendered:

    See https://playwright.dev/versions and drill down into your relevant
    version (run `npm list playwright` ) for what the classes do, and their usage.

We don't promise more than basic markdown support.

    * this
    * would
    * be
    * a
    * list

    =over 4

    =item * this

    =item * would
 
    =item * also

    =item * be

    =item * a

    =item * list

    =back

And we can look at something proper later? You'd get the basic convenience of Markdown but the other features of POD. Older POD parsers would simply default to what we currently have.

No tables until we could figure out something later.

Best,
Ovid
-- 
IT consulting, training, specializing in Perl, databases, and agile development
http://www.allaroundtheworld.fr/. 

Buy my book! - http://bit.ly/beginning_perl

On Mon, Nov 22, 2021, at 3:12 PM, Ovid via perl5-porters wrote:
> What if we punted on much of this and allow:
>
> =poddown

I mostly want to stay out of this, personally — but here I am posting.

There's a lot to do in "extend Pod with Markdown," and I'm not sure it's really practical. Of course, I'm not proposing to do it or forbid it, so this isn't my thing to worry about so much.

But I will say that I think that by far the most tedious thing in Pod is lists, for which I provided myself with a solution in Pod::Elemental::Transformer::List <https://metacpan.org/pod/Pod::Elemental::Transformer::List>, which creates a "=begin :list" region or "=for :list" paragraph, which allows very compact notation for bulleted, definition, or ordered lists that uses Pod sequences instead of Markdown, and so remains very much like all other Pod. Adding support to that to Pod::Simple might be a fair bit simpler than Markdown... but I won't be the one to do it! ???????????

--
rjbs

* Ricardo Signes <perl.p5p@rjbs.manxome.org> [2021-11-27 14:30:13 -0500]:

> On Mon, Nov 22, 2021, at 3:12 PM, Ovid via perl5-porters wrote:
> > What if we punted on much of this and allow:
> >
> > =poddown
>
> I mostly want to stay out of this, personally ? but here I am posting.
>
> There's a lot to do in "extend Pod with Markdown," and I'm not sure it's really practical. Of course, I'm not proposing to do it or forbid it, so this isn't my thing to worry about so much.
>
> But I will say that I think that by far the most tedious thing in Pod is lists, for which I provided myself with a solution in Pod::Elemental::Transformer::List <https://metacpan.org/pod/Pod::Elemental::Transformer::List>, which creates a "=begin :list" region or "=for :list" paragraph, which allows very compact notation for bulleted, definition, or ordered lists that uses Pod sequences instead of Markdown, and so remains very much like all other Pod. Adding support to that to Pod::Simple might be a fair bit simpler than Markdown... but I won't be the one to do it! ?????

What's your workflow for rendering this to POD, is it transformed in the source
after you've written once (e.g., via an external tool or vim plugin), or does it
remain in the source for downstream consumption?

My personal take is that beyond affecting the POD syntax itself, the next most
convenient solutions a tool-based one.

Brett

>
> --
> rjbs

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

On Mon, Nov 29, 2021, at 12:49 PM, Oodler 577 wrote:
> * Ricardo Signes <perl.p5p@rjbs.manxome.org> [2021-11-27 14:30:13 -0500]:
>> But I will say that I think that by far the most tedious thing in Pod is lists, for which I provided myself with a solution in Pod::Elemental::Transformer::List <https://metacpan.org/pod/Pod::Elemental::Transformer::List>, which creates a "=begin :list" region or "=for :list" paragraph, which allows very compact notation for bulleted, definition, or ordered lists that uses Pod sequences instead of Markdown, and so remains very much like all other Pod. Adding support to that to Pod::Simple might be a fair bit simpler than Markdown... but I won't be the one to do it! ?????
>
> What's your workflow for rendering this to POD, is it transformed in the source after you've written once (e.g., via an external tool or vim plugin), or does it remain in the source for downstream consumption?

I use Dist::Zilla, which rewrites the quirky Pod from my git repository's source into standard Pod before building tarballs for the CPAN.

--
rjbs

* Ricardo Signes <perl.p5p@rjbs.manxome.org> [2021-11-29 21:06:07 -0500]:

> On Mon, Nov 29, 2021, at 12:49 PM, Oodler 577 wrote:
> > * Ricardo Signes <perl.p5p@rjbs.manxome.org> [2021-11-27 14:30:13 -0500]:
> >> But I will say that I think that by far the most tedious thing in Pod is lists, for which I provided myself with a solution in Pod::Elemental::Transformer::List <https://metacpan.org/pod/Pod::Elemental::Transformer::List>, which creates a "=begin :list" region or "=for :list" paragraph, which allows very compact notation for bulleted, definition, or ordered lists that uses Pod sequences instead of Markdown, and so remains very much like all other Pod. Adding support to that to Pod::Simple might be a fair bit simpler than Markdown... but I won't be the one to do it! ?????
> >
> > What's your workflow for rendering this to POD, is it transformed in the source after you've written once (e.g., via an external tool or vim plugin), or does it remain in the source for downstream consumption?
>
> I use Dist::Zilla, which rewrites the quirky Pod from my git repository's source into standard Pod before building tarballs for the CPAN.

Thank you. Makes perfect sense using this workflow. It also solves the problem
of "where to keep the editable intermediate format".

FWIW (to all), after doing some more "research" on this issue and what's come
before (even looking at tables in *rof), I saw that the there is a necessary
separation between the content and the formatting - in most "early" cases I
saw this was referred to as "typesetting". We might think of it more as ren-
dering.

So short of doing *anything* with POD to support what's needed, the idea of
approaching this from a tools and workflow angle seems to be the most exped-
iant.

For example, it makes perfect sense to add some shorthand notation that is
meant for author convenience into Dist::Zilla (or similar tool) since it must
live in the precise part of the development workflow for this need. It's also
superior to the idea of having just a utility that can convert some table
syntax into a "neat" ascii table by passing it through an external command
in vim. This necessarily destroys the persistent value of any intermediate
format.

The question then becomes, is Dist::Zilla a good candidate for managing pure
content + hints for tedius POD. This doesn't solve the issue of POD not being
able to handle things like:

* concise lists
* tables
* images

It does, however, provide a spot in the workflow for authors to basically use
any means for managing their intermediate format (say on the git repo) into
a standard representation in the distribution of this module (via CPAN).

What this looks like, extending the "concise list" example, is that, e.g.:

* tables can be defined in some intermediate way, but the resulting distribut-
ed "POD" visible via "perldoc" would be a neat ASCII table contained in a
verbatim section. And this preformatted text would be "safe" in all places
the POD would appear.

I don't know what to say about images, but the complicating factors there are:

1. it's not reasonable to "show" pictures at the commandline nor is it a
good idea to carry images in CPAN distros

2. people might have different ideas of what should be shown in the terminal
versus a canonical way to allow the images to be rendered when looking at
the HTML version of the POD in a web browser (or exported to PDF or whatever).

Images are not something I care much about, but for the 2 cases that do seem
interesting and useful to me (lists and tables), this approach seems to be
a good one.

In summary, my overall point is that the way RJBS is handling concises lists
via Dist::Zilla and the workflow it admits is a good example of where any tool-
based solution of this would need to exist.

Cheers,
Brett

>
> --
> rjbs

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native

On 11/29/21 11:14 PM, Oodler 577 via perl5-porters wrote:
> * tables can be defined in some intermediate way, but the resulting distribut-
> ed "POD" visible via "perldoc" would be a neat ASCII table contained in a
> verbatim section. And this preformatted text would be "safe" in all places
> the POD would appear.

Agree.

In support of the toolchain concept:

Given that POD can now contain Unicode, why not allow or encourage
tables using the "Box Drawing" characters (a derivative of what we used
to call the vt100 and IBM PC Line Drawing sets) as shown at
https://unicode-table.com/en/blocks/box-drawing/

For example, consider this in perlfaq4.pod:

Pictures help... Here's the C<%hash> table:

keys values
+------+------+
| a | 3 |
| x | 7 |
| d | 0 |
| e | 2 |
+------+------+

Why shouldn't this be the nicer-looking:

Pictures help... Here's the C<%hash> table:

keys values
???????????????
? a ? 3 ?
? x ? 7 ?
? d ? 0 |
? e ? 2 ?
???????????????

It is true that my (notional) DECwriter LA-36 won't nicely print that,
but even the Linux console does nowadays.  It is curious that folks were
widely using line-drawing characters in the late 1970s while Perl's
standard POD today still doesn't. Given a toolchain that creates such
tables in POD files, authors wouldn't even have to look up the gyrations
to type all those funny characters. Is there any argument against this,
when (per "perldoc perlpod") POD can officially contain any
Encode::Supported encoding?

The same is true if authors want to use Pikchr or *roff's "pic" or "dia"
or whatever to create diagrams and flowcharts; given the
table/image/diagram source code attached to the module, and a
description of the filters used, that keeps distributions reproducible
and modifiable.

\\/

In helping with a little POD update recently, I was introduced
to Swim,

https://metacpan.org/pod/Swim

Seems to cover most of the cases discussed here as part of
a toolchain.

It also supports includes (e.g., header, footer). E.g.,

https://github.com/ingydotnet/inline-c-pm/blob/master/doc/Inline/C/Cookbook.swim

The table thing is a different problem to solve, but it
seems like that will turn out to use verbatim sections
with the ability to align tables as part of an intermediate
tool, perhaps?

For images, it has a free form HTML capability,

I am not suggesting this as _the_ solution, but if there
was indeed a formalization of the toolchain flow this might
provide a workable option.

Cheers,
Brett

* William Lindley <wlindley@wlindley.com> [2021-11-30 08:57:58 -0500]:

> On 11/29/21 11:14 PM, Oodler 577 via perl5-porters wrote:
> > * tables can be defined in some intermediate way, but the resulting distribut-
> > ed "POD" visible via "perldoc" would be a neat ASCII table contained in a
> > verbatim section. And this preformatted text would be "safe" in all places
> > the POD would appear.
>
> Agree.
>
> In support of the toolchain concept:
>
> Given that POD can now contain Unicode, why not allow or encourage
> tables using the "Box Drawing" characters (a derivative of what we used
> to call the vt100 and IBM PC Line Drawing sets) as shown at
> https://unicode-table.com/en/blocks/box-drawing/
>
> For example, consider this in perlfaq4.pod:
>
> Pictures help... Here's the C<%hash> table:
>
> keys values
> +------+------+
> | a | 3 |
> | x | 7 |
> | d | 0 |
> | e | 2 |
> +------+------+
>
> Why shouldn't this be the nicer-looking:
>
> Pictures help... Here's the C<%hash> table:
>
> keys values
> ???????????????
> ? a ? 3 ?
> ? x ? 7 ?
> ? d ? 0 |
> ? e ? 2 ?
> ???????????????
>
> It is true that my (notional) DECwriter LA-36 won't nicely print that,
> but even the Linux console does nowadays.? It is curious that folks were
> widely using line-drawing characters in the late 1970s while Perl's
> standard POD today still doesn't. Given a toolchain that creates such
> tables in POD files, authors wouldn't even have to look up the gyrations
> to type all those funny characters. Is there any argument against this,
> when (per "perldoc perlpod") POD can officially contain any
> Encode::Supported encoding?
>
> The same is true if authors want to use Pikchr or *roff's "pic" or "dia"
> or whatever to create diagrams and flowcharts; given the
> table/image/diagram source code attached to the module, and a
> description of the filters used, that keeps distributions reproducible
> and modifiable.
>
> \\/
>

--
--
oodler@cpan.org
oodler577@sdf-eu.org
SDF-EU Public Access UNIX System - http://sdfeu.org
irc.perl.org #openmp #pdl #native