Mailing List Archive

[issue42973] argparse: mixing optional and positional arguments... not again
New submission from Tadek Kijkowski <tkijkowski@gmail.com>:

I have following use case for argparse:

./prog --tracks=80 image1 --tracks=40 image2 --tracks=80 image3 ...

I expected that the following parser would be able process that command line:

parser = argparse.ArgumentParser()
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

But it stops with 'unrecognized arguments: image2' error. It turns out that there is no way to prepare parser which could handle this command line.

There was a long discussion about this issue in #14191. There were two approaches proposed:
- Allow 'image' action to trigger multiple times with additional encountered parameters.
- Parse all optionals first and all positionals in second pass.

The first approach was represented by a patch by user guilherme-pg. This patch was slightly criticized (undeservedly IMO) and pretty much ignored.

The discussion then focused on the second approach which ended up with introduction of `parse_intermixed_args` into codebase.

The problem with `parse_intermixed_args` is that it defeats the purpose of having intermixed arguments. When the arguments are effectively reordered, there is no way to match "optionals" with "positionals" with they relate to.

In my option the Guilherme's approach was the correct way to go, I would say it was elegant. The only complaint against that patch is that it unnecessarily modified _StoreAction class - instead of that, the way to go is to use "extend" or custom action.

I allowed myself to simplify the changes a bit for readability. The patch doesn't change default functionality, for the cost of changing public API by adding new argument to ArgumentParser constructor.

With the changes applied, I can parse my command line with this code:

parser = argparse.ArgumentParser(greedy_star=True)
add = parser.add_argument
add('--tracks', choices=['40', '80'])
add('image', nargs='*', action=_CustomAction)
parser.parse_args()

----------
components: Library (Lib)
messages: 385301
nosy: monkeyman79
priority: normal
severity: normal
status: open
title: argparse: mixing optional and positional arguments... not again
type: enhancement
versions: Python 3.10

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Change by Tadek Kijkowski <tkijkowski@gmail.com>:


----------
keywords: +patch
pull_requests: +23083
stage: -> patch review
pull_request: https://github.com/python/cpython/pull/24259

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Change by Raymond Hettinger <raymond.hettinger@gmail.com>:


----------
nosy: +paul.j3, r.david.murray

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
paul j3 <ajipanca@gmail.com> added the comment:

Your patch is incomplete, without documentation or tests.

Your example is incomplete. That is _CustomAction? What namespace does your patch produce.

It's been a while since I worked on the intermixed patch, but as I recall the OP was happy with the fix. Also I don't recall any talk about 'matching "optionals" with "positionals" with they relate to. Was that part of the discussion?

On Stackoverflow I have seen questions about pairing arguments, and answered some. I don't recall the best answers. The tough version is when some of the grouped inputs may be missing

In your example I'd suggest making '--tracks' a nargs=2 and 'append' option. 'choices' would have to be replaced with a either a custom 'type', a custom Action, or post-parsing testing.

I haven't had time yet to test your patch.

One question I frequently ask posters who want to parse complex inputs is: 'how are going to explain this to your users?' 'What kind of usage and help do you want see?'

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

>> Your example is incomplete. That is _CustomAction? What namespace does your patch produce.

>> In your example I'd suggest making '--tracks' a nargs=2 and 'append' option. 'choices' would have to be replaced with a either a custom 'type', a custom Action, or post-parsing testing.

My use case and example were oversimplified. The real life use case would be a program that processes multiple sources, with various processing parameters which may change between sources and writes result to destination:

example.py:

#! /usr/bin/python3

import argparse

class _AddSourceAction(argparse.Action):
def __init__(self, option_strings, dest, nargs=None, const=None,
default=None, type=None, choices=None, required=False,
help=None, metavar=None):
super(_AddSourceAction, self).__init__(
option_strings=option_strings, dest=dest, nargs=nargs,
const=const, default=default, type=type,
choices=choices, required=required, help=help,
metavar=metavar)

def __call__(self, parser, namespace, values, option_string=None):
for source in values:
namespace.sources.append({'name': source,
'frobnicate': namespace.frobnicate,
'massage_level': namespace.massage_level})

def process_files(dest, sources):
for source in sources:
frob = source["frobnicate"]
print("processing %s, %sfrobnication, massage level %d " %
(source["name"],
"default " if frob is None else "no " if frob == False else "",
source["massage_level"]))
print("saving output to %s" % dest)

parser = argparse.ArgumentParser()
if hasattr(parser, "greedy_star"):
setattr(parser, "greedy_star", True)
parser.add_argument('destination')
parser.add_argument('--frobnicate', action='store_true', dest='frobnicate')
parser.add_argument('--no-frobnicate', action='store_false', dest='frobnicate')
parser.add_argument('--massage-level', type=int, default=5)
parser.add_argument('sources', metavar="source", nargs='*', action=_AddSourceAction)
parser.set_defaults(sources=[])
parser.set_defaults(frobnicate=None)
args = parser.parse_args()

process_files(args.destination, args.sources)

Without patch:

$ ./example.py output.txt --massage-level 4 input1.txt input2.txt --massage-level 5 input3.txt --frobnicate input4.txt input5.txt
example.py: error: unrecognized arguments: input1.txt input2.txt input3.txt input4.txt input5.txt

With patch:

$ PYTHONPATH=patch ./example.py output.txt --massage-level 4 input1.txt input2.txt --massage-level 5 input3.txt --frobnicate input4.txt input5.txt
processing input1.txt, default frobnication, massage level 4
processing input2.txt, default frobnication, massage level 4
processing input3.txt, default frobnication, massage level 5
processing input4.txt, frobnication, massage level 5
processing input5.txt, frobnication, massage level 5
saving output to output.txt

Note that I avoided passing greedy_star to constructor, to avoid error with unpatched argparse. Maybe that should be recommended way to enable this feature? N.b. I don't quite like this name 'greedy_star' seems to informal for me, but I didn't come up with anything better.

In this example frobnicate and massage_level apply to all following sources, but _AddSourceAction could be modified to reset their values, so they would only apply to first following source.

>> It's been a while since I worked on the intermixed patch, but as I recall the OP was happy with the fix. Also I don't recall any talk about 'matching "optionals" with "positionals" with they relate to. Was that part of the discussion?
>> On Stackoverflow I have seen questions about pairing arguments, and answered some. I don't recall the best answers. The tough version is when some of the grouped inputs may be missing

The use case above seems quite basic and common to me. I'm surprises that I actually didn't find any question on stackoverflow about this exact use case.

>> One question I frequently ask posters who want to parse complex inputs is: 'how are going to explain this to your users?' 'What kind of usage and help do you want see?'

I would (and I will, because I'm really working on a project which will do something like this) put all options which apply to source files into separate option group and add help string to this option group which explains that those options may be specified between source files and that they apply to all (or just first) following source files.

>> Your patch is incomplete, without documentation or tests.

Yeah, I can work on that if there is any chance that this will go anywhere. This is an effort from my side however, because I have quite slow mobile link at this moment. I tried to checkout python sources, but even with -depth 1 it took too long for my patience and I gave up after 'du -sh .git' showed 2G.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

>> I tried to checkout python sources, but even with -depth 1 it took too long for my patience and I gave up after 'du -sh .git' showed 2G.

Ignore that. I must have been doing something worng.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Change by Glenn Linderman <v+python@g.nevcal.com>:


----------
nosy: +v+python

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

I added tests and docs to the PR. How does it look now?

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

On 1/25/2021 12:43 PM, Tadek Kijkowski wrote:
> Tadek Kijkowski <tkijkowski@gmail.com> added the comment:
>
> I added tests and docs to the PR. How does it look now?
Could you send me the docs privately?

I'm trying to understand what you are suggesting, without reading the code.

I realize that you are trying to apply different values of optional
parameters to positional parameters... how are you saving that context?
Is it up to the special handler of the positional parameter to read and 
save the values of the optional parameters specified so far? What about
the ones unspecified so far? Can one even read and save the default value?

And where does it get stored? Is the value of the positional parameter
turned into a dict or class containing its own value + the saved
optional parameters?

I do agree that the even though the parse_intermixed_args was sufficient
for the use case I had at the time, and has been sufficient for me so
far, that it  not fully flexible, and I think I have seen Unix command
line programs that had rich semantics similar to what you are proposing,
where the sequence of repeated optional args could affect handling of
later positionals but not earlier ones.

So I applaud your efforts here, but simply reading the issue, and having
forgotten most of the internal workings of argument parser since getting
the compromise going, I think reading your docs would help clarify it
for me.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

>> Is it up to the special handler of the positional parameter to read and save the values of the optional parameters specified so far?

Yes, in order to get anything more that just concatenated list of positional parameters, one has to provide specialized action class. Basic functionality is easy to implement, as you can see in the fruits.py example.

>> ... how are you saving that context?

Entire context is stored in the namespace object. Action class has access to all previous parameters via the namespace and it can modify the namespace as needed. Standard 'store' action just adds value to the namespace, but specialized action can anything.

>> What about the ones unspecified so far?

They are obviously not available for the action processing positional parameters preceeding them, but will be present in the resulting namespace and can be processed after parsing is done.

>> Can one even read and save the default value?

I'm not sure what do you mean, but it all depend on how action class deals with it.

>> And where does it get stored? Is the value of the positional parameter turned into a dict or class containing its own value + the saved optional parameters?

It all gets stored in the namespace. Turning positionals + optionals into a dict or class seems most reasonable, but it all depends on how action class will deal with them.

>> I think I have seen Unix command line programs that had rich semantics similar to what you are proposing, where the sequence of repeated optional args could affect handling of later positionals but not earlier ones.

For example 'tar' with '-C' parameter which applies to all files following it.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

This is, I think, smallest functional example for matching optional parameters with positionals - fruits.py:

import argparse

DEFAULT_COLOR="plain"

class AddFruitAction(argparse.Action):
def __call__(self, parser, namespace, values, option_string=None):
for fruit in values:
namespace.fruits.append({'name': fruit, 'color': namespace.color})
namespace.color = DEFAULT_COLOR

def show_fruits(fruits):
for fruit in fruits:
print(f"{fruit['color']} {fruit['name']}")

parser = argparse.ArgumentParser(greedy=True)
parser.add_argument('--color', default=DEFAULT_COLOR)
parser.add_argument('fruits', nargs='*', action=AddFruitAction, default=[])
args = parser.parse_args()
show_fruits(args.fruits)

It starts with 'namespace.color' set to 'DEFAULT_COLOR' - 'default=DEFAULT_COLOR' takes care of that, and with 'namespace.fruits' set to empty list - via 'default=[]'.

For each group of positional command-line arguments, AddFruitAction is called with one or more fruit names in 'value'. The method iterates over them adding series of dicts to the 'fruits' list. The 'namespace.color' is immediately reset to default value, because we want the 'color' to apply to one following 'fruit'. If we wanted the 'color' to apply to all following 'fruits' the action class could just leave it alone.

After parsing is done, we get our namespace assigned to args, with list of color + fruit name pairs in 'args.fruits'.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

OK, I think I see what you are doing here. Thanks for your responses.

And probably it is the bare-bones minimum feature that allows user-implementation of "as complex as desired" combinations of optionals and context-sensitive positionals.

In the docs fruits example, though, I think the input and output are inconsistent: you have a brown banana reported as yellow?

I guess I could think of enhancements to simplify common cases. For example, each positional might want to track/save values for some number of optionals. So one could write custom code to do that, like your fruit basket, or one could imagine a canned approach, where the list of optionals was passed as a parameter when defining the positionals. This could be built as a wrapper for add_argument, that would call a more general version of the AddFruit Action. Perhaps a special parameter could also indicate saving the values of all optionals at the point of parsing the positional, giving each positional a complete set of optionals that apply.

Your feature of having the optional apply to only one positional would be a harder thing, especially if you wanted to have both kinds of optionals: that apply to the next positional, or that apply to all following positionals. But hmm. Maybe not much harder: just have two lists of optionals passed to the positional: a list of each kind.

Does the namespace still contain the default value for an optional after it has been used? It looks like it might, so your fruit example could be reduced to not needing the DEFAULT_COLOR variable, but rather change references to it to use

self._get_default_metavar_for_positional(action)

? That avoids using a global variable, and is for more helpful for the "list of optionals to be reset on each use" rather than re-creating the more complex data structure required to hold all the defaults.

I'm not sure how that plays with optionals that use append or append_const storage methods, nor do I fully understand the comment in "def _copy_items(items):"

I think if you could avoid the global variable, your implementation would be stronger. And if you provide, either as a sample subclass, or argparse new feature, the ability to do the things I mention here, it would be even stronger, as it would allow the user to deal with the more powerful capabilities without needing as much understanding of argparse internals.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

for more helpful => far more helpful

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

>> In the docs fruits example, though, I think the input and output are inconsistent: you have a brown banana reported as yellow?

Yeah, I noticed that and fixed already :)


I agree with your ideas. My goal was just to provide bare minimum that would allow users to implement this functionality on their own, but turning it into something useful out of the box sounds like a good idea.

There could be new predefined action, e.g. 'extend_with_capture' (or just 'extend_capture') which could be used like this:

parser.add_argument('fruits', nargs='*', action='extend_capture', capture_once=['color'])

Where 'capture' would be one of: a list of parameters to capture (could be a string instead of a list if it's just one parameter) or '*' to capture all optional parameters and 'capture_once' would be similar list of parameters to capture and reset to default.

>> Does the namespace still contain the default value for an optional after it has been used? It looks like it might

I think it does - not the namespace, but ArgumentParser, which is also available in Action.__call__, has 'get_defaults'.

>> I'm not sure how that plays with optionals that use append or append_const storage methods, nor do I fully understand the comment in "def _copy_items(items):"

I don't see any danger here, the action will just have to make sure to capture copy of the array, rather than reference to it. The same applies to 'extend'.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
paul j3 <ajipanca@gmail.com> added the comment:

I haven't had a chance to study your longer posts, but it seems to me that the AddFruitAction example could just as well be implemented with

parser.add_argument('--color', nargs='*', action='append')

with post parsing processing to create the 'fruits' dicts from the appended lists.

The basic logic of argparse is to accept optionals in any order, and positionals in strict positional order. 'nargs' allows us to pair any number of strings with each optional's flag.

While custom Action classes can implement interactions between arguments based on values in the namespace, it is usually easier to do this after parsing.

But back to your core change, I wonder if adding a new nargs, such as '**' would be better than than the new parameter 'greedy_star=True'. I prefer not to add parameters that are hard to document. At this point in the argparse development, changes should be minimally invasive.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

This sounds very good to me. Might also want action='store_capture' for a single positional definition?

capture could be a string, or any iterable of strings (tuple comes to mind)

capture_once have similar value as capture, but I wonder if the naming would be better as capture_reset to indicate the value is reset to default, rather than just captured.


Tadek said:
There could be new predefined action, e.g. 'extend_with_capture' (or just 'extend_capture') which could be used like this:

parser.add_argument('fruits', nargs='*', action='extend_capture', capture_once=['color'])

Where 'capture' would be one of: a list of parameters to capture (could be a string instead of a list if it's just one parameter) or '*' to capture all optional parameters and 'capture_once' would be similar list of parameters to capture and reset to default.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

Paul said:
I haven't had a chance to study your longer posts, but it seems to me that the AddFruitAction example could just as well be implemented with

parser.add_argument('--color', nargs='*', action='append')

with post parsing processing to create the 'fruits' dicts from the appended lists.


That was also my first reaction, Paul, when I read Tadek's proposal. But I quickly realized that particularly with the feature of "capture and reset to default" that the number of instances of a particular optional and the number of positionals do not always match, and there is no way to correlate them later, even if they are all captured and saved.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

Paul,

> The basic logic of argparse is to accept optionals in any order, and positionals in strict positional order. 'nargs' allows us to pair any number of strings with each optional's flag.

I started this issue because that was insufficient for my project.

> I wonder if adding a new nargs, such as '**' would be better than than the new parameter 'greedy_star=True'.

Yeah, I considered that earlier, but I thought that would be too much effort to review all the places where it would have to be treated as equal to '*'. But now I agree with you, I'll try to change that.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

I went with Paul's suggestion and added nargs='**' instead of greedy, so at this moment this PR doesn't introduce any changes in any public signatures.

I'm working on 'capture' actions, but I want that to be a separate PR and a separate issue because - first - it may be harder to accept as it is bigger change in both code and documentation and - second - it actually is a separate feature, which could be useful for optional arguments even without nargs='**' implemented.

So, what do you all think about this PR as it now? Docs and tests are updated.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

Yes I think this is a useful enabling step toward enhanced functionality, as is.

But I think the learning curve to achieve the enhanced functionality is a bit high for most people, as it requires too much knowledge of argparse internals, so I really look forward to your followon work on 'capture' actions. which will be an enhancement that will not require internals knowledge, and will resolve a day-one deficiency in argparse in a nicely backward-compatible manner.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
paul j3 <ajipanca@gmail.com> added the comment:

So in the big picture, the purpose of this change is to treat the inputs like a kind of state-machine.

In the bigger example, the `**` positional values are processed one by one, using the interspersed optionals to set state parameters.

`args.sources` then ends up as a list of dicts, each of the form:

{'name': 'input3.txt', 'frobnicate': None, 'massage_level': 5}

where 'name' is the positional value, and 'frobnicate' and 'massage_level' are the latest values (default or user supplied). The optionals can be specified in any order or repeatedly.

While the proposed change to the core parser is (apparently) minor, it does occur at the center of the action. That is not the place we want any (new) bugs or backward incompatibility. And the full implementation requires a new Action subclass that is quite different from existing ones.

Given how different this is from the normal argparse parsing (and the POSIX parsing argparse seeks to emulate), I question the wisdom of adding this, in part or whole, to the stock distribution. It could certainly be published as a pypi. That already has a number of parsers, some built on argparse, others stand alone.

I also wonder whether it would be simpler to do this kind of parsing directly from sys.argv. Just step through that list, consuming the values and flags in sequence.

Sorry to sound like a wet blanket, but I've seen too many seemingly innocent patches that caused unforeseen problems down the line.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
paul j3 <ajipanca@gmail.com> added the comment:

Sometimes patches have unforeseen benefits. My earlier patch for this issue, parse_intermixed_args, has been useful beyond the OP's case.

https://stackoverflow.com/questions/50916124/allow-positional-command-line-arguments-with-nargs-to-be-seperated-by-a-flag

https://bugs.python.org/issue15112
argparse: nargs='*' positional argument doesn't accept any items if preceded by an option and another positional

With

usage: test.py [-h] [-a A] b [c]

and

test.py B -a A C

has problems because the optional '[c]' positional is used up when 'b' is processed. Intermixed gets around this by first processing '-a', and then handling 'b [c]' together.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Glenn Linderman <v+python@g.nevcal.com> added the comment:

paul j3 said:

Given how different this is from the normal argparse parsing (and the POSIX parsing argparse seeks to emulate), I question the wisdom of adding this, in part or whole, to the stock distribution. It could certainly be published as a pypi. That already has a number of parsers, some built on argparse, others stand alone.


I say:

This has been a deficiency of argparse from day one. Tadek's solution seems to enable addressing the deficiency in a backward-compatible manner. Do you, paul, find any test failures? Or any incompatibilities that may not be in the test cases? If not, then it certainly does seem like a wet-blanket comment.


paul j3 forther said:

I also wonder whether it would be simpler to do this kind of parsing directly from sys.argv. Just step through that list, consuming the values and flags in sequence.


I say:

The whole point of argparse, and of deprecation of the prior optparse was to make more functionality available in a more powerful API. Increasing the power of the API seems to be consistent with the purpose of argparse. It took me some time and use cases to discover the limitations of argparse, and although parse_intermixed_args solved the use cases I had, I was well aware that it didn't solve cases of every existing Unix utility.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Tadek Kijkowski <tkijkowski@gmail.com> added the comment:

> So in the big picture, the purpose of this change is to treat the inputs like a kind of state-machine.

Not necessarily that. Simple parsers should be easy to write, complicated parsers should be _possible_ to write. Users should be able to do whatever they wish by providing custom actions. Like writing Conway's Life in Excel or building Turing Machine using bricks and conveyors. Problem with positional parameter is a blocker. Fact that you can: 1. mix positional parameters with options in general, 2. have unlimited number of positional parameters, but 3. can't have 1 and 2 at the same time, makes great number of things impossible. For me that looks like a bug, not a feature.

Capture actions are second priority for me, because if this PR goes through, user will be able to write his own actions.

> While the proposed change to the core parser is (apparently) minor, it does occur at the center of the action. That is not the place we want any (new) bugs or backward incompatibility.

The core change is apparently minor, and also actually minor. It boils out to this:

@@ -2010,17 +2020,23 @@ def consume_positionals(start_index):
match_partial = self._match_arguments_partial
selected_pattern = arg_strings_pattern[start_index:]
arg_counts = match_partial(positionals, selected_pattern)
+ action_index = 0

# slice off the appropriate arg strings for each Positional
# and add the Positional and its args to the list
- for action, arg_count in zip(positionals, arg_counts):
+ for arg_count in arg_counts:
+ action = positionals[action_index]
args = arg_strings[start_index: start_index + arg_count]
start_index += arg_count
take_action(action, args)
+ # if positional action nargs is '**',
+ # never remove it from actions list
+ if action.nargs != AS_MANY_AS_POSSIBLE:
+ action_index += 1

# slice off the Positionals that we just parsed and return the
# index at which the Positionals' string args stopped
- positionals[:] = positionals[len(arg_counts):]
+ positionals[:] = positionals[action_index:]
return start_index

It's not hard to determine that if all action.nargs != AS_MANY_AS_POSSIBLE, old and new code do exactly the same.
Besides that's what tests and code review is for. As for backward compatibility, nargs='**' was illegal until now,
and for all parameters with nargs!='**' the code behaves exactly as before.

> And the full implementation requires a new Action subclass that is quite different from existing ones.

No, that's an extra. That's why it's separate issue and separate PR. This issue is about giving users ability to write their own actions.

> Given how different this is from the normal argparse parsing (and the POSIX parsing argparse seeks to emulate), I question the wisdom of adding this, in part or whole, to the stock distribution. It could certainly be published as a pypi. That already has a number of parsers, some built on argparse, others stand alone.
>
> I also wonder whether it would be simpler to do this kind of parsing directly from sys.argv. Just step through that list, consuming the values and flags in sequence.

In other words, "if argparse doesn't meet your requirements, you can write your own parser". I don't think that's the way to go.

Both GNU getopt and GNU argp_parse allow mixing options with positional arguments without reordering. For getopt it's: options argument string begins with a hyphen, for argp_parse it's ARGP_IN_ORDER flag. I don't see why argparse wouldn't allow that. Original POSIX getopt didn't even have longopts.

> Sorry to sound like a wet blanket, but I've seen too many seemingly innocent patches that caused unforeseen problems down the line.

Yeah, that can be said about any project at any time - changes can introduce bugs, so let's just keep it as it is. I appreciate your ideas, but your criticism does sound unsubstantiated: "POSIX getopts didn't have it", "it can be done on pypi", "enhancements introduce bugs".

I think this change should be applied, because it is small change in code which removes significant obstacle in writing advanced option parsers.

----------

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com
[issue42973] argparse: mixing optional and positional arguments... not again [ In reply to ]
Change by Tadek Kijkowski <tkijkowski@gmail.com>:


----------
pull_requests: +23190
pull_request: https://github.com/python/cpython/pull/24367

_______________________________________
Python tracker <report@bugs.python.org>
<https://bugs.python.org/issue42973>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: https://mail.python.org/mailman/options/python-bugs-list/list-python-bugs%40lists.gossamer-threads.com