argparse: positional args with nargs='*' defaults to []

[cjerdonek]

BPO	16878
Nosy	@bitdancer, @cjerdonek
Files	issue-16878-1.patch issue-16878-2.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2013-01-06.09:41:39.533>
labels = ['3.8', 'type-bug', 'library', '3.9', '3.10']
title = "argparse: positional args with nargs='*' defaults to []"
updated_at = <Date 2020-11-06.22:49:20.356>
user = 'https://github.com/cjerdonek'

bugs.python.org fields:

activity = <Date 2020-11-06.22:49:20.356>
actor = 'iritkatriel'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2013-01-06.09:41:39.533>
creator = 'chris.jerdonek'
dependencies = []
files = ['28623', '28664']
hgrepos = []
issue_num = 16878
keywords = ['patch']
message_count = 10.0
messages = ['179174', '179184', '179236', '179237', '179239', '179289', '179511', '187146', '198035', '223160']
nosy_count = 5.0
nosy_names = ['bethard', 'r.david.murray', 'chris.jerdonek', 'paul.j3', 'sebix']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue16878'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

Linked PRs

• GH-61082: Clarify nargs='*' positional default behavior #150989

[cjerdonek]

In argparse, positional arguments with nargs='*' default to [] rather None, even if default=None is passed explicitly. The documentation says otherwise:

"The default keyword argument of add_argument(), whose value defaults to None, specifies what value should be used if the command-line argument is not present. ... For positional arguments with nargs equal to ? or *, the default value is used when no command-line argument was present:"

(from http://docs.python.org/dev/library/argparse.html#default )

import argparse

def parse(args, **kwargs):
    parser = argparse.ArgumentParser()
    parser.add_argument('foo', **kwargs)
    ns = parser.parse_args(args)
    print(repr(ns.foo))

parse([], nargs='?')                # None
parse([], nargs='*')                # []    <--
parse([], nargs='*', default=None)  # []    <--
parse([], nargs='*', default=False) # False
parse([], nargs='*', default=0)     # 0

Three options include (there may be more):

(1) document the behavior
(2) make a default of None yield None
(3) do (2), but change the default to [] instead of None when nargs='*'

[bitdancer]

I'd prefer to fix it, since having the action='append' and nargs='*' behaviors be different looks like a bug. However, fixing it to match 'append' (default really is None) would be likely to break working code, so we certainly couldn't backport that, and I'd be really reluctant to fix it that way in 3.4, either. (3) is thus preferable to (2), even though it leaves us with an inconsistency. It would still be a judgment call on whether or not to backport it. It seems like it would be reasonably safe, but such changes always worry me :)

[cjerdonek]

I agree it would be very likely to break working code. Can you elaborate on your point about 'append' though? I'm not sure I see it.

Aside from consistency, I'm wondering if there is ever a case where it would help to return None for positional arguments. For example, unlike with optional arguments, it doesn't seem like it would ever make sense to distinguish between the option being present and the option being present with no values (which is why const is needed in the optional case). In other words, there is no loss of information by returning [].

[bitdancer]

>>> p.add_argument('-a', action="append", default=None)
>>> p.parse_args([])
Namespace(a=None)
>>> p.parse_args(['-a', '1', '-a', '2'])
Namespace(a=['1', '2'])

So there's a logical correspondence there (repeated option vs multiple values, each producing a list).

I'm not sure what you mean by the difference between options and positionals. I note that you can use nargs with an optional, and in that case the default works, which is even more unfortunate than the difference with 'append'. But if you can articulate a logical difference between optionals and positional here, maybe we can make it look reasonable :)

[cjerdonek]

I was referring to the fact that optionals have an additional case that positionals don't have: "Note that for optional arguments, there is an additional case -- the option string is present but not followed by a command-line argument."

(from http://docs.python.org/dev/library/argparse.html#nargs )

>>> p.add_argument('--foo', nargs='*', default=None)
>>> p.parse_args([])
Namespace(foo=None)
>>> p.parse_args(['--foo'])
Namespace(foo=[])

So it could be argued that positionals (at least by default) are behaving like the second case. But that's as far as the parallel goes apparently. *default* affects the first case and not the second case for optional arguments:

>>> p.add_argument('--foo', nargs='*', default=False)
>>> p.parse_args([])
Namespace(foo=False)
>>> p.parse_args(['--foo'])
Namespace(foo=[])

[cjerdonek]

Attached is a doc patch. I also improved some other aspects of the *default* section while I was there.

We should probably make sure a test exists for the newly-documented behavior (i.e. for passing no arguments for a positional argument with nargs='*' and default=None).

[cjerdonek]

It turns out that there is already a test case:

http://hg.python.org/cpython/file/05183ce544be/Lib/test/test_argparse.py#l799

(at the line "('', NS(foo=[])),")

I've updated the patch with a note to reflect this.

[paulj3]

In this example:

    >>> p.add_argument('--foo', nargs='*', default=None)
    >>> p.parse_args([])
    Namespace(foo=None)
    >>> p.parse_args(['--foo'])
    Namespace(foo=[])

'p.parse_args([])' just assigns the default to 'foo' in the Namespace.

"p.parse_args(['--foo'])" invokes 'take_action(<dest='foo'>,[]). That is, it 'assigns' an empty array to 'foo'. The same thing would happen if 'foo' was a positional.

'take_action' then passes these arguments to '_get_values'. That is where the differences between '?' and '*' arise.

The key pieces of code in '_get_values' when arg_strings==[] are:

        # optional argument produces a default when not present
        if not arg_strings and action.nargs == OPTIONAL:
            ....value = action.default
            # and evaluate 'value' if is a string

    # when nargs='*' on a positional, if there were no command-line
    # args, use the default if it is anything other than None
    elif (not arg_strings and action.nargs == ZERO_OR_MORE ...):
        if action.default is not None:
            value = action.default
        else:
            value = arg_strings # i.e. []

In other words, if nargs='?', the attribute gets its default value. But for '*', this is true only if the default is not None.

So in:

parse([], nargs='?')                # get the default value: None
parse([], nargs='*')                # default is None, get arg_strings []
parse([], nargs='*', default=None)  # same case
parse([], nargs='*', default=False) # default is not None, get default
parse([], nargs='*', default=0)     # same case

I tried changing the _get_values() so '*' got the default (like '?' does), and got 54 failures when running test_argparse.py.

[paulj3]

On a related point, the 'action.required' value is set differently for '?' and '*' positionals.

    if kwargs.get('nargs') not in [OPTIONAL, ZERO_OR_MORE]:
        kwargs['required'] = True
    if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:
        kwargs['required'] = True

OPTIONAL is always not required, ZERO_OR_MORE is not required if it has a default. But for reasons discussed here, that 'required' value makes little difference.

parse_args checks that all 'required' arguments have been seen, but a ZERO_OR_MORE positional is always seen (i.e. it matches an empty string).

Usage formatting always uses '[%s [%s ...]]' with ZERO_OR_MORE, regardless of the 'required' attribute.

The only place where this 'required' value seems to matter is when adding such an argument to a mutually exclusive group. But if an unused '*' positional is going to get a '[]' value anyways, why should it be excluded from such a use?

If I remove the

    if kwargs.get('nargs') == ZERO_OR_MORE and 'default' not in kwargs:

test, test_argparse.py still runs fine.

http://bugs.python.org/issue18943 is a possibly related issue, involving a 'is not action.default' test in a mutually exclusive group.

[paulj3]

The documentation patch here should note that a positional '*' string default is not parsed (i.e. does not pass through _get_value).

(see http://bugs.python.org/issue17250)