Perceived optparse shortcomings

[Hans-Joachim Widmaier]

I was really pleased when the Optik module found its way into the
standard Python battery compartment, as it matched all my option
parsing requirements. But, as always, there's really nothing that does
all you ever want, especially when it comes to option parsing -
there's just too many schemes to handle them all comfortably.

With this program I'm working on, I started to recognize that not only
the functionality should be delegated to modules, but that most
command line options are private to these modules (call 'em plug-ins)
as well. This calls for a command line like CVS or subverion employ:

<cmd> <generic options> <subcmd> <subcmd options> <args>

There are several ways to do it:
- Let one option parser recognize all options. No drop-in plug-ins,
no possibility for the same options meaning different things in
different plug-ins (no, this isn't neccessarily a bad design anyway).
Obviously horrible.
- Don't allow interspersed arguments. This is something I dislike. I
want to be able to type a forgotten generic option when I'm at the end
of a possibly long line. Even with modern shells its a nuisance to
have to go to the line's beginning, skip the command and insert the
option.
- Augment optparse to just ignore unknown options and treat them as
arguments. Doesn't allow clustering of generic and subcommand-specific
options, doesn't allow for generic options to be arguments to
subcommand-specific options, but these seem as minor points to me. So
I did that. And I'd like to ask if this
might have a chance to get into the standard optparse module (It's
just a few lines).

The bigger problem, though, is internationaliz ation. I don't care for
all the exceptions raised by programmer errors, but the usage, help
and user-visible error messages should really be inernationalize d.
Subclassing optparse doesn't help much here, as the strings are
sprinkled all over. Is it even thinkable that optparse (or, in the
end, the whole standard library) be prepared for internationaliz ation?
Couldn't it be handled like that:

At the module's top:

import __builtin__
try:
__builtin__._
except AttributeError:
def _(s):
return s

and surrounding all relevant strings with _()?

Having all programs only talk english or having to rewrite every
module certainly can't be the answer.

Deutsch labernd der Eure,
Hans-Joachim Widmaier

[Heather Coppersmith]

Re: Perceived optparse shortcomings

On 16 Mar 2004 23:20:10 -0800,
hjwidmaier@web. de (Hans-Joachim Widmaier) wrote:
[color=blue]
> With this program I'm working on, I started to recognize that
> not only the functionality should be delegated to modules, but
> that most command line options are private to these modules
> (call 'em plug-ins) as well. This calls for a command line like
> CVS or subverion employ:[/color]
[color=blue]
> <cmd> <generic options> <subcmd> <subcmd options> <args>[/color]
[color=blue]
> There are several ways to do it:[/color]

[ snip ]

How about having your plugins "register" with your main program
*before* you call optparse? Part of the registration would be
something like "I (the plugin) accespt these options with/without
these arguments, and here are some callbacks to handle them."
During registration, build up the argument strings/arrays/etc. you
need for optparse from that information.

I've not used optparse, but I've used that scheme with getopt.

If you allow interspersed options, you (your main program) will
have to handle conflicts gracefully (i.e., plugin1 and plugin2
both want an "-a" option).

Regards,
Heather

--
Heather Coppersmith
That's not right; that's not even wrong. -- Wolfgang Pauli

[Skip Montanaro]

Re: Perceived optparse shortcomings

>> <cmd> <generic options> <subcmd> <subcmd options> <args>

Heather> How about having your plugins "register" with your main program
Heather> *before* you call optparse? ....

Heather> I've not used optparse, but I've used that scheme with getopt.

I was thinking that with getopt I would call it once to process the generic
options. That would leave the args as the list equivalent of

<subcmd> <subcmd options> <args>

The main program plucks <subcmd> from that list and then passes the
remaining args to the appropriate subcommand, which calls getopt again with
the list equivalent of

<subcmd options> <args>

I don't know if that's possible with optparse, but if so, that's the route
I'd try first.

Skip

[David Goodger]

Re: Perceived optparse shortcomings

Hans-Joachim Widmaier wrote:[color=blue]
> This calls for a command line like CVS or subverion employ:
>
> <cmd> <generic options> <subcmd> <subcmd options> <args>[/color]

I suggest the following:

1. Build an OptionParser for your <cmd> that accepts only the generic
options. Make sure to call
OptionParser.di sable_intersper sed_args() before calling
.parse_args(). The first argument in the returned args list should
be your <subcmd> (be sure to check!).

2. Once you know your <subcmd>, build a new OptionParser that handles
its options. Pass the argument list returned from step 1 (less the
subcmd) to .parse_args().

If you really want the subcmd's option parser to handle the <cmd>'s
generic options too, you could augment the OptionParser built in
step 1:

parser = OptionParser()
parser.add_opti on(...)
...
parser.disable_ interspersed_ar gs()
values, args = parser.parse_ar gs()

subcmd = args[0]
parser.add_opti on(...) # depending on subcmd
...
parser.enable_i nterspersed_arg s() # if you like
values, args = parser.parse_ar gs(args=args[1:], values=values)

You'll have to be careful about conflicts.
[color=blue]
> - Don't allow interspersed arguments. This is something I
> dislike. I want to be able to type a forgotten generic option when
> I'm at the end of a possibly long line.[/color]

You won't have any arguments *before* the subcmd, so the above should
work for you.
[color=blue]
> - Augment optparse to just ignore unknown options and treat them as
> arguments.[/color]

There goes all error-checking.
[color=blue]
> Doesn't allow clustering of generic and subcommand-specific options,[/color]

You'd have to avoid conflicts completely.
[color=blue]
> doesn't allow for generic options to be arguments to
> subcommand-specific options,[/color]

What does that mean?
[color=blue]
> but these seem as minor points to me. So I did that. And I'd like to
> ask if this might have a chance to get into the standard optparse
> module (It's just a few lines).[/color]

Your "minor points" will be major to someone else, so I doubt a
simplistic patch will be accepted. But by all means, show the code
(not here though; see below).
[color=blue]
> The bigger problem, though, is internationaliz ation. ... Is it even
> thinkable that optparse (or, in the end, the whole standard library)
> be prepared for internationaliz ation?[/color]

I imagine that optparse would be open to internationaliz ation. I'd
like to see some of optparse's strings be parameterized instead of
hard-coded. These changes just need someone to implement (and write
tests for, and document) them.

In any case, discussions here are unlikely to be seen, and almost
certainly won't be acted upon. I suggest posting to the optik-users
(@lists.sf.net) list or filing a feature request. A working code,
test, & doc patch would be helpful too.

--
David Goodger http://python.net/~goodger
For hire: http://python.net/~goodger/cv

[Jeff Epler]

Re: Perceived optparse shortcomings

> Hans-Joachim Widmaier wrote:[color=blue][color=green]
> > doesn't allow for generic options to be arguments to
> > subcommand-specific options,[/color]
>[/color]
On Wed, Mar 17, 2004 at 10:03:51AM -0500, David Goodger wrote:[color=blue]
> What does that mean?[/color]

Using the example of CVS again, the "-z#" argument specifies that
compression should be used for network connections. I think that these
have the same effect:
cvs -z3 co foo
cvs co -z3 foo
"-z#" can appear before or after subcommand in the commandline and has
the same meaning in either location.

Jeff

[David M. Cooke]

Re: Perceived optparse shortcomings

At some point, Jeff Epler <jepler@unpytho nic.net> wrote:
[color=blue][color=green]
>> Hans-Joachim Widmaier wrote:[color=darkred]
>> > doesn't allow for generic options to be arguments to
>> > subcommand-specific options,[/color]
>>[/color]
> On Wed, Mar 17, 2004 at 10:03:51AM -0500, David Goodger wrote:[color=green]
>> What does that mean?[/color]
>
> Using the example of CVS again, the "-z#" argument specifies that
> compression should be used for network connections. I think that these
> have the same effect:
> cvs -z3 co foo
> cvs co -z3 foo
> "-z#" can appear before or after subcommand in the commandline and has
> the same meaning in either location.[/color]

You didn't try it, did you :-)? It doesn't work -- generic options have to
come before the subcommand w/ cvs.

--
|>|\/|<
/--------------------------------------------------------------------------\
|David M. Cooke
|cookedm(at)phy sics(dot)mcmast er(dot)ca

[Hans-Joachim Widmaier]

Re: Perceived optparse shortcomings

David Goodger <goodger@python .org> wrote in message news:<mailman.8 3.1079535848.74 2.python-list@python.org >...[color=blue]
> Hans-Joachim Widmaier wrote:[color=green]
> > This calls for a command line like CVS or subverion employ:
> >
> > <cmd> <generic options> <subcmd> <subcmd options> <args>[/color]
>
> I suggest the following:
>
> 1. Build an OptionParser for your <cmd> that accepts only the generic
> options. Make sure to call
> OptionParser.di sable_intersper sed_args() before calling
> .parse_args(). The first argument in the returned args list should
> be your <subcmd> (be sure to check!).[/color]

As I wrote somewhat later: I dislike
OptionParser.di sable_intersper sed_args().
<subcmd> is in reality just another option (as it's not really a
command; it selects a plugin that handles the special requirements of
a device).
[color=blue]
> 2. Once you know your <subcmd>, build a new OptionParser that handles
> its options. Pass the argument list returned from step 1 (less the
> subcmd) to .parse_args().[/color]

I have a generic interface that handles the generic options, scans its
directory for plug-ins and imports them. If one is selected, the
not-consumed arguments (and options) are passed to it. The plug-in
then does standard option parsing.
Works like a charm. (Bigger picture: The whole thing has a GUI also,
where you can select the plug-in interactively; the whole commandline
thing is to allow for a batch mode, too.)
[color=blue]
> If you really want the subcmd's option parser to handle the <cmd>'s
> generic options too, you could augment the OptionParser built in
> step 1:[/color]

There's just one option that needs to be handled, too (so it can be
called itself). Trivial to just define it with SUPPRESS_HELP if not
called standalone.
[color=blue][color=green]
> > - Augment optparse to just ignore unknown options and treat them as
> > arguments.[/color]
>
> There goes all error-checking.[/color]

Nah, it doesn't. Of course this is optional, and after the plug-in
parses the remains, there's nothing left to worry about.
[color=blue][color=green]
> > doesn't allow for generic options to be arguments to
> > subcommand-specific options,[/color]
>
> What does that mean?[/color]

E.g. if you have a generic option --bar and a subcmd-specific option
--foo that takes a string argument. Now if you call

<cmd> <subcmd> --foo --bar

--bar will be seen as a generic option and not as an argument to
--foo. But that's not really a problem, as it _is_ ambigous anyway
(might be a forgotten argument after all).
[color=blue]
> Your "minor points" will be major to someone else, so I doubt a
> simplistic patch will be accepted. But by all means, show the code
> (not here though; see below).[/color]

The patch is just a few lines (and took much less time to write than
this article), and it only implements an optional argument to
OptionParser.pa rse_args() that allows it to treat unknown options as
arguments.
[color=blue]
> In any case, discussions here are unlikely to be seen, and almost
> certainly won't be acted upon. I suggest posting to the optik-users
> (@lists.sf.net) list or filing a feature request. A working code,
> test, & doc patch would be helpful too.[/color]

Thanks for your suggestion.
Hans-Joachim

[Jeff Epler]

Re: Perceived optparse shortcomings

On Thu, Mar 18, 2004 at 12:49:15AM -0500, David M. Cooke wrote:[color=blue]
> You didn't try it, did you :-)? It doesn't work -- generic options have to
> come before the subcommand w/ cvs.[/color]

Well, I tried this and got the error I expected:
$ cvs co -z3
cvs checkout: No CVSROOT specified! Please use the `-d' option
cvs [checkout aborted]: or set the CVSROOT environment variable.
... but now I see that it doesn't actually work that way:
$ cvs diff -z3
diff: invalid option -- z

Jeff

[Hans-Joachim Widmaier]

Re: Perceived optparse shortcomings

Am Wed, 17 Mar 2004 09:36:24 -0600 schrieb Skip Montanaro:
[color=blue]
> The main program plucks <subcmd> from that list and then passes the
> remaining args to the appropriate subcommand, which calls getopt again with
> the list equivalent of
>
> <subcmd options> <args>
>
> I don't know if that's possible with optparse, but if so, that's the route
> I'd try first.[/color]

After some small changes to optparse, it is. It's the route I went, and
I'm satisfied with the result.

Hans-J.