Problem with mixing doctest with gettext _()

[Pierre Rouleau]

I have a problem writing self-testable modules using doctest when these
modules have internationaliz ed strings using gettext _('...').

- The main module of an application (say app.py) calls gettext.install ()
to install the special _ function inside Python builtin. Other modules,
taken from a general purpose collection of Python modules, also support
internationalis ation and doctest testing.

For example:

utstring.py would contain some code like this:

def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**
[color=blue][color=green][color=darkred]
>>> onOffStr(True)[/color][/color][/color]
u'ON'[color=blue][color=green][color=darkred]
>>> onOffStr(False)[/color][/color][/color]
u'OFF'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
"""
if isOn:
return _(u"ON") # notice the underscore
else:
return _(u"OFF")



The utstring module does not call any of the gettext calls, because some
other module does it when the application runs. So the doctest fails:

*************** *************** *************** *************** *****
Failure in example: onOffStr(True)
from line #4 of utstring.onOffS tr
Exception raised:
Traceback (most recent call last):
File "C:\Python23\Li b\doctest.py", line 442, in _run_examples_i nner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "C:\dev\python\ utstring.py", line 513, in onOffStr
return _(u"ON")
TypeError: 'tuple' object is not callable
*************** *************** *************** *************** *****

I tried to define a _() function when testing with the code below but
the doctest still fails. The following code is at the end of my
utstring.py module

def _test():
"""_test() perform docstring test"""

import doctest, utstring
return doctest.testmod (utstring)

if __name__ == "__main__":
def _(aString):
# dummy _() attempting to get doctest to pass.
return aString

_test()


----------

Does anyone know why the doctest still fails when I define the dummy _()
function?


Thanks in advance.

Pierre

[Skip Montanaro]

Re: Problem with mixing doctest with gettext _()


Pierre> I tried to define a _() function when testing with the code
Pierre> below but the doctest still fails. The following code is at the
Pierre> end of my utstring.py module
...

I suspect it's because your dummy _ is not in builtins. Why not just call
gettext.install () where you are currently defining _?

Skip

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Skip Montanaro wrote:
[color=blue]
> Pierre> I tried to define a _() function when testing with the code
> Pierre> below but the doctest still fails. The following code is at the
> Pierre> end of my utstring.py module
> ...
>
> I suspect it's because your dummy _ is not in builtins. Why not just call
> gettext.install () where you are currently defining _?
>[/color]

Skip, this looks like the way to go.

I was trying to avoid it because the translation files for these library
modules are constructed at the application level somewhere else. But I
was also considering creating a translation library for these module, so
I guess that is one more incentive to do so...

Thanks

Pierre

[Skip Montanaro]

Re: Problem with mixing doctest with gettext _()

[color=blue][color=green]
>> I suspect it's because your dummy _ is not in builtins. Why not just
>> call gettext.install () where you are currently defining _?[/color][/color]

Pierre> Skip, this looks like the way to go.

Pierre> I was trying to avoid it because the translation files for these
Pierre> library modules are constructed at the application level
Pierre> somewhere else. But I was also considering creating a
Pierre> translation library for these module, so I guess that is one
Pierre> more incentive to do so...

If you really want a dummy _() you can also stuff your version into
builtins:
[color=blue][color=green][color=darkred]
>>> import __builtin__
>>> def foo(s): return s[/color][/color][/color]
...[color=blue][color=green][color=darkred]
>>> __builtin__._ = foo
>>> _[/color][/color][/color]
<function foo at 0x1d6670>[color=blue][color=green][color=darkred]
>>> _("hi")[/color][/color][/color]
'hi'

Skip

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Skip Montanaro wrote:

[color=blue]
>
> If you really want a dummy _() you can also stuff your version into
> builtins:
>[color=green][color=darkred]
> >>> import __builtin__
> >>> def foo(s): return s[/color][/color]
> ...[color=green][color=darkred]
> >>> __builtin__._ = foo
> >>> _[/color][/color]
> <function foo at 0x1d6670>[color=green][color=darkred]
> >>> _("hi")[/color][/color]
> 'hi'
>[/color]

I tried that, but it only works for the first call...

[Shell buffer started: python.exe]
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import __builtin__
>>> def foo(s): return s[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> __builtin__._ = foo
>>> _[/color][/color][/color]
<function foo at 0x008F98B0>[color=blue][color=green][color=darkred]
>>> _[/color][/color][/color]
<function foo at 0x008F98B0>[color=blue][color=green][color=darkred]
>>> _('hi')[/color][/color][/color]
'hi'[color=blue][color=green][color=darkred]
>>> _[/color][/color][/color]
'hi'[color=blue][color=green][color=darkred]
>>> _('Hello')[/color][/color][/color]
Traceback (most recent call last):
File "<stdin>", line 1, in ?
TypeError: 'str' object is not callable[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]


Pierre

[Peter Otten]

Re: Problem with mixing doctest with gettext _()

Pierre Rouleau wrote:
[color=blue]
> Skip Montanaro wrote:
>[color=green]
>> If you really want a dummy _() you can also stuff your version into
>> builtins:
>>[color=darkred]
>> >>> import __builtin__
>> >>> def foo(s): return s[/color]
>> ...[color=darkred]
>> >>> __builtin__._ = foo
>> >>> _[/color]
>> <function foo at 0x1d6670>[color=darkred]
>> >>> _("hi")[/color]
>> 'hi'[/color]
>
> I tried that, but it only works for the first call...[/color]

Setting __builtin__._ to the result of the last calculation is a side effect
of sys.displayhook . Therefore you need to change that too:

Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
[GCC 3.2] on linux2
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import sys
>>> def mydisplayhook(a ):[/color][/color][/color]
.... if a is not None: sys.stdout.writ e("%r\n" % a)
....[color=blue][color=green][color=darkred]
>>> def foo(s): return s[/color][/color][/color]
....[color=blue][color=green][color=darkred]
>>> sys.displayhook = mydisplayhook
>>> import __builtin__
>>> __builtin__._ = foo
>>> _("hi")[/color][/color][/color]
'hi'[color=blue][color=green][color=darkred]
>>> _("hello")[/color][/color][/color]
'hello'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]

Peter

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Pierre Rouleau wrote:
[color=blue]
> Skip Montanaro wrote:
>[color=green]
>> Pierre> I tried to define a _() function when testing with the code
>> Pierre> below but the doctest still fails. The following code is
>> at the
>> Pierre> end of my utstring.py module
>> ...
>>
>> I suspect it's because your dummy _ is not in builtins. Why not just
>> call
>> gettext.install () where you are currently defining _?
>>[/color]
>
> Skip, this looks like the way to go.
>
> I was trying to avoid it because the translation files for these library
> modules are constructed at the application level somewhere else. But I
> was also considering creating a translation library for these module, so
> I guess that is one more incentive to do so...
>[/color]

I implemented the dictionary and use it in the code just fine, but the
doctest still fails :(

My teststr.py module is:

#--[---------------------------------------------------------------
if __name__ == "__main__":
# install gettext for testing this module because of the string
translation
# performed in the code.
import gettext
gettext.install ('impathpl', './locale', unicode=False)
presLan_en = gettext.transla tion('impathpl' , "./locale",
languages=['en'])
presLan_en.inst all()

def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**
[color=blue][color=green][color=darkred]
>>> onOffStr(True)[/color][/color][/color]
'ON'[color=blue][color=green][color=darkred]
>>> onOffStr(False)[/color][/color][/color]
'OFF'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
"""
if isOn:
return _("ON")
else:
return _("OFF")


def _test():
"""_test() perform docstring test"""

import doctest, teststr
return doctest.testmod (teststr)

if __name__ == "__main__":
_test()

#--]-------------------------------------------------


Running the following script shows that the module runs OK:

[Shell buffer started: python.exe]
Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
win32
Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>> import teststr
>>> import gettext
>>> gettext.install ('impathpl', './locale', unicode=False)
>>> presLan_en = gettext.transla tion('impathpl' , "./locale",[/color][/color][/color]
languages=['en'])[color=blue][color=green][color=darkred]
>>> presLan_en.inst all()
>>> teststr.onOffSt r(True)[/color][/color][/color]
'ON'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]


BUT, the doctest fails:

D:\dev\python>t eststr
*************** *************** *************** *************** *****
Failure in example: onOffStr(False)
from line #6 of teststr.onOffSt r
Exception raised:
Traceback (most recent call last):
File "c:\Python23\li b\doctest.py", line 442, in _run_examples_i nner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "teststr.py ", line 23, in onOffStr
return _("OFF")
TypeError: 'str' object is not callable
*************** *************** *************** *************** *****
1 items had failures:
1 of 2 in teststr.onOffSt r
***Test Failed*** 1 failures.


I'm still puzzled...

Pierre

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Peter Otten wrote:
[color=blue]
> Pierre Rouleau wrote:
>
>[color=green]
>>Skip Montanaro wrote:
>>
>>[color=darkred]
>>>If you really want a dummy _() you can also stuff your version into
>>>builtins:
>>>
>>> >>> import __builtin__
>>> >>> def foo(s): return s
>>> ...
>>> >>> __builtin__._ = foo
>>> >>> _
>>> <function foo at 0x1d6670>
>>> >>> _("hi")
>>> 'hi'[/color]
>>
>>I tried that, but it only works for the first call...[/color]
>
>
> Setting __builtin__._ to the result of the last calculation is a side effect
> of sys.displayhook . Therefore you need to change that too:
>
> Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
> [GCC 3.2] on linux2
> Type "help", "copyright" , "credits" or "license" for more information.
>[color=green][color=darkred]
>>>>import sys
>>>>def mydisplayhook(a ):[/color][/color]
>
> ... if a is not None: sys.stdout.writ e("%r\n" % a)
> ...
>[color=green][color=darkred]
>>>>def foo(s): return s[/color][/color]
>
> ...
>[color=green][color=darkred]
>>>>sys.display hook = mydisplayhook
>>>>import __builtin__
>>>>__builtin__ ._ = foo
>>>>_("hi")[/color][/color]
>
> 'hi'
>[color=green][color=darkred]
>>>>_("hello" )[/color][/color]
>
> 'hello'
>[/color]

Thanks Peter, it does work!

[Skip Montanaro]

Re: Problem with mixing doctest with gettext _()


Pierre> BUT, the doctest fails:
...

Looks like Peter Otten's sys.displayhook hack is required for doctest.

Skip

[Skip Montanaro]

Re: Problem with mixing doctest with gettext _()


Pierre> I tried that, but it only works for the first call...

:-)

Pierre> [Shell buffer started: python.exe]
Pierre> Python 2.3.3 (#51, Dec 18 2003, 20:22:39) [MSC v.1200 32 bit (Intel)] on
Pierre> win32
Pierre> Type "help", "copyright" , "credits" or "license" for more information.[color=blue][color=green][color=darkred]
>>>> import __builtin__
>>>> def foo(s): return s[/color][/color][/color]
Pierre> ...[color=blue][color=green][color=darkred]
>>>> __builtin__._ = foo
>>>> _[/color][/color][/color]
Pierre> <function foo at 0x008F98B0>[color=blue][color=green][color=darkred]
>>>> _[/color][/color][/color]
Pierre> <function foo at 0x008F98B0>[color=blue][color=green][color=darkred]
>>>> _('hi')[/color][/color][/color]
Pierre> 'hi'[color=blue][color=green][color=darkred]
>>>> _[/color][/color][/color]
Pierre> 'hi'[color=blue][color=green][color=darkred]
>>>> _('Hello')[/color][/color][/color]
Pierre> Traceback (most recent call last):
Pierre> File "<stdin>", line 1, in ?
Pierre> TypeError: 'str' object is not callable[color=blue][color=green][color=darkred]
>>>>[/color][/color][/color]

That's true. In interactive mode _ is assigned the value of the last
expression evaluated. Try it from a script.

Skip

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Pierre Rouleau wrote:
[color=blue]
> Peter Otten wrote:
>[color=green]
>> Pierre Rouleau wrote:
>>
>>[color=darkred]
>>> Skip Montanaro wrote:
>>>
>>>
>>>> If you really want a dummy _() you can also stuff your version into
>>>> builtins:
>>>>
>>>> >>> import __builtin__
>>>> >>> def foo(s): return s
>>>> ...
>>>> >>> __builtin__._ = foo
>>>> >>> _
>>>> <function foo at 0x1d6670>
>>>> >>> _("hi")
>>>> 'hi'
>>>
>>>
>>> I tried that, but it only works for the first call...[/color]
>>
>>
>>
>> Setting __builtin__._ to the result of the last calculation is a side
>> effect
>> of sys.displayhook . Therefore you need to change that too:
>>
>> Python 2.3.3 (#1, Jan 3 2004, 13:57:08)
>> [GCC 3.2] on linux2
>> Type "help", "copyright" , "credits" or "license" for more information.
>>[color=darkred]
>>>>> import sys
>>>>> def mydisplayhook(a ):[/color]
>>
>>
>> ... if a is not None: sys.stdout.writ e("%r\n" % a)
>> ...
>>[color=darkred]
>>>>> def foo(s): return s[/color]
>>
>>
>> ...
>>[color=darkred]
>>>>> sys.displayhook = mydisplayhook
>>>>> import __builtin__
>>>>> __builtin__._ = foo
>>>>> _("hi")[/color]
>>
>>
>> 'hi'
>>[color=darkred]
>>>>> _("hello")[/color]
>>
>>
>> 'hello'
>>[/color]
>
> Thanks Peter, it does work![/color]


It worked, BUT only for a simple function, it fails if I add a another
simple function:

My updated teststr.py script:

#--[--------------------------------------
def onOffStr(isOn) :
"""Return the "ON" string for True, "OFF" for False.

**Example**
[color=blue][color=green][color=darkred]
>>> onOffStr(True)[/color][/color][/color]
u'ON'[color=blue][color=green][color=darkred]
>>> onOffStr(False)[/color][/color][/color]
u'OFF'[color=blue][color=green][color=darkred]
>>>[/color][/color][/color]
"""
if isOn:
return _(u"ON")
else:
return _(u"OFF")


def inList(longStri ng, stringList) :
"""Return True if one of the string in `stringList` is inside
`longString`.

Also return the list index.
**Example**
[color=blue][color=green][color=darkred]
>>> L = ["**", "/*"]
>>> inList("aksdkaj shd",L)[/color][/color][/color]
(False, 0)[color=blue][color=green][color=darkred]
>>> inList("aksdkaj sh**d",L)[/color][/color][/color]
(True, 0)

"""

theIndex = 0
for theStr in stringList:
if longString.find (theStr) >= 0:
return (True,theIndex)
theIndex +=1
return (False,0)


def _test():
"""_test() perform docstring test"""

import doctest, teststr
return doctest.testmod (teststr)

if __name__ == "__main__":
import sys

def test_displayhoo k(a):
if a is not None: sys.stdout.writ e("%r\n" % a)

def test_translator (aString):
return aString

sys.displayhook = test_displayhoo k

import __builtin__
__builtin__._ = test_translator
_test()

#--]--------------------------------------

Running the test fails:

D:\dev\python>t eststr
*************** *************** *************** *************** *****
Failure in example: inList("aksdkaj shd",L)
from line #6 of teststr.inList
Exception raised:
Traceback (most recent call last):
File "c:\Python23\li b\doctest.py", line 442, in _run_examples_i nner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "D:\dev\python\ teststr.py", line 50, in test_displayhoo k
if a is not None: sys.stdout.writ e("%r\n" % a)
TypeError: not all arguments converted during string formatting
*************** *************** *************** *************** *****
Failure in example: inList("aksdkaj sh**d",L)
from line #8 of teststr.inList
Exception raised:
Traceback (most recent call last):
File "c:\Python23\li b\doctest.py", line 442, in _run_examples_i nner
compileflags, 1) in globs
File "<string>", line 1, in ?
File "D:\dev\python\ teststr.py", line 50, in test_displayhoo k
if a is not None: sys.stdout.writ e("%r\n" % a)
TypeError: not all arguments converted during string formatting
*************** *************** *************** *************** *****
1 items had failures:
2 of 3 in teststr.inList
***Test Failed*** 2 failures.

#------------------------------------------

So far, I don't have a solution for writing internationaliz ed Python
that support doctest. Surely, I am not the first one trying to do that...


Pierre

[Peter Otten]

Re: Problem with mixing doctest with gettext _()

Pierre Rouleau wrote:
[color=blue]
> It worked, BUT only for a simple function, it fails if I add a another
> simple function:[/color]

Haven't read your post completely, but judging from the error message, it's
just a minor glitch in mydisplayhook() . Try the following instead:

[color=blue][color=green][color=darkred]
>>> def mydisplayhook(a ):[/color][/color][/color]
.... if a is not None:
.... sys.stdout.writ e("%r\n" % (a,))
....

That should be able to cope with tuples as commandline results.

Peter

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

Peter Otten wrote:[color=blue]
> Pierre Rouleau wrote:
>
>[color=green]
>>It worked, BUT only for a simple function, it fails if I add a another
>>simple function:[/color]
>
>
> Haven't read your post completely, but judging from the error message, it's
> just a minor glitch in mydisplayhook() . Try the following instead:
>
>
>[color=green][color=darkred]
>>>>def mydisplayhook(a ):[/color][/color]
>
> ... if a is not None:
> ... sys.stdout.writ e("%r\n" % (a,))
> ...
>
> That should be able to cope with tuples as commandline results.
>
> Peter[/color]


You're right! It does work. I must admit, that I don't see why though.
(a,) makes a tuple out of the `a` argument. Does the %r conversion
require a tuple?

[Peter Otten]

Re: Problem with mixing doctest with gettext _()

Pierre Rouleau wrote:
[color=blue]
> Peter Otten wrote:[color=green]
>> Pierre Rouleau wrote:
>>
>>[color=darkred]
>>>It worked, BUT only for a simple function, it fails if I add a another
>>>simple function:[/color]
>>
>>
>> Haven't read your post completely, but judging from the error message,
>> it's just a minor glitch in mydisplayhook() . Try the following instead:
>>
>>
>>[color=darkred]
>>>>>def mydisplayhook(a ):[/color]
>>
>> ... if a is not None:
>> ... sys.stdout.writ e("%r\n" % (a,))
>> ...
>>
>> That should be able to cope with tuples as commandline results.
>>
>> Peter[/color]
>
>
> You're right! It does work. I must admit, that I don't see why though.
> (a,) makes a tuple out of the `a` argument. Does the %r conversion
> require a tuple?[/color]

The formatting operator behaves differently depending on whether the
righthand argument is a tuple or something else.

formatstr % tuple

ensures that there is a corresponding format expression - e.g. "%s" or "%d"
- for every item in the tuple, whereas

formatstr % nontuple

expects exactly one format expression. Therefore

"%r\n" % a

raises an exception when a is a tuple with more or less than one item and
wrongly prints the item's representation instead of the tuple's for
one-tuples.
Wrapping it like so (a,) fixes the problem because now we have always a
tuple with one item - where this item is sometimes a tuple.

An alternative approach would be to ensure that the righthand side is always
a nontuple:

"%s\n" % repr(a)

Peter

[Pierre Rouleau]

Re: Problem with mixing doctest with gettext _()

[color=blue][color=green]
>>
>>
>>You're right! It does work. I must admit, that I don't see why though.
>> (a,) makes a tuple out of the `a` argument. Does the %r conversion
>>require a tuple?[/color]
>
>
> The formatting operator behaves differently depending on whether the
> righthand argument is a tuple or something else.
>
> formatstr % tuple
>
> ensures that there is a corresponding format expression - e.g. "%s" or "%d"
> - for every item in the tuple, whereas
>
> formatstr % nontuple
>
> expects exactly one format expression. Therefore
>
> "%r\n" % a
>
> raises an exception when a is a tuple with more or less than one item and
> wrongly prints the item's representation instead of the tuple's for
> one-tuples.
> Wrapping it like so (a,) fixes the problem because now we have always a
> tuple with one item - where this item is sometimes a tuple.
>
> An alternative approach would be to ensure that the righthand side is always
> a nontuple:
>
> "%s\n" % repr(a)
>[/color]

Thanks for this clear explanation!

Pierre