I have a new enhancement to pyparsing that doubles the parse speed (using a
technique called "packrat parsing"), but which is not suitable for all
parsers, specifically those that have complex parse actions. I don't want
to just enable this feature by default - I think there is too much risk of
it breaking existing code. Actually, I think the chance is <10%, but I
think I'm better off leaving the feature dormant unless explicitly enabled,
so that it doesn't take someone by surprise when they install the new
version.
The alternatives I've come up with for the user to enable this packrat parse
mode are:
1. Add a staticmethod enablePackrat() to the pyparsing ParserElement class,
to modify the ParserElement defintion of the internal (non-packrat) parse()
method. This method essentially runs code looking like:
ParserElement.p arse,ParserElem ent.originalPar se = \
ParserElement.p ackratParse,Par serElement.pars e
where the packratParse method is a memoizing wrapper around calls to
originalParse. This technique requires the caller to invoke
ParserElement.e nablePackrat() after having imported pyparsing.
This renaming trick actually breaks when using psyco, if psyco has compiled
the pyparsing methods before calling enablePackrat() . You have to import
pyparsing, call enablePackrat() , then call psyco.full(), or whatever psyco
commands to cache compiled versions of the pyparsing function code.
2. Implement some sort of "onImport" hook to allow the calling routine to
use a specialized import statement, something like "from pyparsing import
packratParsing" that would do the same as the above enablePackrat() call,
but perhaps in a more Pythonic-looking manner. (I was inspired along these
lines from Jim Hugunin's IronPython talk, in which he used import hooks to
modify the behavior of the base Python string class.) I've looked a bit at
imp, imputil, and ihooks, and these all seem to be modifiers to the import
mechanism as called from the importing module. What import hooks are
accessible from within the module being imported? Is there any way to see
which form of import was used, and to access the arguments of the import
command as invoked from the calling module?
I was hoping that by hooking into the import command, I could make this
mode-setting command more atomic with the actual import of pyparsing's class
and function definitions, to avoid problems like the psyco issue.
Plus I'm open to other suggestions as well.
Thanks,
-- Paul
technique called "packrat parsing"), but which is not suitable for all
parsers, specifically those that have complex parse actions. I don't want
to just enable this feature by default - I think there is too much risk of
it breaking existing code. Actually, I think the chance is <10%, but I
think I'm better off leaving the feature dormant unless explicitly enabled,
so that it doesn't take someone by surprise when they install the new
version.
The alternatives I've come up with for the user to enable this packrat parse
mode are:
1. Add a staticmethod enablePackrat() to the pyparsing ParserElement class,
to modify the ParserElement defintion of the internal (non-packrat) parse()
method. This method essentially runs code looking like:
ParserElement.p arse,ParserElem ent.originalPar se = \
ParserElement.p ackratParse,Par serElement.pars e
where the packratParse method is a memoizing wrapper around calls to
originalParse. This technique requires the caller to invoke
ParserElement.e nablePackrat() after having imported pyparsing.
This renaming trick actually breaks when using psyco, if psyco has compiled
the pyparsing methods before calling enablePackrat() . You have to import
pyparsing, call enablePackrat() , then call psyco.full(), or whatever psyco
commands to cache compiled versions of the pyparsing function code.
2. Implement some sort of "onImport" hook to allow the calling routine to
use a specialized import statement, something like "from pyparsing import
packratParsing" that would do the same as the above enablePackrat() call,
but perhaps in a more Pythonic-looking manner. (I was inspired along these
lines from Jim Hugunin's IronPython talk, in which he used import hooks to
modify the behavior of the base Python string class.) I've looked a bit at
imp, imputil, and ihooks, and these all seem to be modifiers to the import
mechanism as called from the importing module. What import hooks are
accessible from within the module being imported? Is there any way to see
which form of import was used, and to access the arguments of the import
command as invoked from the calling module?
I was hoping that by hooking into the import command, I could make this
mode-setting command more atomic with the actual import of pyparsing's class
and function definitions, to avoid problems like the psyco issue.
Plus I'm open to other suggestions as well.
Thanks,
-- Paul
Comment