Using sax libxml2 html parser

[cesar.ortiz@gmail.com]

Hi all,

I have created an example using libxml2 based in the code that appears
in http://xmlsoft.org/python.html.
My example processes an enough amount of html files to see that the
memory consumption rises till the process ends (I check it with the
'top' command).

I don´t know if I am forgetting something in the code, as I have not
been able to find any example on the web.

Thanks in advance, Cesar

Note: I have also tried to put the cleanup functions inside the 'for'
loop.

*************** *************** **********] The Code
[*************** *************** **********

#!/usr/bin/python -u
import libxml2

#------------------------------------------------------------------------------


# Memory debug specific
libxml2.debugMe mory(1)

#------------------------------------------------------------------------------

class callback:
def startDocument(s elf):
print "."

def endDocument(sel f):
pass

def startElement(se lf, tag, attrs):
pass

def endElement(self , tag):
pass

def characters(self , data):
pass

def warning(self, msg):
pass

def error(self, msg):
pass

def fatalError(self , msg):
pass

#------------------------------------------------------------------------------
#------------------------------------------------------------------------------

import os
import sys

programName = os.path.basenam e(sys.argv[0])

if len(sys.argv) != 2:
print "Use: %s <dir html files>" % programName
sys.exit(1)

inputPath = sys.argv[1]

if not os.path.exists (inputPath):
print "Error: directory does not exist"
sys.exit(1)

inputFileNames = []
dirContent = os.listdir(inpu tPath)
for fichero in dirContent:
extension1=fich ero.rfind(".htm ")
extension2=fich ero.rfind(".htm l")
dot = fichero.rfind(" .")
extension = max(extension1, extension2)
if extension != -1 and extension == dot:
inputFileNames. append (fichero)

if len(inputFileNa mes) == 0:
print "Error: no input files"
sys.exit(1)


handler = callback()
NUM_ITERS = 5
for i in range(NUM_ITERS ):
for inputFileName in inputFileNames:
print inputFileName
inputFilePath = inputPath + inputFileName
f = open(inputFileP ath)
data = f.read()
f.close()

ctxt = libxml2.htmlCre atePushParser(h andler, "", 0, inputFileName)

ctxt.htmlParseC hunk(data, len(data), 1)
ctxt = None


# Memory debug specific
libxml2.cleanup Parser()
if libxml2.debugMe mory(1) == 0:
print "OK"
else:
print "Memory leak %d bytes" % (libxml2.debugM emory(1))
libxml2.dumpMem ory()

# Other cleanup functions
#libxml2.cleanu pCharEncodingHa ndlers()
#libxml2.cleanu pEncodingAliase s()
#libxml2.cleanu pGlobals()
#libxml2.cleanu pInputCallbacks ()
#libxml2.cleanu pOutputCallback s()
#libxml2.cleanu pPredefinedEnti ties()

[Stefan Behnel]

Re: Using sax libxml2 html parser

cesar.ortiz@gma il.com wrote:

I have created an example using libxml2 based in the code that appears
in http://xmlsoft.org/python.html.
My example processes an enough amount of html files to see that the
memory consumption rises till the process ends (I check it with the
'top' command).

Try the lxml binding for libxml2. It relieves you from having to do your own
memory management and makes using that library plain easy.

http://codespeak.net/lxml/

Stefan