why am I getting a segmentation fault?

**Paul McGuire** · Jul 18 '05, 08:31 PM

Re: why am I getting a segmentation fault?

"Jay donnell" <jaydonnell@gma il.com> wrote in message
news:1106330476 .419418.46920@f 14g2000cwb.goog legroups.com...[color=blue]
> I have a short multi-threaded script that checks web images to make
> sure they are still there. I get a segmentation fault everytime I run
> it and I can't figure out why. Writing threaded scripts is new to me so
> I may be doing something wrong that should be obvious :(
>[/color]
<snip>

Here is a code excerpt from your link (the main routine, omits the class
definition for ImageChecker, which extends threading.Threa d):

db = MySQLdb.connect (host="localhos t", user="xxx", passwd="xxx", db="xxx")
cursor = db.cursor()
query = "select * from item order by rand() limit 0, 100"
#query = "select * from item"
cursor.execute( query)
result = cursor.fetchall ()

maxThreads = 5

for r in result:
while(threading .activeCount() > maxThreads):
pass
flag='good'
#pass
#print str(r[0]) + ', ' + str(r[7])
tmp = r[7].split('/')
filename = tmp[-1]
#print 'filename ' + filename

filename = '/tmp/'+filename

threadList = []

#r[7] is the url of the image
#r[0] is the id for the row
imageChecker = ImageChecker(r[7], filename, r[0])
imageChecker.st art()
threadList.appe nd(imageChecker )

----------------------------------------------
1. What happens after you fall out of the loop "for r in result"? Shouldn't
you wait for the remaining threads to finish? Perhaps you need another
busy-loop to wait for the threads to finish, something like
while threading.activ eCount() > 0: pass
2. Is this the best way to busy-wait? What about some kind of
thread.join()? At least throw a sleep call in there or something, or this
loop will churn and churn, diverting CPU from your threads that are actually
trying to do some real work.
3. I find it easier to work with named variables than numeric subscripts.
At the top of your for loop, try something like:
id,height,width ,numBytes,whate ver,slkdjf1,slk djf2,url = r
This way you have much more meaningful names than r[0] and r[7], which you
later have to comment to explain whats going on!
4. filename=r[7].split('/')[-1] is not terribly portable. See if there is a
standard module for parsing filespecs (I'll bet there is).

-- Paul

**Jeff Shannon** · Jul 18 '05, 08:31 PM

Re: why am I getting a segmentation fault?

Paul McGuire wrote:
[color=blue]
> 4. filename=r[7].split('/')[-1] is not terribly portable. See if there is a
> standard module for parsing filespecs (I'll bet there is).[/color]

Indeed there is -- os.path. In particular, os.path.basenam e() seems
to do exactly that snippet is intending, in a much more robust (and
readable) fashion.

Jeff Shannon
Technician/Programmer
Credit International

**John Machin** · Jul 18 '05, 08:31 PM

Re: why am I getting a segmentation fault?

Jay donnell wrote:[color=blue]
> I have a short multi-threaded script that checks web images to make
> sure they are still there. I get a segmentation fault everytime I run
> it and I can't figure out why. Writing threaded scripts is new to me[/color]
so[color=blue]
> I may be doing something wrong that should be obvious :(
>[/color]

def run(self):
try:
self.site = urllib.urlopen( self.url)
self.f=open(sel f.filename, 'w')
self.im = Image.open(self .filename)
self.size = self.im.size
self.flag = 'yes'
self.q = "yadda yadda"

That's SIX names that don't need to be attributes of the object; they
are used _only_ in this method, so they can be local to the method,
saving you a whole lot of typing "self." and saving puzzlement &
careful scrutiny by those trying to read your code.

Back to your real problem: does it work when you set maxThreads to 1?
did it work before you added the threading code? what does your
debugger tell you about the location of the seg fault?

MOST IMPORTANTLY, sort this mess out:

self.q = "update item set goodImage = '" + self.flag + "' where
productId='" + str(self.id) + "'"
print self.q
self.cursor.exe cute(query) ############### ############### #

### "query" is a global variable[YUK!] (see below) which isn't going to
do what you want. Looks like you meant "self.q".

self.db.close()

db = MySQLdb.connect (host="localhos t", user="xxx", passwd="xxx",
db="xxx")
cursor = db.cursor()
query = "select * from item order by rand() limit 0, 100"

### Have you looked in your database to see if the script has actually
updated item.goodImage? Do you have a test plan?

**Jay donnell** · Jul 18 '05, 08:32 PM

Re: why am I getting a segmentation fault?

Thank you.

I made all the changes you recommended and everything seems to be
working.

**Jay donnell** · Jul 18 '05, 08:32 PM

Re: why am I getting a segmentation fault?

>### Have you looked in your database to see if the >script has
actually[color=blue]
>updated item.goodImage? Do you have a test plan?[/color]

Thank you for the help. Sorry for the messy code. I was under time
constraints. I had class, and I was rushing to get this working before
class. I should waited a day and read over it before I asked. Sorry
again.

**John Machin** · Jul 18 '05, 08:32 PM

Re: why am I getting a segmentation fault?

Jay donnell wrote:[color=blue][color=green]
> >### Have you looked in your database to see if the >script has[/color]
> actually[color=green]
> >updated item.goodImage? Do you have a test plan?[/color]
>
> Thank you for the help. Sorry for the messy code. I was under time
> constraints. I had class, and I was rushing to get this working[/color]
before[color=blue]
> class. I should waited a day and read over it before I asked. Sorry
> again.[/color]

Apologise to _yourself_ for the messy code. If 'query' had been safely
tucked away as a local variable instead of a global, that problem
(typing 'query' instead of 'self.q') would have caused an exception on
the first time around.

A few points: Yes, don't rush. I've-forgotten-whom wrote something like
"Don't program standing up". Good advice.

Build things a piece at a time. Build your tests at the same time or
earlier. In this case, a stand-alone method or function that checked
that one file was OK, would have been a reasonable place to start. Then
add the code to query the db. Then add the threading stuff, gingerly.
If you build and test incrementally, then a segfault or other disaster
is highly likely to have been caused by the latest addition.

AND SOME MORE CRUFT:
def __init__(self, url, filename, id):
self.t = time.time() <<<<<<<<<====== = never used again
threading.Threa d.__init__(self )
self.db = MySQLdb.connect (host="localhos t", user="xxx",
passwd="xxx", db="xxx")
# create a cursor
self.cursor = db.cursor() <<<=== should be self.db.cursor( )
#### picks up the *GLOBAL* 'db'
self.url = url
self.filename = filename
self.id = id
=============== ============
threadList = []
[snip]
threadList.appe nd(imageChecker )
===>>>> that doesn't achieve much!
=============== =============

N.B. You still have two problems: (1) Your script as you said now
"seems to work". That doesn't sound like you have a test plan. (2) You
have shuffled your code around and the segfault went away; i.e. you
waved a dead chicken and the volcano stopped erupting. Most of the
changes suggested by others and myself were of a style/clarity nature.
The self.q/query thing would have caused a select instead an update;
hardly segfault territory. I wouldn't expect that busy-wait loop to
cause a segfault. You still don't know what caused the segfault. That
means you don't know how to avoid it in the future. You are still
living in the shadow of the volcano. Will the chicken trick work next
time?

Looking forward to the next episode,
John

why am I getting a segmentation fault?

why am I getting a segmentation fault?

Comment

Comment

Comment

Comment

Comment

Comment