Massive unit test vs MySQL

**Stephan Schaumann** · Jul 18 '05, 02:17 AM

Re: Massive unit test vs MySQL

"Richard Wesley" <hawkfish@trust edmedianetworks .com> schrieb:[color=blue]
> I don't know if this is really a Python question of a MySQL question,
> but i am hopen that there is enough overlap that someone can help me ;-)
>
> I have a unit test suite for our server that loads a clean database
> image for many of the tests. I use
>
> p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
> p.write(sql_com mands)[/color]

Do you p.close() when you are finished? Otherwise the connection opend by
mysql will stay open.
[color=blue]
> to load the database.
>
> While the tests are running, I find that mysql has a growing number of
> "Sleeping" threads in the 'show processlist' table. Towards the end of
> the test suite, the database loads start to fail with a MySQL error of
> "ERROR 1040: Too many connections".
>
> So my question would be, is there any way to shut down the child process
> so that MySQL notices and cleans up after itself before it runs out of
> threads?[/color]

**Neil Padgen** · Jul 18 '05, 02:18 AM

Re: Massive unit test vs MySQL

On Thursday 04 September 2003 20:35, Richard Wesley wrote:
[color=blue]
> I have a unit test suite for our server that loads a clean database
> image for many of the tests. I use
>
> p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
> p.write(sql_com mands)[/color]

Your pipe is still open at this point, and therefore you have a
connection to the database open.
[color=blue]
> So my question would be, is there any way to shut down the child
> process so that MySQL notices and cleans up after itself before it
> runs out of threads?[/color]

p.close()

-- Neil

**Richard Wesley** · Jul 18 '05, 02:18 AM

Re: Massive unit test vs MySQL

In article <bja8ls$k1j$1@n ntp0.reith.bbc. co.uk>,
Neil Padgen <neil.padgen@mo n.bbc.co.uk> wrote:
[color=blue]
> On Thursday 04 September 2003 20:35, Richard Wesley wrote:
>[color=green]
> > I have a unit test suite for our server that loads a clean database
> > image for many of the tests. I use
> >
> > p = os.popen('mysql -u uid -ppassword mydatabase', 'w')
> > p.write(sql_com mands)[/color]
>
> Your pipe is still open at this point, and therefore you have a
> connection to the database open.
>[color=green]
> > So my question would be, is there any way to shut down the child
> > process so that MySQL notices and cleans up after itself before it
> > runs out of threads?[/color]
>
> p.close()[/color]

Yeah, I tried this, but it had no effect.

The odd thing is that the process list cleans up _immediately_ when the
script terminates. Maybe this is some sort of gc problem?

--

- rmgw

<http://www.trustedmedi anetworks.com/>

----------------------------------------------------------------------------
Richard Wesley Trusted Media Networks, Inc.

"You're confusing boredom with motivation."
- Sherman in "Sherman's Lagoon"

**Richard Wesley** · Jul 18 '05, 02:27 AM

Re: Massive unit test vs MySQL

In article <hawkfish-7B21B7.09501410 092003@spectato r.sj.sys.us.xo. net>,
Richard Wesley <hawkfish@trust edmedianetworks .com> wrote:
[color=blue]
> In article <bjkdoa$3ea$1@n ntp0.reith.bbc. co.uk>,
> Neil Padgen <neil.padgen@mo n.bbc.co.uk> wrote:
>[color=green]
> > On Monday 08 September 2003 22:28, Richard Wesley wrote:
> >[color=darkred]
> > > In article <bji5h7$bqo$1@n ntp0.reith.bbc. co.uk>,
> > > Neil Padgen <neil.padgen@mo n.bbc.co.uk> wrote:
> > >[/color]
> >[color=darkred]
> > > I doubt it - it is generated by mysqldump. And there is no problem
> > > from the command line.
> > >[/color]
> >
> > I use exactly the same approach to set up my unit tests, but I have a
> > different way to get the mysqldump data into the database:
> >
> > os.system('mysq l database_name < dumpfile')[/color]
>
> Sadly, this did not work, nor did garbage collection (gc.collect()).
>
> For some reason, mysql is holding onto a whole lot of connections from
> my script until the script terminates.[/color]

OK, I'm an idiot. It had nothing to do with the reload script. The
connections were from the TestCase subclass I have that instantiates a
connection management object. It appears that the unittest module does
not dispose of the objects it instantiates until the end of the run, so
you need to REALLY clean up in the tearDown method. I discovered this
by making a persistent connection for the reload operation and noticing
that the number of connections still increased.

Thanks to all who responded.

--
Best regards,

Richard Wesley
Co-President, Electric Fish, Inc.
<http://www.electricfis h.com/>
(v) +1-206-493-1690x210
(f) +1-206-493-1697
(h) +1-206-632-4536
(m) +1-206-409-4536

**Neil Padgen** · Jul 18 '05, 02:29 AM

Re: Massive unit test vs MySQL

>>>>> "Richard" == Richard Wesley <hawkfish@trust edmedianetworks .com> writes:

Richard> OK, I'm an idiot. It had nothing to do with the reload
Richard> script. The connections were from the TestCase subclass
Richard> I have that instantiates a connection management object.
Richard> It appears that the unittest module does not dispose of
Richard> the objects it instantiates until the end of the run, so
Richard> you need to REALLY clean up in the tearDown method. I
Richard> discovered this by making a persistent connection for the
Richard> reload operation and noticing that the number of
Richard> connections still increased.

Doh! I should have thought of this, as I've had exactly the same
problem in the past.

Thanks for posting your solution.

-- Neil

Massive unit test vs MySQL

Massive unit test vs MySQL

Comment

Comment

Comment

Comment

Comment