no space left on device

[Nigel J. Andrews]

Re: no space left on device


On Fri, 9 Jan 2004, Aurangzeb M. Agha wrote:[color=blue]
>
> Scott -- I'm at 93% disk usage:
>
> [postgres - DB]$ df -m .
> Filesystem 1M-blocks Used Available Use% Mounted on
> - 63328 55308 4803 93% /
>[/color]


BTW, don't do -m with df it confuses us old folk, as you probably noticed from
the responses.

:)


--
Nigel


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to majordomo@postg resql.org)

[Aurangzeb M. Agha]

Re: no space left on device

Right! Thus my quandry.

Re inodes, how can I check this? But why would this be? Is Postgres
sucking up inodes just sitting there as a read-only DB?

AMA

On Fri, 9 Jan 2004, Joshua D. Drake wrote:

:Hello,
:
: Wait... from the df you provided you have space left on the device:
:
:postgres - DB]$ df -m .
:Filesystem 1M-blocks Used Available Use% Mounted on
:- 63328 55308 4803 93% /
:
:Perhaps you are out of inodes?
:
:Sincerely,
:
:Josuha D. Drake
:
:
:
:
:Aurangzeb M. Agha wrote:
:
:>No, I've not added any new DB's. In fact, what's puzzling is that this DB
:>has been running without issue (except for one server restart) for the
:>last nine months. Now, all of a sudden, with no DB changes, additions,
:>etc... I'm getting this problem.
:>
:>Do you suggest that I still run a vacuumdb?
:>
:>Rgs,
:>
:>Zeb
:>
:>
:>
:>On Fri, 9 Jan 2004, Joshua D. Drake wrote:
:>
:>:Aurangzeb M. Agha wrote:
:>:
:>:>I've not run a vacuum in quite some time, and that's because I've only
:>:>been doing reads from this DB. I was under the impression that I should
:>:>run vacuum when tables are heavily modified:
:>:>
:>:>
:>:
:>:That would be accurate. Did you recently add a second database?
:>:
:>:Sincerely,
:>:
:>:Joshua D. Drake
:>:
:>:
:>:>http://www.postgresql.org/docs/aw_pg...k/node110.html
:>:>
:>:>I guess I must have been mistaken?
:>:>
:>:>I'm looking through the docs now, but am having trouble finding this: how
:>:>can I vacuum the entire DB at once?
:>:>
:>:>Thx,
:>:>
:>:>Zeb
:>:>
:>:>On Fri, 9 Jan 2004, Joshua D. Drake wrote:
:>:>
:>:>:
:>:>:>
:>:>:>Note that the pg_xlog dir is huge! Here's its contents:
:>:>:>
:>:>:>[postgres - DB/pg_xlog]$ ls -al
:>:>:>total 32816
:>:>:>drwx------ 2 postgres admin 4096 Mar 29 2003 .
:>:>:>drwx------ 6 postgres admin 4096 Jan 9 15:04 ..
:>:>:>-rwx------ 1 postgres admin 16777216 Jan 9 15:09 000000000000000 1
:>:>:>-rwx------ 1 postgres admin 16777216 Mar 29 2003 000000000000000 2
:>:>:>
:>:>:>What are these files, and what can I do to resolve this issue?
:>:>:>
:>:>:>
:>:>:The are check_point files. You need them. Have you ran a vacuum recently?
:>:>:
:>:>:Sincerely ,
:>:>:
:>:>:Joshua D. Drake
:>:>:
:>:>:
:>:>:
:>:>:>Thx,
:>:>:>
:>:>:>Zeb
:>:>:>
:>:>:>
:>:>:>--
:>:>:>DEBUG: statistics collector process (pid 2523) exited with exit code 1
:>:>:>PGSTAT: Error closing temp stats file
:>:>:>PGSTAT: /usr/local/G101/App/DB/./global/pgstat.tmp.7823 : No space left on
:>:>:>device
:>:>:>PGSTAT: AbDEBUG: statistics collector process (pid 2979) exited with exit
:>:>:>code
:>:>:>1
:>:>:>FATAL 2: write of clog file 43, offset 188416 failed: No space left on
:>:>:>device
:>:>:>DEBUG: server process (pid 3741) exited with exit code 2
:>:>:>DEBUG: terminating any other active server processes
:>:>:>NOTICE: Message from PostgreSQL backend:
:>:>:> The Postmaster has informed me that some other backend
:>:>:> died abnormally and possibly corrupted shared memory.
:>:>:> I have rolled back the current transaction and am
:>:>:> going to terminate your database system connection and exit.
:>:>:> Please reconnect to the database system and repeat your query.
:>:>:>NOTICE: Message from PostgreSQL backend:
:>:>:> The Postmaster has informed me that some other backend
:>:>:> died abnormally and possibly corrupted shared memory.
:>:>:> I have rolled back the current transaction and am
:>:>:> going to terminate your database system connection and exit.
:>:>:> Please reconnect to the database system and repeat your query.
:>:>:>NOTICE: Message from PostgreSQL backend:
:>:>:> The Postmaster has informed me that some other backend
:>:>:> died abnormally and possibly corrupted shared memory.
:>:>:> I have rolled back the current transaction and am
:>:>:> going to terminate your database system connection and exit.
:>:>:> Please reconnect to the database system and repeat your query.
:>:>:>DEBUG: all server processes terminated; reinitializing shared memory and
:>:>:>semaph
:>:>:>ores
:>:>:>DEBUG: database system was interrupted at 2004-01-09 05:22:52 EST
:>:>:>DEBUG: checkpoint record is at 0/138CFD4
:>:>:>DEBUG: redo record is at 0/138CFD4; undo record is at 0/0; shutdown FALSE
:>:>:>DEBUG: next transaction id: 45811837; next oid: 65205
:>:>:>DEBUG: database system was not properly shut down; automatic recovery in
:>:>:>progre
:>:>:>ss
:>:>:>DEBUG: redo starts at 0/138D014
:>:>:>FATAL 2: write of clog file 43, offset 188416 failed: No space left on
:>:>:>device
:>:>:>DEBUG: startup process (pid 3785) exited with exit code 2
:>:>:>DEBUG: aborting startup due to startup process failure
:>:>:>
:>:>:>---------------------------(end of broadcast)---------------------------
:>:>:>TIP 6: Have you searched our list archives?
:>:>:>
:>:>:> http://archives.postgresql.org
:>:>:>
:>:>:>
:>:>:
:>:>:
:>:>:
:>:>
:>:>
:>:
:>:
:>:
:>
:>
:>
:
:
:

--
Aurangzeb M. Agha | Email : ama@mltp.com
| Home : +1 413 586.4863
| Pager : +1 413 785.7568
| : 4137857568@myai rmail.com
73 Bridge St. #15 | Mobile: <coming soon>
Northampton, MA 01060 | e-Fax : +1 978 246.0770
USA | PGP id: <coming soon>


---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Tom Lane]

Re: no space left on device

"scott.marl owe" <scott.marlowe@ ihs.com> writes:[color=blue][color=green]
>> [postgres - DB]$ df -m .
>> Filesystem 1M-blocks Used Available Use% Mounted on
>> - 63328 55308 4803 93% /[/color][/color]
[color=blue]
> Do you have root access to it? if so, set the reserved space for root to
> be 0%, and then try vacuuming. Vacuuming requires some free space, and
> since you're pretty much out, it isn't gonna be able to complete.[/color]

Look again --- it's showing free space in MB not KB. He's got 4.8GB
free. (Although that might be free-from-root's-point-of-view, rather
than what an unprivileged user can use ...)

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Dennis Björklund]

Re: no space left on device

On Fri, 9 Jan 2004, Aurangzeb M. Agha wrote:
[color=blue]
> DEBUG: statistics collector process (pid 2523) exited with exit code 1
> PGSTAT: Error closing temp stats file
> PGSTAT: /usr/local/G101/App/DB/./global/pgstat.tmp.7823 : No space left on
> device[/color]

To me it does not sound strange that the database is growing when the stat
collector updates the tables with statistics. And since there are updates
it would have been good to have vacuumed avery once in a while.

I don't know the internals of pg as well as some of the other people who
have answered, but as far as I know the stat collector is not special in
any way but is updating the stat tables.

--
/Dennis Björklund


---------------------------(end of broadcast)---------------------------
TIP 9: the planner will ignore your desire to choose an index scan if your
joining column's datatypes do not match

[Tom Lane]

Re: no space left on device

"Aurangzeb M. Agha" <ama-list@mltp.com> writes:[color=blue]
> Re inodes, how can I check this?[/color]

"df -i" should help.
[color=blue]
> But why would this be? Is Postgres
> sucking up inodes just sitting there as a read-only DB?[/color]

I think you have missed the point here. Postgres is using 0.1 percent
of your disk; whatever is eating disk space or inodes is somewhere in
the 92.9% of the disk that you have not told us about. You are focusing
on killing the messenger instead of finding the true source of the
problem.

You should also check into the per-user-quota possibility.

regards, tom lane

---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[scott.marlowe]

Re: no space left on device

On Fri, 9 Jan 2004, Tom Lane wrote:
[color=blue]
> "scott.marl owe" <scott.marlowe@ ihs.com> writes:[color=green][color=darkred]
> >> [postgres - DB]$ df -m .
> >> Filesystem 1M-blocks Used Available Use% Mounted on
> >> - 63328 55308 4803 93% /[/color][/color]
>[color=green]
> > Do you have root access to it? if so, set the reserved space for root to
> > be 0%, and then try vacuuming. Vacuuming requires some free space, and
> > since you're pretty much out, it isn't gonna be able to complete.[/color]
>
> Look again --- it's showing free space in MB not KB. He's got 4.8GB
> free. (Although that might be free-from-root's-point-of-view, rather
> than what an unprivileged user can use ...)[/color]

Good catch. I'm so used to using raw df output...

Yeah, it looks like root's reserved space is getting him to me, but since
it's the root partition, it's possible it's out of inodes as well.

Aurangzeb, try running df -i to see how many inodes you have left...


---------------------------(end of broadcast)---------------------------
TIP 1: subscribe and unsubscribe commands go to majordomo@postg resql.org

[scott.marlowe]

Re: no space left on device

On Fri, 9 Jan 2004, Aurangzeb M. Agha wrote:
[color=blue]
> Right! Thus my quandry.
>
> Re inodes, how can I check this? But why would this be? Is Postgres
> sucking up inodes just sitting there as a read-only DB?[/color]

If you are out of inodes, I seriously doubt it is Postgresql's fault, as
you seem to be running everything on the root partition here, it could be
any other process more likely than postgresql is using all the inodes.
Basically, when you make a lot of small files you can run out of inodes.
Since postgresql tends to make a few rather large files, it's usually not
a concern.

df -i shows inode usage.

On linux, you can change the % reserved for root to 1% with tune2fs:

tune2fs -m 1



---------------------------(end of broadcast)---------------------------
TIP 6: Have you searched our list archives?

http://archives.postgresql.org

[Mark Kirkwood]

Re: no space left on device

I would suspect some *other* service is using the 4G for transient
storage every now and again, and it just so happens that Pg is getting
tripped up.

What else does this machine run ?

regards

Mark



Nigel J. Andrews wrote:
[color=blue]
>On Fri, 9 Jan 2004, Aurangzeb M. Agha wrote:
>
>
>[color=green]
>>Here's the output of "df -m":
>>
>>[postgres - DB]$ df -m .
>>Filesystem 1M-blocks Used Available Use% Mounted on
>>- 63328 55308 4803 93% /
>>
>>
>>[/color]
>
>But your du, below, of the postgres data directory shows 53MB in use. That's an
>order of magnitude smaller than the 55GB the above appears to be saying is used
>in the db.
>
>
>
>[/color]


---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to majordomo@postg resql.org)

[Nigel J. Andrews]

Re: no space left on device

On Fri, 9 Jan 2004, scott.marlowe wrote:
[color=blue]
> On Fri, 9 Jan 2004, Aurangzeb M. Agha wrote:
>[color=green]
> > Right! Thus my quandry.
> >
> > Re inodes, how can I check this? But why would this be? Is Postgres
> > sucking up inodes just sitting there as a read-only DB?[/color]
>
> If you are out of inodes, I seriously doubt it is Postgresql's fault, as
> you seem to be running everything on the root partition here, it could be
> any other process more likely than postgresql is using all the inodes.
> Basically, when you make a lot of small files you can run out of inodes.[/color]

And a common culprit is whatever is being used for usenet caching/serving...or
ordinary mail which is just accumulating in /var/mail (or whereever).

[color=blue]
> Since postgresql tends to make a few rather large files, it's usually not
> a concern.
>
> df -i shows inode usage.
>
> On linux, you can change the % reserved for root to 1% with tune2fs:
>
> tune2fs -m 1[/color]

--
Nigel J. Andrews


---------------------------(end of broadcast)---------------------------
TIP 4: Don't 'kill -9' the postmaster

[Nigel J. Andrews]

Re: no space left on device


On Sat, 10 Jan 2004, Nigel J. Andrews wrote:[color=blue]
> And a common culprit is whatever is being used for usenet caching/serving...or
> ordinary mail which is just accumulating in /var/mail (or whereever).[/color]

Sheesh. Did I really put ordinary mailbox mail in the uses up inodes category?
I should taken out and whi....errrr... on the other hand better not might be too
exciting for some and spark off a whole new xxx web site.


Nigel Andrews


---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings

[Bruno Wolff III]

Re: no space left on device

On Sat, Jan 10, 2004 at 00:38:43 +0000,
"Nigel J. Andrews" <nandrews@inves tsystems.co.uk> wrote:[color=blue]
>
> On Sat, 10 Jan 2004, Nigel J. Andrews wrote:[color=green]
> > And a common culprit is whatever is being used for usenet caching/serving...or
> > ordinary mail which is just accumulating in /var/mail (or whereever).[/color]
>
> Sheesh. Did I really put ordinary mailbox mail in the uses up inodes category?
> I should taken out and whi....errrr... on the other hand better not might be too
> exciting for some and spark off a whole new xxx web site.[/color]

While mbox mailboxes only take one inode per mailbox, maildir mailboxes take
one inode per message. So if you are using maildir you could potentially
use a significant number of inodes for email.

---------------------------(end of broadcast)---------------------------
TIP 2: you can get off all lists at once with the unregister command
(send "unregister YourEmailAddres sHere" to majordomo@postg resql.org)

[Christopher Browne]

Re: no space left on device

nandrews@invest systems.co.uk ("Nigel J. Andrews") writes:[color=blue]
> On Sat, 10 Jan 2004, Nigel J. Andrews wrote:[color=green]
>> And a common culprit is whatever is being used for usenet caching/serving...or
>> ordinary mail which is just accumulating in /var/mail (or whereever).[/color]
>
> Sheesh. Did I really put ordinary mailbox mail in the uses up inodes category?
> I should taken out and whi....errrr... on the other hand better not might be too
> exciting for some and spark off a whole new xxx web site.[/color]

Mail accumulating in "mbox" spools shouldn't chew up inodes too badly,
but if you're using "Maildir" to spool mail, whether incoming or
outgoing, it sure can...
--
let name="cbbrowne" and tld="libertyrms .info" in String.concat "@" [name;tld];;
<http://dev6.int.libert yrms.com/>
Christopher Browne
(416) 646 3304 x124 (land)

[Aurangzeb M. Agha]

Re: no space left on device

On Fri, 9 Jan 2004, Tom Lane wrote:

:"scott.marlowe " <scott.marlowe@ ihs.com> writes:
:>> [postgres - DB]$ df -m .
:>> Filesystem 1M-blocks Used Available Use% Mounted on
:>> - 63328 55308 4803 93% /
:
:> Do you have root access to it? if so, set the reserved space for root to
:> be 0%, and then try vacuuming. Vacuuming requires some free space, and
:> since you're pretty much out, it isn't gonna be able to complete.
:
:Look again --- it's showing free space in MB not KB. He's got 4.8GB
:free. (Although that might be free-from-root's-point-of-view, rather
:than what an unprivileged user can use ...)

Tom -- You're right here. This account is running on a virtual server, so
the 4.8GB free is not for this user.

Re i-nodes:

[admin - temp]$ df -i .
Filesystem Inodes IUsed IFree IUse% Mounted on
- 8241152 1819166 6421986 23% /

However, I did just get word from the ISP that they had some sort of error
log rotation error which was keeping logs from being deleted off the
machine, taking up a lot of space (for this user account). So the 93% is
aparently not a good representation of the disk usage, as its not for this
specific user account.

Rgs,

Zeb


---------------------------(end of broadcast)---------------------------
TIP 7: don't forget to increase your free space map settings