Reading large log files

[PHPstarter]

Hi guys.

Im creating a log-reader online for my game server admins, so that I won't have to give FTP access etc.

The files are from 30-60M MB max in size and I have a working reader atm, but it's not perfect and it slows down fast.

In the middle of the file reading, Im working with lines to replace or add HTML (and some kind of filter for different in-game chat types (a lot of IF and StriStr operations etc)), so I'm wondering what's the best way to do this?

I have a mysql database available if that could be useful, storing each line in there maybe?

IDK, but I hope some of you do :)

[John Doe]

PHPstarter,

This really depends on what you're doing with your logfiles.

There's no point just dumping them into a database as you'll just end up reading them from the database sequentially instead of from the disk.

To go the database route, you need to determine which pieces of information are relevant from the lines in the log file, the split them up and store them in various fields. Doing this allows you to create the appropriate indexes, which will speed up your searching.

Determine what you want to do first though. It may be quicker to split the files up into smaller text files and strip out all irrelevant info for the working files.

Once you're determined what you want to do, specifically, then decide on the appropriate mechanism.

[Rabbit]

If what you're doing is dumping out the entire log file, you could page it. Like they do with search results. Output a megabyte at a time with links to the other pages.