question regarding image processing

[Michael Fesser]

Re: question regarding image processing

..oO(The Natural Philosopher)

>Jerry Stuckle wrote:

>Peter H. Coffin wrote:

>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>Álvaro G. Vicario wrote:
>>>>Some guys adore storing binary data in databases but of course
>>>>that's a good old discussion. I particularly find no benefits in it
>>>>(you can't do searches like "WHERE category_id=20 AND picture
>>>>PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>related tasks.
>>>>>
>>>>>
>>>The ability to search is not the only reason to store things in
>>>databases. And if you have hundreds of thousands of images, a
>>>database can be more efficient.
>>>
>>It's a little rough on the incremental backups, though... (:
>>>

>>
>Only the first time. But it makes backups easier - you don't have to
>back up files and database. Just the database.
>>

A database *is* files, Jerry.

Not necessarily. You can keep entire databases in memory (can't get any
faster) or even store multiple databases in a single file, which is what
InnoDB does for example. But for the user and often even for the admin
it's of no interest how the database actually stores the data. Messing
with that and directly accessing the container files in case of a disk-
based DB calls for trouble. Real trouble!

Micha

[The Natural Philosopher]

Re: question regarding image processing

Rik Wasmus wrote:

On Sat, 24 May 2008 10:34:10 +0200, The Natural Philosopher <a@b.cwrote:
>

>Jerry Stuckle wrote:

>>Peter H. Coffin wrote:
>>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>Álvaro G. Vicario wrote:
>>>>>Some guys adore storing binary data in databases but of course
>>>>>that's a good old discussion. I particularly find no benefits in
>>>>>it (you can't do searches like "WHERE category_id=20 AND picture
>>>>>PORTRAIT S A 'black cat'") and it add an annoying overhead in all
>>>>>related tasks.
>>>>>>
>>>>>>
>>>>The ability to search is not the only reason to store things in
>>>>databases . And if you have hundreds of thousands of images, a
>>>>database can be more efficient.
>>>>
>>>It's a little rough on the incremental backups, though... (:
>>>>
>> Only the first time. But it makes backups easier - you don't have
>>to back up files and database. Just the database.
>>>

> A database *is* files, Jerry.

>
... but backing up the files of a database instead of using proper
mechanisms is asking for corruption. It's not the physical presence on a
hard drive that's the problem, it's having to apply 2 different
processes. As a bonus, keeping a database in sync in another location is
far easier (with more mature tools) then syncing files. (Allthough I
could not imagine life without rsync)

So in effect, C-ISAM apart, backing up a database is actually HARDER
than backing up ordinary files?

[The Natural Philosopher]

Re: question regarding image processing

Michael Fesser wrote:

.oO(The Natural Philosopher)
>

>Jerry Stuckle wrote:

>>Peter H. Coffin wrote:
>>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>Álvaro G. Vicario wrote:
>>>>>Some guys adore storing binary data in databases but of course
>>>>>that's a good old discussion. I particularly find no benefits in it
>>>>>(you can't do searches like "WHERE category_id=20 AND picture
>>>>>PORTRAIT S A 'black cat'") and it add an annoying overhead in all
>>>>>related tasks.
>>>>>>
>>>>>>
>>>>The ability to search is not the only reason to store things in
>>>>databases . And if you have hundreds of thousands of images, a
>>>>database can be more efficient.
>>>It's a little rough on the incremental backups, though... (:
>>>>
>>Only the first time. But it makes backups easier - you don't have to
>>back up files and database. Just the database.
>>>

> A database *is* files, Jerry.

>
Not necessarily. You can keep entire databases in memory (can't get any
faster)

Cant get any worse if the system crashes..

or even store multiple databases in a single file, which is what
InnoDB does for example.

It's still a file ;-)

But for the user and often even for the admin
it's of no interest how the database actually stores the data. Messing
with that and directly accessing the container files in case of a disk-
based DB calls for trouble. Real trouble!
>

Actually I have backed up copied and restored C-ISAM files for years
with no problems.

I've even done it without shutting down the copied-from database engine.

On the basis that if I got a bunch of junk when using the copy, I'd do
it properly the longer away round..but in fact it worked.

Micha

[Rik Wasmus]

Re: question regarding image processing

On Sat, 24 May 2008 11:22:54 +0200, The Natural Philosopher <a@b.cwrote:

Rik Wasmus wrote:

>On Sat, 24 May 2008 10:34:10 +0200, The Natural Philosopher <a@b.c>
>wrote:
>>

>>Jerry Stuckle wrote:
>>>Peter H. Coffin wrote:
>>>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>>>Álvaro G. Vicario wrote:
>>>>>>Some guys adore storing binary data in databases but of course
>>>>>>that's a good old discussion. I particularly find no benefits in
>>>>>>it (you can't do searches like "WHERE category_id=20 AND picture
>>>>>>PORTRAI TS A 'black cat'") and it add an annoying overhead in all
>>>>>>related tasks.
>>>>>>>
>>>>>>>
>>>>>The ability to search is not the only reason to store things in
>>>>>database s. And if you have hundreds of thousands of images, a
>>>>>database can be more efficient.
>>>>>
>>>>It's a little rough on the incremental backups, though... (:
>>>>>
>>> Only the first time. But it makes backups easier - you don't have
>>>to back up files and database. Just the database.
>>>>
>> A database *is* files, Jerry.

> ... but backing up the files of a database instead of using proper
>mechanisms is asking for corruption. It's not the physical presence on
>a hard drive that's the problem, it's having to apply 2 different
>processes. As a bonus, keeping a database in sync in another location
>is far easier (with more mature tools) then syncing files. (Allthough I
>could not imagine life without rsync)

>
So in effect, C-ISAM apart, backing up a database is actually HARDER
than backing up ordinary files?

Certainly not. Either point to another database server and say 'just keep
a most recent copy of that server', or use the tool (mysql would be
mysqldump) for the job. Easier to do it right then files, with a lot less
duplicate overhead in most cases. Mind the 'do it right', of course just
downloading / copying files to another point is toddler-easy, but on its
own that's not an ideal backup method. I had more troubles setting up
rsync then a slave mysql database.
--
Rik Wasmus
....spamrun finished

[Jerry Stuckle]

Re: question regarding image processing

The Natural Philosopher wrote:

Jerry Stuckle wrote:

>Peter H. Coffin wrote:

>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>Álvaro G. Vicario wrote:
>>>>Some guys adore storing binary data in databases but of course
>>>>that's a good old discussion. I particularly find no benefits in it
>>>>(you can't do searches like "WHERE category_id=20 AND picture
>>>>PORTRAITS A 'black cat'") and it add an annoying overhead in all
>>>>related tasks.
>>>>>
>>>>>
>>>The ability to search is not the only reason to store things in
>>>databases. And if you have hundreds of thousands of images, a
>>>database can be more efficient.
>>>
>>It's a little rough on the incremental backups, though... (:
>>>

>>
>Only the first time. But it makes backups easier - you don't have to
>back up files and database. Just the database.
>>

A database *is* files, Jerry.

Not necessarily. It depends on the implementation.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===

[Peter H. Coffin]

Re: question regarding image processing

On Fri, 23 May 2008 12:10:15 -0400, Jerry Stuckle wrote:

Peter H. Coffin wrote:

>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:

>>Álvaro G. Vicario wrote:
>>>Some guys adore storing binary data in databases but of course that's a
>>>good old discussion. I particularly find no benefits in it (you can't do
>>>searches like "WHERE category_id=20 AND picture PORTRAITS A 'black
>>>cat'") and it add an annoying overhead in all related tasks.
>>>>
>>>>
>>The ability to search is not the only reason to store things in
>>databases. And if you have hundreds of thousands of images, a database
>>can be more efficient.

>>
>It's a little rough on the incremental backups, though... (:
>>

>
Only the first time. But it makes backups easier - you don't have to
back up files and database. Just the database.

Which DBMS do incremental backups well? I'm mostly familiar with MySQL
and DB2 and both of those are pretty much "shut things down, back up the
whole thing" every single time, while using filesystem storage for
100000 images would mean backing up 99940 only once, then the handful of
new or changed ones every increment. And it takes quite a different
amount of time on my ancient tape drives to back up 10GB instead of
2-3MB.

--
Better to teach a man to fish than to give him a fish. And if he can't
be bothered to learn to fish and starves to death, that's a good enough
outcome for me.
-- Steve VanDevender

[Jerry Stuckle]

Re: question regarding image processing

Peter H. Coffin wrote:

On Fri, 23 May 2008 12:10:15 -0400, Jerry Stuckle wrote:

>Peter H. Coffin wrote:

>>On Fri, 23 May 2008 07:40:37 -0400, Jerry Stuckle wrote:
>>>Álvaro G. Vicario wrote:
>>>>Some guys adore storing binary data in databases but of course that's a
>>>>good old discussion. I particularly find no benefits in it (you can't do
>>>>searches like "WHERE category_id=20 AND picture PORTRAITS A 'black
>>>>cat'") and it add an annoying overhead in all related tasks.
>>>>>
>>>>>
>>>The ability to search is not the only reason to store things in
>>>databases. And if you have hundreds of thousands of images, a database
>>>can be more efficient.
>>It's a little rough on the incremental backups, though... (:
>>>

>Only the first time. But it makes backups easier - you don't have to
>back up files and database. Just the database.

>
Which DBMS do incremental backups well? I'm mostly familiar with MySQL
and DB2 and both of those are pretty much "shut things down, back up the
whole thing" every single time, while using filesystem storage for
100000 images would mean backing up 99940 only once, then the handful of
new or changed ones every increment. And it takes quite a different
amount of time on my ancient tape drives to back up 10GB instead of
2-3MB.
>

DB2 actually will do incremental backups based on the binary logs.
MySQL doesn't do that yet - but you can emulate it with a slave.

You probably could do an incremental backup with MySQL based on the
binary logs, but I don't know how hard it would be.

--
=============== ===
Remove the "x" from my email address
Jerry Stuckle
JDS Computer Training Corp.
jstucklex@attgl obal.net
=============== ===