Large Data handle with Vector!

[dmjpro]

Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana

[JosAH]

Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana

50K elements isn't that much if each element isn't that big. If you do want to
change the heap space start "java -X" for help on the X options. (one of them
changes the maximum heap space) but I doubt whether you really need it;
I smell that the OOME is caused by something else.

kind regards,

Jos

[dmjpro]

50K elements isn't that much if each element isn't that big. If you do want to
change the heap space start "java -X" for help on the X options. (one of them
changes the maximum heap space) but I doubt whether you really need it;
I smell that the OOME is caused by something else.

kind regards,

Jos

Vector.addEleme nt gets this Error.
Sorry Josh no of Elements i mentioned wrong.
The actuall data is 1404648.
Now what I am doing is copying the the table values into the Vetor from a ResultSet.
That's why I getting this error.
Without changing the java heap space could not I get the fine tuning over Vector?
One more thing Josh .... Then how ResultSet APIs handle with this large amount of Data?

Debasis Jana

[JosAH]

Vector.addEleme nt gets this Error.
Sorry Josh no of Elements i mentioned wrong.
The actuall data is 1404648.
Now what I am doing is copying the the table values into the Vetor from a ResultSet.
That's why I getting this error.
Without changing the java heap space could not I get the fine tuning over Vector?
One more thing Josh .... Then how ResultSet APIs handle with this large amount of Data?

Debasis Jana

Ah, ok: you're talking abourt +- 30 times as much data now. It depends on the
size of the individual objects, do your math. You could use the flag -Xmx:1G
if you want but why are you loading an entire table in memory? What's the use
of it? Your dbms can give you the wanted records; don't fetch just all of them,
that's what databases are for. btw don't forget to close your ResultSet after
you're done with it.

kind regards,

Jos

[dmjpro]

Ah, ok: you're talking abourt +- 30 times as much data now. It depends on the
size of the individual objects, do your math. You could use the flag -Xmx:1G
if you want but why are you loading an entire table in memory? What's the use
of it? Your dbms can give you the wanted records; don't fetch just all of them,
that's what databases are for. btw don't forget to close your ResultSet after
you're done with it.

kind regards,

Jos

Josh I able to do that.
I used ResultSet instead of using Vector.
And then all the MetaData populated from there insert into a target table.

Debasis Jana.

[chaarmann]

Actually when I trying to put 50000+ data into Vector then it gets out of memory.
How should I handle with it efficiently without change the Java Heap Space?


Debasis Jana

If you want to store integers for example, you should not use Vector() and then add elements by adding new Integer() objects to it. It wastes a lot of time for inboxing-outboxing and memory space because of storing whole objects.
The best way for this is to use arrays of primitive types, like int[] array. This is much faster and uses up significant less memory space. You already know the number of elements to store, so initializing the array to the correct size should be no problem. There is no need to have a "dynamicall y growable" array, such as Vector().

You can also use compression techniques if you need to store Strings. Just use String.intern() for example before storing it into your Vector (or better String[] array). Usually the strings that you get back from your resultset are all using up memory, even if they are the same. By using String.intern() , you will store each different String only once and for each additional string which is the same you store only the reference instead of the string itself. This takes much less memory.

If you still have problems, then you can use a compression technique like Winzip is using.