Storing and Retrieving a large amount of data

[jothivel]

Hi Friends,

I have to store large amount of data and retrieve the same data then write into file in C++.
Currently I am using vector to store and retrieve. But vector is taking more time to store and retrieve the element. Is any other best data structure to store and retrieve large amount of data in unordered way?
Please suggest me the best solution to reduce the time.

Example code:

int I1 = 700,I2 = 32, I3 = 16;
//declare and resize the vector size
vector< vector < vector < vector<DOUBLE> > > > vPARAM; 
vPARAM.resize(I1, vector< vector < vector<DOUBLE> > >
(I2, vector< vector<DOUBLE> > (I3, vector<DOUBLE> (0))));

//Insert
//In the Loop
//The final index "value" it will go unlimited
//I1,I2,I3 also, i will store randomly...
//For eg,First time i will store 100the element,then 50th....
vPARAM.at(I1).at(I2).at(I3).push_back(value);

//Retrieve
//In the Loop
for(lines =0; lines < vecPARAMETER.size();lines++)
iValue = vecPARAMETER[I1][I2][I3][lines];
fwrite(&iValue,8,1,fp);

Thanks,
Jothi

[weaknessforcats]

If the data is not ordered, then why are you using a three-dimensional vector?

You have doubles, say 1000000 of them, then why not:

double* data = new double[1000000];
read ( data, sizeof(double), 1000000 *sizeof(double), fp);

If you don't have a binary file then read the array record by record. Each record can be read into a calculated location in the array. There is never any need to fill dimensions individually.

Please note that there are no 3d arrays in C++. There are only one-dmensional arrays. Read this: http://bytes.com/topic/c/insights/77...rrays-revealed

[jothivel]

Hi,

Thanks for your response.
What I am trying to say is, the data storing in vector is not sequential. I will access any index in the vector at any time.
In between some element can be blank also. Though my vector size is more, I will store the data some index only, some index can be blank. So i cant read and write directly like double[100000]. When I am storing data in all the index, that time we can use this concept. I am in different scenario, that's why i am asking which is best data structure to store and retrieve data with less time complexity?? But I need 4 dimensional to store and retrieve data..

[weaknessforcats]

Vectors expand as you add elements and I susect this expansion is what is consuming ever larger amounts of time.

You might try creating a vector of the right size by using the reserve() method. Then I would initialize every element of the vector with a value that you conider "empty". Now you should be able to hop around any of the elements with no increase in time, ever.

Possibly you could use a struct:

struct Data
{
   double value;
   bool   state;
};

Here you have a vector<Data> where you can use the state memer to denote "empty" or not.