creating a big array using virtual memory

[harsh123]

I wish to add a new datatype to help me in doing mathametical computations.We all know that the system has got limited amount of memory ie we cannot create an array of very big size. for example a[100000000....]. The new data type will partially use RAM and partially file.For example if i declare an array of size 100000000
Try to compile this program

#include"stdio. h"

int a[10000][10000] = {1,2,3,4};

int main()
{
return 0;
}

So there is a limit to the memory allocation for bigger size. So the new datatype should be able to handle big values (value of 64 bits) and support large number of memory locations.

Hint : We all have the idea of virtual memory.The pages are partially in RAM and partially on HARDDISK. It is similar to this problem. The part of array is in ram and part on a file. We can use space of 10000 bytes from RAM to store the values.Every access to the variable can have two possiblity. It can be in RAM or in file.

[JosAH]

Is that (humongous) array sparse or dense? If it's dense all bets are off, i.e. it's
the OS that allocated memory to your process and it can't allocate more than
its maximum. Or you need a bigger computer.

If the array is sparse there are lots of alternatives to think about.

kind regards,

Jos

[weaknessforcats]

Try to compile this program

I did. It worked.

Max array size is 0x7fffffff (2,147,483,647) which is larger than 100000000.

[JosAH]

I did. It worked.

Max array size is 0x7fffffff (2,147,483,647) which is larger than 100000000.

I think the domain of applicability for an array of 10,000x10,000 ints is very small.
Maybe the OP should elaborate a bit on what those 'mathematical computations'
are all about.

kind regards,

Jos

[Banfa]

Have you though of using a memory mapped file?

Some systems support these directly, you just open the file and create a memory map to it.

[RRick]

Its possible to use the memory/swap space for your array, but how you access the array is very important. If you plan to randomly access the array for reads and writes, then your proformance is going degrade drastically. In computerese, this is called thrashing.

[harsh123]

I did. It worked.

Max array size is 0x7fffffff (2,147,483,647) which is larger than 100000000.

If ur code worked fine, can u provide the code to help me out !!!

[Shankhan]

Have you though of using a memory mapped file?

Some systems support these directly, you just open the file and create a memory map to it.

How can I create a memory map into a file?

[weaknessforcats]

If ur code worked fine, can u provide the code to help me out !!!

I used a local array.
[code=cpp]
#include"stdio. h"
int main()
{
int a[10000][10000] = {1,2,3,4};
return 0;
}
[/code]

Global memory is limited. Besides, there'e never a good reason for a global variable anyhow.

[Banfa]

I used a local array.
[code=cpp]
#include"stdio. h"
int main()
{
int a[10000][10000] = {1,2,3,4};
return 0;
}
[/code]

Global memory is limited. Besides, there'e never a good reason for a global variable anyhow.

Personally I would shy away from putting 10000x10000xsiz eof int (400,000,000 bytes) onto the stack, surely the heap is the place for this if the computer actually has the memory to do it.

I still think create a file on disk and memory map the file is the sensibly way to go.

[probleen]

Personally I would shy away from putting 10000x10000xsiz eof int (400,000,000 bytes) onto the stack, surely the heap is the place for this if the computer actually has the memory to do it.

I still think create a file on disk and memory map the file is the sensibly way to go.

How to place the memory on the heap and what do u mean by memory mapping the file. I m a newbie to C, it'll be very thankful of u to provide a code to explain the same!

[Banfa]

Well you allocate of the heap using new or malloc (depending on language C++ or C).

By memory mapping the file I mean you have a buffer that is smaller than the actual file that you use to smaller load chunks of the file into memory to work on.

As someone has said a lot of non-sequential accesses would result in hitting the hard disk quite hard and probably slow down the program, but it may be the only way to do it.

I know some platforms (Windows) provide an interface to memory map a file for you. Others you would have to do it yourself by hand.