Binary files and 100% CPU usage haunts me

[AaronL]

Hello,

I've recently started to program my own file structure in visual basic 6 and I found out the way to do this of course is binary files. However, my problem is, I don't know anything lol! I know the basics of reading and writing binary files. I came up with a method of pulling records with a comma seperation but it isn't near efficiant because everytime I go read a record, I get 100% CPU usage and that isn't exactly what I want my software to do. Especially when reading multiple records in. Long story short, why does this cause 100% cpu usage and what is a better way to manipulate binary files? The bigger the file, the worse it is...

Dim getloop As Long
Dim binaryin As Byte
getloop = 1
Open "c:\test\test.alr" For Binary As 1
    While Not EOF(1)
        Get #1, getloop, binaryin
        getloop = getloop + 1
    Wend
Close 1

I tried this too

For getloop = 1 to FileLen("c:\test\test.alr") 
     Get #1, getloop, binaryin
Next getloop

I know these processes are going through the entire file, however my question is, how do you know where a record starts and ends without something like

If binaryin = <ascii code for comma or something> Then
Record = Record + 1
End If

I know I'm probably going about this whole thing wrong, I suck. I hope someone can shed some light on this in layman's terms.

[willakawill]

Hi, a couple of things you need when dealing with binary file input and output.
Firstly you should avoid variable length records where possible.
e.g. if you are storing a string it could be

Dim mystr1 As String

which is a variable length or

Dim mystr2 As String * 50

which has a fixed length of 50

You also need to know if the number of records will change or not in which case you may need to store the number of records at the beginning of the binary file

[Killer42]

There may be more to come, as I'm very busy right now. But I think your basic problem is that you're reading one byte at a time.

You can read in just about any size chunk you want. Have a play with this variation...

Dim DataChunk As String
Open "c:\test\test.alr" For Binary Access Read Shared As #1
DataChunk = Space$(LOF(1)) ' [B]Set size of buffer to read into.[/B]
Get #1, , DataChunk ' [B]Read entire buffer in one go.[/B]
Close 1

Note, I believe you can also use a binary array, and there may be many reasons why it's better than a string. I just don't remember how, as I started using strings way back before the binary data type existed.

As you can see for yourself, since the size of the buffer you are reading into determines how much you actually read, you can adjust your buffer size to tweak performance against memory usage. Generally speaking, the bigger the chunks you deal with in each operation, the quicker the read/write will be.

[Killer42]

...You also need to know if the number of records will change or not in which case you may need to store the number of records at the beginning of the binary file

Unless they're fixed length, of course. In which case you can easily calculate the number of records from the file size.

Also, RANDOM may be useful in such a case rather than BINARY.

[AaronL]

Cool, I'm storing the data as a byte though for my file encryption method, how do I allocate memory to a byte variable type

There may be more to come, as I'm very busy right now. But I think your basic problem is that you're reading one byte at a time.

You can read in just about any size chunk you want. Have a play with this variation...

Dim DataChunk As String
Open "c:\test\test.alr" For Binary Access Read Shared As #1
DataChunk = Space$(LOF(1)) ' [B]Set size of buffer to read into.[/B]
Get #1, , DataChunk ' [B]Read entire buffer in one go.[/B]
Close 1

Note, I believe you can also use a binary array, and there may be many reasons why it's better than a string. I just don't remember how, as I started using strings way back before the binary data type existed.

As you can see for yourself, since the size of the buffer you are reading into determines how much you actually read, you can adjust your buffer size to tweak performance against memory usage. Generally speaking, the bigger the chunks you deal with in each operation, the quicker the read/write will be.

[Killer42]

Cool, I'm storing the data as a byte though for my file encryption method, how do I allocate memory to a byte variable type

A Byte variable can only hold a single byte (hence the name), but you can make an array of Byte type, just like any other data type.

Defining an array is as simple as adding the number of occurrences when you define the variable (Eg. Dim DataByte(1 To 1024) As Byte) but I'm afraid I haven't used a byte array in this way for years, and have forgotten how.

I expect willakawill can probably help with this.

One thing to note - the calls to do the IO are the really slow part, so even taking the time to stuff the bytes into a string and then writing the string in one chunk would make things a bit faster. Likewise, to read it back, you could read it into a string and then pick that apart into a Byte array. However, if you're already dealing with Byte variables, you shouldn't have to jump through hoops like that, as you can just work directly with the array.

In fact I've just tried it, and you can read an entire Byte array from a binary file like so...

Dim Data(1 To 50) As Byte
Open "c:\test\test.alr" For Binary Access Read Shared As #1
' I don't recall how to control the number of bytes read/written.
Get #1, , Data ' Read entire array in one go.
Close 1

[AaronL]

Thank you so much, problem solved :)