How to speed-up access to strings

Collapse
X
 
  • Time
  • Show
Clear All
new posts
  • Rabbit
    Recognized Expert MVP
    • Jan 2007
    • 12517

    #31
    The larger issue here is that you are treating binary data like strings. Just read, process, and write the data as a byte array. Skip all this extra overhead of string handling.

    Comment

    • SioSio
      Contributor
      • Dec 2019
      • 272

      #32
      Is there a problem with my code?
      If it is harmful answer, I delete it.

      Comment

      • cactusdata
        Recognized Expert New Member
        • Aug 2007
        • 223

        #33
        VBA is too slow for such tasks.
        I would use C#.

        Comment

        • Rabbit
          Recognized Expert MVP
          • Jan 2007
          • 12517

          #34
          Siosio, your code to profile two different approaches is fine. My reply was directed at Ricardo at their overall approach of using strings in the first place

          Comment

          • SioSio
            Contributor
            • Dec 2019
            • 272

            #35
            Rabbit, thank you for following me.
            Next week, I will finally post the code that solves the Sx2 problem.
            Ricardo de Mira, please wait for a while.

            Comment

            • isladogs
              Recognized Expert Moderator Contributor
              • Jul 2007
              • 479

              #36
              SioSio
              I'm confused.
              IIRC you have posted and removed code twice now before I had a chance to study it properly
              I may still have that code in my notification emails.
              Are you suggesting I should just ignore earlier code as a different version will follow in a few days?

              Comment

              • SioSio
                Contributor
                • Dec 2019
                • 272

                #37
                Hi isladogs.
                The code I posted isn't perfect in terms of speeding up solutions, but it makes sense to want to learn how to use the OP's StrPtr, so I sent a message only to him.
                The code I posted and deleted still has Sx2 issues. This needs to be resolved.

                Comment

                • isladogs
                  Recognized Expert Moderator Contributor
                  • Jul 2007
                  • 479

                  #38
                  OK thanks for letting me know

                  Comment

                  • Rabbit
                    Recognized Expert MVP
                    • Jan 2007
                    • 12517

                    #39
                    CactusData, VBA is perfectly fine for this from a speed perspective. You just have to use the right approach.

                    Treat the data as a byte array and you can process 4 million bytes in roughly 10 milliseconds. The slowest portion of working with file data is the bottleneck at the disk IO.

                    The slowness of Ricardo's code has to do with constntly rebuilding the string variable. As suggested by others, it can be sped up by applying the change directly by also placing the Mid call on the left side of the equation. But this is roughly 10 times slower than just dealing directly with byte arrays, 100 ms vs 10 ms.
                    Last edited by Rabbit; Feb 12 '21, 11:10 PM.

                    Comment

                    • NeoPa
                      Recognized Expert Moderator MVP
                      • Oct 2006
                      • 32633

                      #40
                      @SioSio.
                      As Administrator, or any Moderator, I see no problems with any of your code. It doesn't break any rules. I just struggle to see how it helps. It seems to me that you have maybe not properly understood the requirement, but I say that with the understanding that you're working in a foreign language, so without criticism. Certainly don't worry that you're doing anything I need to be involved with.

                      I would suggest, only suggest mind you, that you read what Rabbit has said very carefully before you post any sort of solution. It would be a shame to waste time on a solution that either won't work or that would be more complicated than it needs to be.

                      @Rabbit.
                      Very interesting. Perhaps you could flesh out the concept with techniques for loading and saving the Byte Arrays you speak of. I doubt most people would know where to start on that and it's your idea so only fair (to you) that you get the first opportunity to fill out the answer so it can be used.

                      Please consider the idea of Best Answer so that your single post includes all parts of the solution including an explanation of why you suggest what you do.

                      Comment

                      • NeoPa
                        Recognized Expert Moderator MVP
                        • Oct 2006
                        • 32633

                        #41
                        @Ricardo.
                        May I suggest that you read the existing replies carefully before deciding that VarPtr() & StrPtr() are necessarily what you need. It seems to me that you haven't understood what's been written here already. Again, there may be difficulties with the language, but we can only really help in English so it makes sense that you try to understand what is already there. Otherwise you'll miss the main benefits of having so many very clever experts who are trying to help you.

                        Comment

                        • Rabbit
                          Recognized Expert MVP
                          • Jan 2007
                          • 12517

                          #42
                          Perhaps I made it sound more obscure than I intended. When you read a file as binary, it returns an array of bytes. When you write to a file as binary, you pass an array of bytes.

                          Comment

                          • NeoPa
                            Recognized Expert Moderator MVP
                            • Oct 2006
                            • 32633

                            #43
                            Interesting. I've done a bit of work using Open#, Close#, Input# & Write# but I was unable to to find anything in help that tells how to do that. Neither old 2003 (proper) help files nor the new ones in docs.microsoft. com mention anything about Byte Arrays. I should add here that though the old Help system was brilliant and the newer ones have been a real let-down, much work has been done in the last number of years to get this back to the level it was at before.

                            Anyway, having found nothing helpful in the Help systems, and not having much of a lead from your post, I dug up some old work of my own and at least re-discovered the Get# & Put# statements. These are not linked to from the Open# but clearly are fundamental to follow where you're going. However, even these I found to refer to string variables for input & output.

                            Eventually, after a great deal of going back & forth in the help systems, I found a short paragraph under opening for Random (Within the Get Statement.), that explains how a Byte Array might work. Later on it goes on to explain which of the paragraphs relating to Random also pertain to Binary. The relevant paragraph was :
                            Originally posted by 2003 Help
                            2003 Help:
                            If the variable being read into is a fixed-size array, Get reads only the data. No descriptor is read.
                            Though I've used Get# & Put# before (Obviously - from my earlier comments.), I've never used Arrays with it. I'd always previously used strings (Pre-set to the desired length). I've found the performance (When buffered properly etc.) to be extremely impressive. Mostly my code works on specific parts of large files but I've hardly noticed any delays.

                            This is extraordinarily useful & powerful. It's also extremely difficult to get helpful Help on. I've reported the page as being less helpful than it could be.

                            For anyone thinking of using these commands I recommend great care. Test your code on copies of files until you know it's working solidly. It's very powerful - and can do great damage if not used carefully.
                            Last edited by NeoPa; Feb 13 '21, 05:29 AM.

                            Comment

                            • Rabbit
                              Recognized Expert MVP
                              • Jan 2007
                              • 12517

                              #44
                              In essence, you'll do something like this:
                              Code:
                              Dim arrBytes() As Byte
                              Dim intFileNum As Long
                              Dim i As Long
                              
                              intFileNum = FreeFile
                              Open "C:\somefile.ext" For Binary Access Read As intFileNum
                              ReDim arrBytes(LOF(intFileNum) - 1)
                              Get intFileNum, , arrBytes
                                  
                              For i = 0 To UBound(arrBytes)
                                  arrBytes(i) = (arrBytes(i) + 1) Mod 256
                              Next i
                              
                              Close intFileNum
                              The portion that manipulates the bytes is a many times faster than the string version but the magnitude of the improvement isn't huge. Both the string and byte array versions are extremely quick, the byte array is just slightly quicker. Does it matter if one takes a second and the other takes a quarter of a second? Probably not, but the byte array version is quicker and cleaner.
                              Last edited by Rabbit; Feb 13 '21, 05:36 PM.

                              Comment

                              • ADezii
                                Recognized Expert Expert
                                • Apr 2006
                                • 8834

                                #45
                                Code:
                                Dim intFileNum%
                                Data Type Identifier at the end of a Variable Declaration? I thought that I was old school, wait till NeoPa and isladogs see this, then they will have someone else to pick on! (LOL)

                                Comment

                                Working...