Possible bug in .net sockets implementation? Reproducible code inside

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • jack

    #1

    Possible bug in .net sockets implementation? Reproducible code inside

    At this link I have two c# projects, one is a client, the other is a
    server. Just point the ip address of the client at the server




    The server does little more than the examples in the c# documentation
    for how to set up an asynchronous server. It just accepts connections,
    reads data into a buffer continously, doing nothing with it.

    The client connects 200 sockets, sends some data with each continously,
    and occasionally disconnects a socket and reconnects a new one.

    Things of note:

    1. in this scenario memory increases forever. Not just the private
    bytes but the gen2 heap increases forever, even after a forced gen2
    garbage collection

    2. memory profilers indicate no left behind object references

    3.the memory profiler indicates "holes" in the gen2 heap, which can be
    caused by "pinned objects". I think perhaps that the data in the
    receive buffers is being held onto indefinitely, despite all sockets
    being closed on the server.


    4. If you change the client to not send any data, the memory will not
    increase. If you change the client to not disconnect and reconnect any
    sockets, memory will not increase.

    If someone can verify if this is a .net bug, or point out my idiocy I
    would greatly appreciate it.

  • jack

    #2
    Re: Possible bug in .net sockets implementation? Reproducible code inside

    Some more info:

    trying this test with 10 sockets will not lead to memory increase, but
    with 50 sockets it will. odd!

    Comment

    • Willy Denoyette [MVP]

      #3
      Re: Possible bug in .net sockets implementation? Reproducible code inside


      "jack" <jack.mott@gmai l.com> wrote in message
      news:1114696874 .112529.200150@ f14g2000cwb.goo glegroups.com.. .[color=blue]
      > At this link I have two c# projects, one is a client, the other is a
      > server. Just point the ip address of the client at the server
      >
      > http://www.slip-angle.com/hosted/bug/
      >
      >
      > The server does little more than the examples in the c# documentation
      > for how to set up an asynchronous server. It just accepts connections,
      > reads data into a buffer continously, doing nothing with it.
      >
      > The client connects 200 sockets, sends some data with each continously,
      > and occasionally disconnects a socket and reconnects a new one.
      >
      > Things of note:
      >
      > 1. in this scenario memory increases forever. Not just the private
      > bytes but the gen2 heap increases forever, even after a forced gen2
      > garbage collection
      >
      > 2. memory profilers indicate no left behind object references
      >
      > 3.the memory profiler indicates "holes" in the gen2 heap, which can be
      > caused by "pinned objects". I think perhaps that the data in the
      > receive buffers is being held onto indefinitely, despite all sockets
      > being closed on the server.
      >
      >
      > 4. If you change the client to not send any data, the memory will not
      > increase. If you change the client to not disconnect and reconnect any
      > sockets, memory will not increase.
      >
      > If someone can verify if this is a .net bug, or point out my idiocy I
      > would greatly appreciate it.
      >[/color]


      No, this is not a bug in the socket library, this relates to the way you
      handle your receive buffers and the way your testclient is implemented.
      Let me try to explain what happens(I hope I succeed).

      You noticed two important things:
      1. The Gen2 heap keeps growing, while the number of objects don't.
      2. There are holes in the Gen2 heap.

      Now if you look at the #Pinned objects perf. counter, you will see there are
      a considerable number of them.
      These objects are long lived so they end in the Gen2 heap, worse, they end
      near the top of the heap, as such preventing a compactation of the heap.
      Note that collection is not prevented by this, which explains the ( nearly)
      constant # of objects.

      Now, why are objects aging into gen2, which objects are these and which are
      the pinned objects?
      These objects are (1)the MsgState instances, they get instantiated when the
      Client connects and they live until the client closes the outgoing socket.
      The MsgState contains a reference (2) dataBuffer to a byte[] used as socket
      buffer, you know by now that these are pinned for the duration of the
      BeginReceive/EndReceive and ProcessMessage sequence. But the problem (lets
      call it 'issue1') is that their lifetime is tied to the lifetime of the
      MsgState object.
      Great, but why are they aging into gen2? This is related to the large number
      of objects (sockets) and the way your client is implemented.
      Your client creates 200 sockets, sends a (too) small buffer to each of them
      in sequence, when done it closes and re-opens one single socket in the
      sequence they were created. At that time the Socket (and related objects)
      it's MsgState and dataBuffer become eligible for collection. Lets say the GC
      kicks in, he won't have a problem to collect the garbage, but heck, he can't
      compact the heap (Gen0 to start with) because there are pinned objects at
      the top of it preventing reallocation, the result is a (small) hole. After
      each GC run, the hole becomes bigger as the same pinned object (your lastly
      created socket buffer) prevents compactation.
      At some point in time, the Gen0 pointer gets adjusted and becomes the Gen1
      pointer and after another number of GC runs the Gen1 pointer becomes the
      Gen2 pointer, but at/near the top of it you will always find a (number of)
      pinned object.

      Ok, how to solve this?
      1. In ProcessMessage, you should move the contents of the (pinned) receive
      buffer to another buffer and release the buffer (re-create a new instance).
      This will shorten the period the buffer is pinned, so it won't end as an
      aged object.
      2. You should change your test method (or at least your message sending
      algorithm), I understand that you are trying to simulate a large number of
      clients, but this is not the right way of doing. You need separate client
      machines, and each one must run a number of processes and each process must
      consist of a number of threads each accessing a single socket.

      Another remark is that you are touching the UI from thread pool threads,
      this is a definitive no no in windows, you have to marshal the call sing
      Control.Invoke.

      Willy.







      Comment

      • jack

        #4
        Re: Possible bug in .net sockets implementation? Reproducible code inside

        Thanks, I will try moving the contents of the buffer and recreating it
        see if that does the trick (though I think I used to do it that way
        with the same problem).

        As for my test method, I wrote that test to simulate what is actually
        happening on our live server, and it is causing the same result.

        Comment

        • jack

          #5
          Re: Possible bug in .net sockets implementation? Reproducible code inside

          Well, making copies of the msgstate and freeing up the old one seems to
          have done the trick with my test case.

          Very interesting. I obviously need to read up more on how the
          compacting process works in .net

          Comment

          • Willy Denoyette [MVP]

            #6
            Re: Possible bug in .net sockets implementation? Reproducible code inside


            "jack" <jack.mott@gmai l.com> wrote in message
            news:1114780810 .636978.78820@g 14g2000cwa.goog legroups.com...[color=blue]
            > Thanks, I will try moving the contents of the buffer and recreating it
            > see if that does the trick (though I think I used to do it that way
            > with the same problem).
            >
            > As for my test method, I wrote that test to simulate what is actually
            > happening on our live server, and it is causing the same result.
            >[/color]

            No, that alone won't do the trick, you also have to change the clients test
            algorithm, make it asynchronous too, and use a number of multithreaded
            clients handling a single or a limited number of sockets per thread.
            Both issues are related, not releasing the buffers at EndReceiveData keep
            them pinned for the duration of the connection, but that's only part of the
            problem. Creating that large number of sockets in sequence followed by
            sending data in the same sequence will result in a lot of aged/pinned
            objects at the server.
            Don't forget that the way you did, it takes 400 seconds before each socket
            has received it's 200 messages and gets closed.

            I'm also not clear on why you send such small buffers (only 6 bytes per
            send), this is a very inefficient way of data transfer over TCP sockets

            Willy.


            Comment

            • jack

              #7
              Re: Possible bug in .net sockets implementation? Reproducible code inside

              I'm simulating what will happen on our live server. Clients will
              connect (a few hundred of them) occasionally send a small bit of data,
              and disconnect after a while. These clients are out there in the
              population so I have to deal with them somehow.

              Comment

              • Willy Denoyette [MVP]

                #8
                Re: Possible bug in .net sockets implementation? Reproducible code inside


                "jack" <jack.mott@gmai l.com> wrote in message
                news:1114791849 .309988.59300@o 13g2000cwo.goog legroups.com...[color=blue]
                > I'm simulating what will happen on our live server. Clients will
                > connect (a few hundred of them) occasionally send a small bit of data,
                > and disconnect after a while. These clients are out there in the
                > population so I have to deal with them somehow.
                >[/color]

                Jack,

                I understand, but IMO it's not very realistic (not impossible) that a client
                waits 2 seconds after a connect request before sending part of the data,
                wait again 2 seconds and wait... etc. and finaly need 6 minutes to send 1200
                bytes to a server. Wouldn't it be better to collect the data at the client
                and send it in one go?

                Just my 2c.
                Willy.


                Comment

                • jack

                  #9
                  Re: Possible bug in .net sockets implementation? Reproducible code inside

                  The clients allow realtime human interaction (chat, forwarding browser
                  events etc). If the user wants to send a short chat message every few
                  seconds, or send scroll events in realtime from time to time, that is
                  what they will do. And they might be in a session for an hour or more.

                  Comment

                  • jack

                    #10
                    Re: Possible bug in .net sockets implementation? Reproducible code inside

                    Anyone else care to comment?

                    It seems kind of ridiculous that dot net can't support behavior like
                    this?

                    How would one go about writing a large scale server that deals with
                    realtime input from users that come and go?

                    Comment

                    • Willy Denoyette [MVP]

                      #11
                      Re: Possible bug in .net sockets implementation? Reproducible code inside


                      "jack" <jack.mott@gmai l.com> wrote in message
                      news:1115321646 .287568.106660@ o13g2000cwo.goo glegroups.com.. .[color=blue]
                      > Anyone else care to comment?
                      >
                      > It seems kind of ridiculous that dot net can't support behavior like
                      > this?
                      >
                      > How would one go about writing a large scale server that deals with
                      > realtime input from users that come and go?
                      >[/color]

                      What do you mean, I thought this was solved by copying your message state.

                      Willy.


                      Comment

                      • Cor Ligthert

                        #12
                        Re: Possible bug in .net sockets implementation? Reproducible code inside

                        Jack,

                        You are using a solution from the beginning of Internet.

                        Ever thought of using a webservice. That makes you less dependend from all
                        kinds of hardware.

                        Just my thought,

                        Cor


                        Comment

                        • jack

                          #13
                          Re: Possible bug in .net sockets implementation? Reproducible code inside

                          We have thought about it, thought I'm not sure how well that would work
                          with realtime interactions that have to go both directions (clients
                          pushing to server and server pushing to clients).

                          Willy, copying the message states made my test case stop growing
                          memory, but not our real production server.

                          I did just this morning come up with a solution that seems to have done
                          the trick.

                          I made a pool of message states. Grab one from the pool as sockets
                          connect, put them back as they disconnect.

                          Comment

                          • Willy Denoyette [MVP]

                            #14
                            Re: Possible bug in .net sockets implementation? Reproducible code inside


                            "jack" <jack.mott@gmai l.com> wrote in message
                            news:1115393312 .453136.192080@ f14g2000cwb.goo glegroups.com.. .[color=blue]
                            > We have thought about it, thought I'm not sure how well that would work
                            > with realtime interactions that have to go both directions (clients
                            > pushing to server and server pushing to clients).
                            >
                            > Willy, copying the message states made my test case stop growing
                            > memory, but not our real production server.
                            >
                            > I did just this morning come up with a solution that seems to have done
                            > the trick.
                            >
                            > I made a pool of message states. Grab one from the pool as sockets
                            > connect, put them back as they disconnect.
                            >[/color]

                            Great, I have written a similar test case as yours, the only thing I did is
                            make sure that the pinned IO buffers are decoupled from the state objects.
                            I'm actually running this code on a server (W2K3) with 2000 sockets
                            connected over a real network (not the loopback interface), managed heap
                            consumption seems stable until now.

                            Willy.


                            Comment

                            • jack

                              #15
                              Re: Possible bug in .net sockets implementation? Reproducible code inside

                              If you are curious to see what I've been working on, you can check it
                              out here:



                              Thanks for your assistance!

                              Comment

                              Working...