detect bytes written on abort

Collapse
This topic is closed.
X
X
 
  • Time
  • Show
Clear All
new posts
  • Shailesh Humbad

    #1

    detect bytes written on abort

    Here is an advanced PHP question. Can anyone think of a way to detect
    the number of bytes written to output when a script is aborted?

    I am sending a large file to the client, and I want to record how many
    bytes are actually sent. I can detect abort of the script using a
    shutdown handler. In the shutdown handler, I tried ob_get_length, but
    it returns false. I tried to read the server's log file, but it is does
    not contain the information until the script fully quits. I tried to
    use fwrite to php://output, and then get the bytes written return value.
    However, if the script aborts in the middle of the write, then bytes
    written is never returned.

    The only thing that worked was writing one byte at a time using
    fread/fwrite. But this also made the processor load 100% and download
    speed very slow (700KB/sec versus 24MB/sec on localhost using wget).

    I know I can poll the web server's log file using a background process.
    But if I can do everything from within the script, the system becomes
    simpler and more responsive.

    Below is the PHP 5.1.1 test script. I suspect it can not be done, but
    any advice would be appreciated!

    sendfile.php

    ignore_user_abo rt(false);
    set_time_limit( 60);
    register_shutdo wn_function("ha ndleShutdown");

    $fp = false;
    $fp = fopen("largefil e.html", "rb");
    fpassthru($fp); // script is aborted in mid-execution

    function handleShutdown( ) {
    global $fp;
    if($fp !== false) {
    fclose($fp);
    }
    $byteswritten = 0; // how to detect here?
    $shutdownmessag e =
    "bytes: ".$byteswritten ."\n".
    "status: ".connection_st atus()."\n".
    "aborted: ".connection_ab orted()."\n";

    file_put_conten ts("shutdownlog .txt", $shutdownmessag e);
    }

    Here is a windows batch file to run the request:

    ECHO Hit Ctrl-C to simulate abort
    IF EXIST sendfile.php.1 del sendfile.php.1
    wget http://localhost/sendfile.php --tries=1
  • Gordon Burditt

    #2
    Re: detect bytes written on abort

    >Here is an advanced PHP question. Can anyone think of a way to detect[color=blue]
    >the number of bytes written to output when a script is aborted?[/color]

    What are you going to use that information for?
    [color=blue]
    >I am sending a large file to the client, and I want to record how many
    >bytes are actually sent.[/color]

    Bytes sent != bytes received. Why is there a problem with the
    script aborting? Modem drops carrier? Spontaneous Windows reboot?
    Browser crash?
    [color=blue]
    >I can detect abort of the script using a
    >shutdown handler. In the shutdown handler, I tried ob_get_length, but
    >it returns false. I tried to read the server's log file, but it is does
    >not contain the information until the script fully quits. I tried to
    >use fwrite to php://output, and then get the bytes written return value.
    >However, if the script aborts in the middle of the write, then bytes
    >written is never returned.[/color]

    If one end of a TCP connection quietly goes away, you might be
    able to write 64k more bytes before you realize it has gone away.
    [color=blue]
    >The only thing that worked was writing one byte at a time using
    >fread/fwrite. But this also made the processor load 100% and download
    >speed very slow (700KB/sec versus 24MB/sec on localhost using wget).[/color]

    What do you intend to use this for? If you're trying to figure out
    where to restart sending the file, it won't be accurate.

    Gordon L. Burditt

    Comment

    • Shailesh Humbad

      #3
      Re: detect bytes written on abort

      Gordon Burditt wrote:[color=blue][color=green]
      >> Here is an advanced PHP question. Can anyone think of a way to detect
      >> the number of bytes written to output when a script is aborted?[/color]
      >
      > What are you going to use that information for?
      >[color=green]
      >> I am sending a large file to the client, and I want to record how many
      >> bytes are actually sent.[/color]
      >
      > Bytes sent != bytes received. Why is there a problem with the
      > script aborting? Modem drops carrier? Spontaneous Windows reboot?
      > Browser crash?
      >[color=green]
      >> I can detect abort of the script using a
      >> shutdown handler. In the shutdown handler, I tried ob_get_length, but
      >> it returns false. I tried to read the server's log file, but it is does
      >> not contain the information until the script fully quits. I tried to
      >> use fwrite to php://output, and then get the bytes written return value.
      >> However, if the script aborts in the middle of the write, then bytes
      >> written is never returned.[/color]
      >
      > If one end of a TCP connection quietly goes away, you might be
      > able to write 64k more bytes before you realize it has gone away.
      >[color=green]
      >> The only thing that worked was writing one byte at a time using
      >> fread/fwrite. But this also made the processor load 100% and download
      >> speed very slow (700KB/sec versus 24MB/sec on localhost using wget).[/color]
      >
      > What do you intend to use this for? If you're trying to figure out
      > where to restart sending the file, it won't be accurate.
      >
      > Gordon L. Burditt[/color]

      TCP is a reliable transport, meaning that at the application layer, one
      always know exactly how much data the client received, and this is
      always equal to how much was successfully sent. I don't care how many
      bytes were transferred by TCP in the data link layer.

      I don't want to restart sending the file. I also do not care why the
      script aborted. Of course, it must be from a network/client abort, not
      a server reboot or such, because the script must finish executing. I
      only want to be able to track how many bytes were sent to the client,
      which equals the value that is eventually written to the server log file.

      The reason I need it is because in this system, I want to be able to
      show the user how many bytes the server sent them. This will tell them
      how much data transfer they have used.

      I need the status of bytes sent as soon as possible after the script
      completes or aborts. Thanks.

      Comment

      • Chung Leong

        #4
        Re: detect bytes written on abort

        Shailesh Humbad wrote:[color=blue]
        > Here is an advanced PHP question. Can anyone think of a way to detect
        > the number of bytes written to output when a script is aborted?[/color]

        If you are using Apache and can recompile the SAPI module, you can add
        the following function to php_functions.c :

        /* {{{ proto integer apache_get_byte s_sent(void)
        Get the number of bytes actually sent */
        PHP_FUNCTION(ap ache_get_bytes_ sent)
        {
        php_struct *ctx;
        ctx = SG(server_conte xt);
        RETURN_LONG(ctx->r->bytes_sent);
        }
        /* }}} */


        If TCP/IP guarenteed delivery is to be believed, then that should be
        the exact number of bytes received (but not necessarily saved) by the
        client.

        Comment

        • Gordon Burditt

          #5
          Re: detect bytes written on abort

          >TCP is a reliable transport, meaning that at the application layer, one[color=blue]
          >always know exactly how much data the client received, and this is
          >always equal to how much was successfully sent.[/color]

          The above is *NOT* a conventional definition of "reliable transport".
          And it's not what TCP tries to implement.

          Stdio buffering put on a "reliable transport" as you define it above
          makes it unreliable, as a successful fwrite() on a socket may simply
          mean that the data has been placed in a buffer on the sender, not
          even passed to the OS yet. You also don't know how much data is
          buffered by Apache or web proxies. You don't know that the other
          end of the TCP connection is on the user's browser.

          In a scenario where the communication channel is going to be cut
          at some point in time (corresponding to, say, a modem dropping
          carrier or network connectivity otherwise going down and staying
          down), and no further message traffic is possible, it is impossible
          to implement a protocol where the sender and receiver always agree
          exactly on the number of bytes received. If you send a packet and
          get no answer, you don't know whether the sent packet got lost or
          the acknowledgement got lost. You can get the uncertainty down to
          one byte by sending single-byte packets all the time. Slow. Wasteful
          of bandwidth. Even the Theory of Relativity is relevant here. The
          Speed of Data, as well as the Speed of Light, is finite and does
          not permit instantaneous communication of information.
          [color=blue]
          >I don't care how many
          >bytes were transferred by TCP in the data link layer.[/color]
          [color=blue]
          >I don't want to restart sending the file. I also do not care why the
          >script aborted.[/color]

          You DO care if the *client* aborted. Just because the browser got the
          data from TCP doesn't mean it was safely saved to disk before someone
          tripped over the power cord.
          [color=blue]
          >Of course, it must be from a network/client abort, not
          >a server reboot or such, because the script must finish executing. I
          >only want to be able to track how many bytes were sent to the client,
          >which equals the value that is eventually written to the server log file.
          >
          >The reason I need it is because in this system, I want to be able to
          >show the user how many bytes the server sent them. This will tell them
          >how much data transfer they have used.[/color]

          Why would the user care? Unless you're billing them against a quota
          or something, which is quite a different problem from being able
          to restart a file transfer.
          [color=blue]
          >I need the status of bytes sent as soon as possible after the script
          >completes or aborts. Thanks.[/color]

          It won't happen reliably. You might get something accurate enough
          for *quotas*, but not for restarting file transfers. The way things
          like FTP do this is get the size of the partially-transferred file
          on the client side and start from there.

          Gordon L. Burditt

          Comment

          • Shailesh Humbad

            #6
            Re: detect bytes written on abort

            Chung Leong wrote:[color=blue]
            > Shailesh Humbad wrote:[color=green]
            >> Here is an advanced PHP question. Can anyone think of a way to detect
            >> the number of bytes written to output when a script is aborted?[/color]
            >
            > If you are using Apache and can recompile the SAPI module, you can add
            > the following function to php_functions.c :
            >
            > /* {{{ proto integer apache_get_byte s_sent(void)
            > Get the number of bytes actually sent */
            > PHP_FUNCTION(ap ache_get_bytes_ sent)
            > {
            > php_struct *ctx;
            > ctx = SG(server_conte xt);
            > RETURN_LONG(ctx->r->bytes_sent);
            > }
            > /* }}} */
            >
            >
            > If TCP/IP guarenteed delivery is to be believed, then that should be
            > the exact number of bytes received (but not necessarily saved) by the
            > client.
            >[/color]

            Here is my hosting provider's configuration:
            apache_get_vers ion: Apache/1.3.33 (Debian GNU/Linux)
            mod_throttle/3.1.2 mod_ssl/2.8.22 OpenSSL/0.9.7d
            php_sapi_name: apache

            I am unfamiliar with PHP internals and how they interact with the web
            server. Will that only work on Apache 2 and with sapi as
            "apache2fil ter" or "apache2handler "? I don't think I can recompile the
            SAPI module, but I could build my own module and have my hosting
            provider install it. Is it possible to add such a function in a custom
            module? If it is too hard, then I will just go the low-tech route
            (polling the log file).

            Thanks.

            Comment

            • Shailesh Humbad

              #7
              Re: detect bytes written on abort

              Gordon Burditt wrote:[color=blue][color=green]
              >> TCP is a reliable transport, meaning that at the application layer, one
              >> always know exactly how much data the client received, and this is
              >> always equal to how much was successfully sent.[/color]
              >
              > The above is *NOT* a conventional definition of "reliable transport".
              > And it's not what TCP tries to implement.
              >
              > Stdio buffering put on a "reliable transport" as you define it above
              > makes it unreliable, as a successful fwrite() on a socket may simply
              > mean that the data has been placed in a buffer on the sender, not
              > even passed to the OS yet. You also don't know how much data is
              > buffered by Apache or web proxies. You don't know that the other
              > end of the TCP connection is on the user's browser.
              >
              > In a scenario where the communication channel is going to be cut
              > at some point in time (corresponding to, say, a modem dropping
              > carrier or network connectivity otherwise going down and staying
              > down), and no further message traffic is possible, it is impossible
              > to implement a protocol where the sender and receiver always agree
              > exactly on the number of bytes received. If you send a packet and
              > get no answer, you don't know whether the sent packet got lost or
              > the acknowledgement got lost. You can get the uncertainty down to
              > one byte by sending single-byte packets all the time. Slow. Wasteful
              > of bandwidth. Even the Theory of Relativity is relevant here. The
              > Speed of Data, as well as the Speed of Light, is finite and does
              > not permit instantaneous communication of information.
              >[color=green]
              >> I don't care how many
              >> bytes were transferred by TCP in the data link layer.[/color]
              >[color=green]
              >> I don't want to restart sending the file. I also do not care why the
              >> script aborted.[/color]
              >
              > You DO care if the *client* aborted. Just because the browser got the
              > data from TCP doesn't mean it was safely saved to disk before someone
              > tripped over the power cord.
              >[color=green]
              >> Of course, it must be from a network/client abort, not
              >> a server reboot or such, because the script must finish executing. I
              >> only want to be able to track how many bytes were sent to the client,
              >> which equals the value that is eventually written to the server log file.
              >>
              >> The reason I need it is because in this system, I want to be able to
              >> show the user how many bytes the server sent them. This will tell them
              >> how much data transfer they have used.[/color]
              >
              > Why would the user care? Unless you're billing them against a quota
              > or something, which is quite a different problem from being able
              > to restart a file transfer.
              >[color=green]
              >> I need the status of bytes sent as soon as possible after the script
              >> completes or aborts. Thanks.[/color]
              >
              > It won't happen reliably. You might get something accurate enough
              > for *quotas*, but not for restarting file transfers. The way things
              > like FTP do this is get the size of the partially-transferred file
              > on the client side and start from there.
              >
              > Gordon L. Burditt[/color]

              I don't care how much data the client actually saved, only how much was
              transferred. Yes, my eventual aim is to bill against a quota. To solve
              the file restart problem, I can implement HTTP range handling later.

              "Reliable Delivery - Once a connection has been established, TCP
              guarantees that data is delivered in exactly the same order it was sent,
              with no loss, and no duplication. If a failure prevents reliable
              delivery, the sender is informed.", Internetworking with TCP/IP Vol.
              III, p. 103

              Comment

              • Gordon Burditt

                #8
                Re: detect bytes written on abort

                >>> TCP is a reliable transport, meaning that at the application layer, one[color=blue][color=green][color=darkred]
                >>> always know exactly how much data the client received, and this is
                >>> always equal to how much was successfully sent.[/color]
                >>
                >> The above is *NOT* a conventional definition of "reliable transport".
                >> And it's not what TCP tries to implement.
                >>
                >> Stdio buffering put on a "reliable transport" as you define it above
                >> makes it unreliable, as a successful fwrite() on a socket may simply
                >> mean that the data has been placed in a buffer on the sender, not
                >> even passed to the OS yet. You also don't know how much data is
                >> buffered by Apache or web proxies. You don't know that the other
                >> end of the TCP connection is on the user's browser.
                >>
                >> In a scenario where the communication channel is going to be cut
                >> at some point in time (corresponding to, say, a modem dropping
                >> carrier or network connectivity otherwise going down and staying
                >> down), and no further message traffic is possible, it is impossible
                >> to implement a protocol where the sender and receiver always agree
                >> exactly on the number of bytes received. If you send a packet and
                >> get no answer, you don't know whether the sent packet got lost or
                >> the acknowledgement got lost. You can get the uncertainty down to
                >> one byte by sending single-byte packets all the time. Slow. Wasteful
                >> of bandwidth. Even the Theory of Relativity is relevant here. The
                >> Speed of Data, as well as the Speed of Light, is finite and does
                >> not permit instantaneous communication of information.
                >>[color=darkred]
                >>> I don't care how many
                >>> bytes were transferred by TCP in the data link layer.[/color]
                >>[color=darkred]
                >>> I don't want to restart sending the file. I also do not care why the
                >>> script aborted.[/color]
                >>
                >> You DO care if the *client* aborted. Just because the browser got the
                >> data from TCP doesn't mean it was safely saved to disk before someone
                >> tripped over the power cord.
                >>[color=darkred]
                >>> Of course, it must be from a network/client abort, not
                >>> a server reboot or such, because the script must finish executing. I
                >>> only want to be able to track how many bytes were sent to the client,
                >>> which equals the value that is eventually written to the server log file.
                >>>
                >>> The reason I need it is because in this system, I want to be able to
                >>> show the user how many bytes the server sent them. This will tell them
                >>> how much data transfer they have used.[/color]
                >>
                >> Why would the user care? Unless you're billing them against a quota
                >> or something, which is quite a different problem from being able
                >> to restart a file transfer.
                >>[color=darkred]
                >>> I need the status of bytes sent as soon as possible after the script
                >>> completes or aborts. Thanks.[/color]
                >>
                >> It won't happen reliably. You might get something accurate enough
                >> for *quotas*, but not for restarting file transfers. The way things
                >> like FTP do this is get the size of the partially-transferred file
                >> on the client side and start from there.
                >>
                >> Gordon L. Burditt[/color]
                >
                >I don't care how much data the client actually saved, only how much was
                >transferred. Yes, my eventual aim is to bill against a quota.[/color]

                Why do you care about getting these numbers exact? You don't
                seem to care about what is transmitted at the data link layer,
                which is probably how your provider will bill YOU if your
                agreement with them involves traffic-sensitive costs.
                [color=blue]
                >"Reliable Delivery - Once a connection has been established, TCP
                >guarantees that data is delivered in exactly the same order it was sent,
                >with no loss, and no duplication. If a failure prevents reliable
                >delivery, the sender is informed.", Internetworking with TCP/IP Vol.
                >III, p. 103[/color]

                This says nothing about knowing HOW MUCH was delivered in the case
                of a failure. If the session fails, you know not all of it got
                delivered. You also know that they didn't get any more than you
                sent. When a write() on a socket returns, you don't know that ANY
                of it got delivered (yet). A failure may be reported later. Much
                later. The above quote does not say "If a failure prevents reliable
                delivery, the sender is informed instantaneously with an itemized
                report of how much was delivered".

                Gordon L. Burditt

                Comment

                • Peter Fox

                  #9
                  Re: detect bytes written on abort

                  Following on from Shailesh Humbad's message. . .

                  8><
                  [color=blue]
                  >
                  >I don't care how much data the client actually saved, only how much was
                  >transferred. Yes, my eventual aim is to bill against a quota. To solve
                  >the file restart problem, I can implement HTTP range handling later.
                  >
                  >"Reliable Delivery - Once a connection has been established, TCP
                  >guarantees that data is delivered in exactly the same order it was sent,
                  >with no loss, and no duplication. If a failure prevents reliable
                  >delivery, the sender is informed.", Internetworking with TCP/IP Vol.
                  >III, p. 103[/color]

                  Gordon, being the sort of person who spots the cracks down which bad
                  things happen that most people miss, is suggesting that what you sent
                  will not be the same as what was received unless completed and there
                  will be buffers of various sorts between your code and your cable to the
                  Internet and then some. In this he is trying to be helpful because the
                  effort required to engineer a byte-accurate solution is going to be a
                  great deal more than to obtain an estimate.

                  Here is a 'free' solution: If a file transfer fails then charge for
                  half the full size. Either there are very few failures in practice in
                  which case the issue is not very important in the scheme of things; or
                  it happens all the time, in which case this will work out fine
                  statistically. [But you will lose all your customers for reasons of crap
                  service.]


                  --
                  PETER FOX Not the same since the bolt company screwed up
                  peterfox@eminen t.demon.co.uk.n ot.this.bit.no. html
                  2 Tees Close, Witham, Essex.
                  Gravity beer in Essex <http://www.eminent.dem on.co.uk>

                  Comment

                  • Shailesh Humbad

                    #10
                    Re: detect bytes written on abort

                    Gordon Burditt wrote:[color=blue][color=green][color=darkred]
                    >>>> TCP is a reliable transport, meaning that at the application layer, one
                    >>>> always know exactly how much data the client received, and this is
                    >>>> always equal to how much was successfully sent.
                    >>> The above is *NOT* a conventional definition of "reliable transport".
                    >>> And it's not what TCP tries to implement.
                    >>>
                    >>> Stdio buffering put on a "reliable transport" as you define it above
                    >>> makes it unreliable, as a successful fwrite() on a socket may simply
                    >>> mean that the data has been placed in a buffer on the sender, not
                    >>> even passed to the OS yet. You also don't know how much data is
                    >>> buffered by Apache or web proxies. You don't know that the other
                    >>> end of the TCP connection is on the user's browser.
                    >>>
                    >>> In a scenario where the communication channel is going to be cut
                    >>> at some point in time (corresponding to, say, a modem dropping
                    >>> carrier or network connectivity otherwise going down and staying
                    >>> down), and no further message traffic is possible, it is impossible
                    >>> to implement a protocol where the sender and receiver always agree
                    >>> exactly on the number of bytes received. If you send a packet and
                    >>> get no answer, you don't know whether the sent packet got lost or
                    >>> the acknowledgement got lost. You can get the uncertainty down to
                    >>> one byte by sending single-byte packets all the time. Slow. Wasteful
                    >>> of bandwidth. Even the Theory of Relativity is relevant here. The
                    >>> Speed of Data, as well as the Speed of Light, is finite and does
                    >>> not permit instantaneous communication of information.
                    >>>
                    >>>> I don't care how many
                    >>>> bytes were transferred by TCP in the data link layer.
                    >>>> I don't want to restart sending the file. I also do not care why the
                    >>>> script aborted.
                    >>> You DO care if the *client* aborted. Just because the browser got the
                    >>> data from TCP doesn't mean it was safely saved to disk before someone
                    >>> tripped over the power cord.
                    >>>
                    >>>> Of course, it must be from a network/client abort, not
                    >>>> a server reboot or such, because the script must finish executing. I
                    >>>> only want to be able to track how many bytes were sent to the client,
                    >>>> which equals the value that is eventually written to the server log file.
                    >>>>
                    >>>> The reason I need it is because in this system, I want to be able to
                    >>>> show the user how many bytes the server sent them. This will tell them
                    >>>> how much data transfer they have used.
                    >>> Why would the user care? Unless you're billing them against a quota
                    >>> or something, which is quite a different problem from being able
                    >>> to restart a file transfer.
                    >>>
                    >>>> I need the status of bytes sent as soon as possible after the script
                    >>>> completes or aborts. Thanks.
                    >>> It won't happen reliably. You might get something accurate enough
                    >>> for *quotas*, but not for restarting file transfers. The way things
                    >>> like FTP do this is get the size of the partially-transferred file
                    >>> on the client side and start from there.
                    >>>
                    >>> Gordon L. Burditt[/color]
                    >> I don't care how much data the client actually saved, only how much was
                    >> transferred. Yes, my eventual aim is to bill against a quota.[/color]
                    >
                    > Why do you care about getting these numbers exact? You don't
                    > seem to care about what is transmitted at the data link layer,
                    > which is probably how your provider will bill YOU if your
                    > agreement with them involves traffic-sensitive costs.
                    >[color=green]
                    >> "Reliable Delivery - Once a connection has been established, TCP
                    >> guarantees that data is delivered in exactly the same order it was sent,
                    >> with no loss, and no duplication. If a failure prevents reliable
                    >> delivery, the sender is informed.", Internetworking with TCP/IP Vol.
                    >> III, p. 103[/color]
                    >
                    > This says nothing about knowing HOW MUCH was delivered in the case
                    > of a failure. If the session fails, you know not all of it got
                    > delivered. You also know that they didn't get any more than you
                    > sent. When a write() on a socket returns, you don't know that ANY
                    > of it got delivered (yet). A failure may be reported later. Much
                    > later. The above quote does not say "If a failure prevents reliable
                    > delivery, the sender is informed instantaneously with an itemized
                    > report of how much was delivered".
                    >
                    > Gordon L. Burditt[/color]

                    You have good points, but I just don't need that much resolution or
                    accuracy. The socket will time out in 30 seconds if there is a problem
                    sending data. The bytes returned by the write call, even if known 30
                    seconds later, is all I need, and I know somewhere internally in PHP it
                    is being recorded.

                    When socket write returns (if being called in blocking-mode), it returns
                    the number of bytes written successfully to the socket. This is the
                    number of bytes guaranteed to be delivered to the client's receiving
                    socket (though the client may not have written it all to disk or other
                    issues may have occurred). The reason why TCP can know the number of
                    bytes sent with certainty is because every sent packet is replied to
                    with an acknowledgment (ACK) packet.

                    For my purposes, this number is going to be a decent approximation of
                    actual bandwidth used, and I realize it's not going to be exact. Thanks.

                    Comment

                    • Gordon Burditt

                      #11
                      Re: detect bytes written on abort

                      >> Why do you care about getting these numbers exact? You don't[color=blue][color=green]
                      >> seem to care about what is transmitted at the data link layer,
                      >> which is probably how your provider will bill YOU if your
                      >> agreement with them involves traffic-sensitive costs.
                      >>[color=darkred]
                      >>> "Reliable Delivery - Once a connection has been established, TCP
                      >>> guarantees that data is delivered in exactly the same order it was sent,
                      >>> with no loss, and no duplication. If a failure prevents reliable
                      >>> delivery, the sender is informed.", Internetworking with TCP/IP Vol.
                      >>> III, p. 103[/color]
                      >>
                      >> This says nothing about knowing HOW MUCH was delivered in the case
                      >> of a failure. If the session fails, you know not all of it got
                      >> delivered. You also know that they didn't get any more than you
                      >> sent. When a write() on a socket returns, you don't know that ANY
                      >> of it got delivered (yet). A failure may be reported later. Much
                      >> later. The above quote does not say "If a failure prevents reliable
                      >> delivery, the sender is informed instantaneously with an itemized
                      >> report of how much was delivered".
                      >>
                      >> Gordon L. Burditt[/color]
                      >
                      >You have good points, but I just don't need that much resolution or
                      >accuracy. The socket will time out in 30 seconds if there is a problem
                      >sending data. The bytes returned by the write call, even if known 30
                      >seconds later, is all I need, and I know somewhere internally in PHP it
                      >is being recorded.
                      >
                      >When socket write returns (if being called in blocking-mode), it returns
                      >the number of bytes written successfully to the socket.[/color]

                      This has absolutely nothing to do with the number of bytes received
                      by the other side. The write() call (*IN BLOCKING MODE*) may return
                      before even one packet has been assembled to be sent. The write()
                      call will block if you run out of buffering. If it blocks, and
                      then returns, it still might not have even tried to send any data
                      from the last dozen or so write calls prior to the one that just
                      returned.
                      [color=blue]
                      >This is the
                      >number of bytes guaranteed to be delivered to the client's receiving
                      >socket (though the client may not have written it all to disk or other
                      >issues may have occurred).[/color]

                      It is guaranteed that that data will be delivered to the client
                      *EVENTUALLY* or you will *EVENTUALLY* get an error. There is no
                      guarantee whatever that any of that data has been delivered (or even
                      attempted to be sent) at the time the write() returns.
                      [color=blue]
                      >The reason why TCP can know the number of
                      >bytes sent with certainty is because every sent packet is replied to
                      >with an acknowledgment (ACK) packet.[/color]

                      And the write() call *IN BLOCKING MODE* does not wait for such an
                      acknowledgement to be received. It doesn't even have to wait for
                      even one packet to be sent. It would be horribly inefficient if
                      you couldn't overlap, say, disk reads and network writes in a
                      single-threaded process that is sending a file down a socket, so
                      writes *DO NOT* wait until the client has received the data written.
                      Not even blocking writes.
                      [color=blue]
                      >For my purposes, this number is going to be a decent approximation of
                      >actual bandwidth used, and I realize it's not going to be exact. Thanks.[/color]

                      If you're willing to put up with, say, an extra 32k or 64k sent but
                      not received when the modem drops carrier, you'll get a decent
                      approximation. If you're expecting much better than that, you won't.

                      Gordon L. Burditt

                      Comment

                      • Shailesh Humbad

                        #12
                        Re: detect bytes written on abort

                        Gordon Burditt wrote:[color=blue][color=green][color=darkred]
                        >>> Why do you care about getting these numbers exact? You don't
                        >>> seem to care about what is transmitted at the data link layer,
                        >>> which is probably how your provider will bill YOU if your
                        >>> agreement with them involves traffic-sensitive costs.
                        >>>
                        >>>> "Reliable Delivery - Once a connection has been established, TCP
                        >>>> guarantees that data is delivered in exactly the same order it was sent,
                        >>>> with no loss, and no duplication. If a failure prevents reliable
                        >>>> delivery, the sender is informed.", Internetworking with TCP/IP Vol.
                        >>>> III, p. 103
                        >>> This says nothing about knowing HOW MUCH was delivered in the case
                        >>> of a failure. If the session fails, you know not all of it got
                        >>> delivered. You also know that they didn't get any more than you
                        >>> sent. When a write() on a socket returns, you don't know that ANY
                        >>> of it got delivered (yet). A failure may be reported later. Much
                        >>> later. The above quote does not say "If a failure prevents reliable
                        >>> delivery, the sender is informed instantaneously with an itemized
                        >>> report of how much was delivered".
                        >>>
                        >>> Gordon L. Burditt[/color]
                        >> You have good points, but I just don't need that much resolution or
                        >> accuracy. The socket will time out in 30 seconds if there is a problem
                        >> sending data. The bytes returned by the write call, even if known 30
                        >> seconds later, is all I need, and I know somewhere internally in PHP it
                        >> is being recorded.
                        >>
                        >> When socket write returns (if being called in blocking-mode), it returns
                        >> the number of bytes written successfully to the socket.[/color]
                        >
                        > This has absolutely nothing to do with the number of bytes received
                        > by the other side. The write() call (*IN BLOCKING MODE*) may return
                        > before even one packet has been assembled to be sent. The write()
                        > call will block if you run out of buffering. If it blocks, and
                        > then returns, it still might not have even tried to send any data
                        > from the last dozen or so write calls prior to the one that just
                        > returned.
                        >[color=green]
                        >> This is the
                        >> number of bytes guaranteed to be delivered to the client's receiving
                        >> socket (though the client may not have written it all to disk or other
                        >> issues may have occurred).[/color]
                        >
                        > It is guaranteed that that data will be delivered to the client
                        > *EVENTUALLY* or you will *EVENTUALLY* get an error. There is no
                        > guarantee whatever that any of that data has been delivered (or even
                        > attempted to be sent) at the time the write() returns.
                        >[color=green]
                        >> The reason why TCP can know the number of
                        >> bytes sent with certainty is because every sent packet is replied to
                        >> with an acknowledgment (ACK) packet.[/color]
                        >
                        > And the write() call *IN BLOCKING MODE* does not wait for such an
                        > acknowledgement to be received. It doesn't even have to wait for
                        > even one packet to be sent. It would be horribly inefficient if
                        > you couldn't overlap, say, disk reads and network writes in a
                        > single-threaded process that is sending a file down a socket, so
                        > writes *DO NOT* wait until the client has received the data written.
                        > Not even blocking writes.
                        >[color=green]
                        >> For my purposes, this number is going to be a decent approximation of
                        >> actual bandwidth used, and I realize it's not going to be exact. Thanks.[/color]
                        >
                        > If you're willing to put up with, say, an extra 32k or 64k sent but
                        > not received when the modem drops carrier, you'll get a decent
                        > approximation. If you're expecting much better than that, you won't.
                        >
                        > Gordon L. Burditt[/color]

                        Okay, I think I see what you're saying. The "write" actually writes
                        first into internal buffers of the operating system, and so the number
                        of bytes it returns is not necessarily the number of bytes delivered.
                        "Write" only blocks if the internal buffers are full. So the number of
                        bytes returned by "write" (or "send") is the number of bytes written
                        into the TCP stack, but not the number of bytes that received ACKS
                        indicating delivery.

                        In any case, an extra 32-64K of bandwidth used on failed connections is
                        acceptable. I'll have to figure out a way to prevent this from becoming
                        an issue.

                        Question, could someone take advantage of this by writing a network
                        driver that requests data, but then doesn't send ACK replies after the
                        initial request is completed? My server's TCP stack would happily send
                        out 32-64KB of data, waiting for the ACKs. The nefarious recipient
                        would receive and save this data, drop the connection, and then request
                        the next range of bytes. Then, they would be able to download the data
                        without it registering as bandwidth-used on my server. Is this possible?

                        Comment

                        • Chung Leong

                          #13
                          Re: detect bytes written on abort

                          Well, as it's not your own server you can't really modify the binary.
                          It doesn't sound like you need too accurate a number in any event. I
                          would just do something like this:

                          <?

                          ignore_user_abo rt(true);
                          $f = fopen($filename , "rb");
                          $sent = 0;
                          while($chunk = fread($f, 1024)) {
                          echo $chunk;
                          flush();
                          if(!connection_ aborted()) {
                          $sent += strlen($chunk);
                          }
                          else {
                          /* record failure here */
                          break;
                          }
                          }

                          ?>

                          Comment

                          • Gordon Burditt

                            #14
                            Re: detect bytes written on abort

                            >> It is guaranteed that that data will be delivered to the client[color=blue][color=green]
                            >> *EVENTUALLY* or you will *EVENTUALLY* get an error. There is no
                            >> guarantee whatever that any of that data has been delivered (or even
                            >> attempted to be sent) at the time the write() returns.
                            >>[color=darkred]
                            >>> The reason why TCP can know the number of
                            >>> bytes sent with certainty is because every sent packet is replied to
                            >>> with an acknowledgment (ACK) packet.[/color]
                            >>
                            >> And the write() call *IN BLOCKING MODE* does not wait for such an
                            >> acknowledgement to be received. It doesn't even have to wait for
                            >> even one packet to be sent. It would be horribly inefficient if
                            >> you couldn't overlap, say, disk reads and network writes in a
                            >> single-threaded process that is sending a file down a socket, so
                            >> writes *DO NOT* wait until the client has received the data written.
                            >> Not even blocking writes.
                            >>[color=darkred]
                            >>> For my purposes, this number is going to be a decent approximation of
                            >>> actual bandwidth used, and I realize it's not going to be exact. Thanks.[/color]
                            >>
                            >> If you're willing to put up with, say, an extra 32k or 64k sent but
                            >> not received when the modem drops carrier, you'll get a decent
                            >> approximation. If you're expecting much better than that, you won't.
                            >>
                            >> Gordon L. Burditt[/color]
                            >
                            >Okay, I think I see what you're saying. The "write" actually writes
                            >first into internal buffers of the operating system, and so the number
                            >of bytes it returns is not necessarily the number of bytes delivered.
                            >"Write" only blocks if the internal buffers are full. So the number of
                            >bytes returned by "write" (or "send") is the number of bytes written
                            >into the TCP stack, but not the number of bytes that received ACKS
                            >indicating delivery.[/color]

                            Yes.

                            And remember, it is impossible for the end sending the file to tell
                            whether a packet it sent, or the ack for it, got dropped. Either
                            way, it looks just the same. There's always one packet full of
                            uncertainity, no matter what you do to the protocol. And one byte
                            packets are horribly inefficient.
                            [color=blue]
                            >In any case, an extra 32-64K of bandwidth used on failed connections is
                            >acceptable. I'll have to figure out a way to prevent this from becoming
                            >an issue.[/color]

                            *HOW* would this "become an issue"? Please explain *WHY* you want
                            to bill the way you have said you want to.

                            Someone needs to get better networking facilities so the problem
                            of aborted file transfers doesn't happen so often?

                            Experiment some time with this. Download something, then pull the
                            network cable or phone cable. Look at the difference in bytes saved
                            vs. what you recorded as bytes sent. Is it consistent (sort of)?
                            Perhaps you could approximate things with "if it failed, charge
                            bytes sent minus X".
                            [color=blue]
                            >Question, could someone take advantage of this by writing a network
                            >driver that requests data, but then doesn't send ACK replies after the
                            >initial request is completed?[/color]

                            Unless you're charging, and people are actually PAYING, more than
                            $5 a *byte* for this stuff, I think you have a completely overblown
                            idea of how much your data is worth. Perhaps you ought to be
                            delivering it by limousine accompanied by armed thugs who collect
                            payment before turning over the data.
                            [color=blue]
                            >My server's TCP stack would happily send
                            >out 32-64KB of data, waiting for the ACKs. The nefarious recipient
                            >would receive and save this data, drop the connection, and then request
                            >the next range of bytes. Then, they would be able to download the data
                            >without it registering as bandwidth-used on my server. Is this possible?[/color]

                            Yes. Nobody would do it if you didn't blab on the net that this
                            was how you were charging. To be practical, someone would have to
                            arrange for their TCP stack to do it only on PARTICULAR connections
                            as doing it on all of them would really screw up trying to do
                            anything on the net. (But I can see how such a hack would be
                            possible, perhaps with a custom ioctl() call to turn on such wierd
                            treatment.)

                            Now, why is your data worth going to that effort to steal? You can
                            also keep counts of failed and successful transfers, and someone
                            trying the same 5 MB file repeatedly will probably stick out like
                            a sore thumb in the statistics. Then you look closer and maybe
                            shut off the account.

                            You cannot request a range of bytes with TCP. You can with HTTP, but
                            I think that requires using headers that PHP can see and refuse to
                            deliver if you're worried about this. And the simplest way to deal
                            with this is to charge by the byte SENT. Then there's no incentive to
                            do that. In fact, there's an incentive to make connections work
                            reliably the first time.

                            A lot of people handle this by selling a particular file, at which
                            time the customer is charged for it, and he then gets as many tries
                            as he wants to download the file over the next few days (or perhaps
                            forever). There usually isn't much reason to keep downloading the
                            file over and over (unless they are having trouble), since it doesn't
                            get updated. The user has to log in, and posting his username/password
                            on the internet can be detected by massive use from all over.

                            Gordon L. Burditt

                            Comment

                            Working...