Lots of noise about user agent strings

[Richard Cornford]

Re: Lots of noise about user agent strings

Peter Michaux wrote:

On May 28, 3:37 pm, Richard Cornford wrote:

<snip>

>That illustrates why UA strings are not a viable means of
>identifying web browsers

>
If you were a system administrator and you wanted to send
gzipped JavaScript files to save bandwidth, how would you
determine which browsers could accept gzipped files and
which could not?

The HTTP Accept-Encoding header sent with the request would seem
like the obvious place to start (as that is precisely what it is
for).

I have only read explanations how to do this with the user
agent string.

Incredible, and incredibly foolish as HTTP very explicitly allows
proxies to change the encoding. That is, if a client cannot handle
gzip but the proxy can it can ask the server for gzip, decompress
it and send the identity encoded result to the client. It could
also do this the other way around, but it would be unlikely
that doing so would be seen as a good idea. And it could also
disregard any client preference for a compressed encoding and only
make identity requests to servers itself.

So a proxy may or may not send on the client's UA string or
substitute an alternative (which does not matter as the UA string
is arbitrary) and it may or may not impose the same encoding
limitations as the client. That would make looking at the UA
string at all in this context extremely foolish. Indeed more
foolish that ignoring q values in the Accept header when content
negotiating HTML/XHTML.

I have some ideas but have never tried any of them. For
example, send a gzipped file and then a non-gzipped file
to see if the first file worked. I'm curious what you
would do or if you have any experience with this area where
user agent string is used.

I have probably just answered both of those questions(?).

Richard.

[Thomas 'PointedEars' Lahn]

Re: Lots of noise about user agent strings

Peter Michaux wrote:

On May 28, 3:37 pm, "Richard Cornford" <Rich...@litote s.demon.co.uk>
wrote:

>That illustrates why UA strings are not a viable means of identifying
>web browsers

>
If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not?

One would scan the Accept-Encoding request header value for "gzip", and then
use gzip(1) or a gzip implementation to compress the message body. There
are libraries like cgi_buffer which are capable of that. Apache 2.0+ has
mod_deflate.

I have only read explanations how to do this with the user agent string.

That is a pity. Had you read RFCs 1945 and 2616 more thoroughly as I think
I recommended to you before, you would also have found the specification of
this header.


PointedEars
--
realism: HTML 4.01 Strict
evangelism: XHTML 1.0 Strict
madness: XHTML 1.1 as application/xhtml+xml
-- Bjoern Hoehrmann

[Peter Michaux]

Re: Lots of noise about user agent strings

On May 29, 4:19 pm, "Richard Cornford" <Rich...@litote s.demon.co.uk>
wrote:

Peter Michaux wrote:

On May 28, 3:37 pm, Richard Cornford wrote:

<snip>

That illustrates why UA strings are not a viable means of
identifying web browsers

>

If you were a system administrator and you wanted to send
gzipped JavaScript files to save bandwidth, how would you
determine which browsers could accept gzipped files and
which could not?

>
The HTTP Accept-Encoding header sent with the request would seem
like the obvious place to start (as that is precisely what it is
for).

I believe that the issue is that IE6 claims it can accept gzip but in
actual fact it cannot due to a decompression bug. This bug may only
apply to files over a certain size. This leads to the use of the user
agent string.

I have only read explanations how to do this with the user
agent string.

>
Incredible, and incredibly foolish as HTTP very explicitly allows
proxies to change the encoding. That is, if a client cannot handle
gzip but the proxy can it can ask the server for gzip, decompress
it and send the identity encoded result to the client. It could
also do this the other way around, but it would be unlikely
that doing so would be seen as a good idea. And it could also
disregard any client preference for a compressed encoding and only
make identity requests to servers itself.
>
So a proxy may or may not send on the client's UA string or
substitute an alternative (which does not matter as the UA string
is arbitrary) and it may or may not impose the same encoding
limitations as the client. That would make looking at the UA
string at all in this context extremely foolish. Indeed more
foolish that ignoring q values in the Accept header when content
negotiating HTML/XHTML.

Interesting. I do need to read these documents more.

[snip]

Thanks,
Peter

[Dr J R Stockton]

Re: Lots of noise about user agent strings

In comp.lang.javas cript message <4f28e817-0351-4268-928e-73f7590011ee@j3
3g2000pri.googl egroups.com>, Wed, 28 May 2008 21:27:09, RobG
<rgqld@iinet.ne t.auposted:

>
>Many people consider bandwidth to be bytes sent by the server, whereas
>it should be measured as bits transmitted by the modem after the
>addition of network stuff and compression.

Not necessarily. My pages are (almost all) small enough that, with any
reasonably recent modem, the download time itself will not upset any
possibly-significant readers. I would support efficiency of transfer,
on general grounds. But what is most important to me is the transfer
count maintained, erratically, by the server system, because there is a
monthly limit.

For any in a similar situation : I put in a ROBOTS.TXT file, denying
access to all but the Home Page INDEX.HTM. Over the course of a month
(during which I re-enabled robot access to some categories, and restored
some hidden topics), access was down by three quarters and still
dropping. I disabled ROBOTS.TXT a fortnight ago, and access had doubled
(to half the original) and is still rising.

--
(c) John Stockton, nr London UK. ?@merlyn.demon. co.uk IE7 FF2 Op9 Sf3
news:comp.lang. javascript FAQ <URL:http://www.jibbering.c om/faq/index.html>.
<URL:http://www.merlyn.demo n.co.uk/js-index.htmjscr maths, dates, sources.
<URL:http://www.merlyn.demo n.co.uk/TP/BP/Delphi/jscr/&c, FAQ items, links.

[Michael Wojcik]

Re: Lots of noise about user agent strings

Peter Michaux wrote:

On May 28, 9:27 pm, RobG <rg...@iinet.ne t.auwrote:

>On May 29, 12:22 pm, Peter Michaux <petermich...@g mail.comwrote:

>>On May 28, 3:37 pm, "Richard Cornford" <Rich...@litote s.demon.co.uk>
>>wrote:

>>
>1. modems are optimised to compress text for transmission and so
>likely compress files better (or at least no worse) than general
>purpose compression programs

That's dubious. Modem compression has to be real-time, whereas
general-purpose compression is often run out-of-band. When you gzip,
you usually don't do it while sending the data.

Consequently, modem compression (LAP-M with BTLZ, V.44, MNP-5, or
whatever) has to make different trade-offs than general-purpose
compression. The modem compression standards use smaller dictionaries
and windows than the most aggressive general-purpose LZ compressors
(eg level-9 gzip).

For that matter, the modem compression standards I'm familiar with are
not optimized for text; they're general-purpose adaptive entropy encoders.

>2. letting the modem do the work saves CPU effort at both ends

True, unless you're using a software modem (like the so-called
"Winmodems" ).

I didn't know modems do this. There must be a standard compression
algorithm to ensure the receiver knows how to decompress.

Yes, since the late 1980s or early 1990s. (I don't recall the exact
history and a trivial Google search didn't turn one up in the first
few hits.) Look up MNP-5 (an early, widely-supported proprietary
protocol), V.42bis (the first ITU standard that included compression),
BTLZ (the version of Lempel-Ziv used in V.42bis), and V.44 (a later
and more aggressive compressor).

Compression for modems came along shortly after decent synchronous
protocols (most notably LAPM, an asymmetric HDLC protocol) were
introduced, getting rid of the async framing overhead and allowing for
decent blocking of data.

>3. if the file is zipped before transmission, subsequent modem
>compression may actually result in more data to transmit (though
>likely not much more)

Any growth should be negligible. Modem compression protocols have
uncompressed modes.

Hmm. This is quite contrary to the current popular thought about
gzipping JavaScript before sending it over the wire.

Actually, it isn't, if you study the subject in a bit more depth.

First, as I explained above, out-of-band compression with
general-purpose compressors typically will yield better compression
than what a POTS modem will achieve.

Second, many people are not using POTS modems for their connections.
Sometimes they're on uncompressed LANs. Sometimes they're using
high-speed (so-called "broadband" , though that's a misnomer)
connections, like cable or DSL or FiOS. I'll admit that I haven't
looked into what kinds of compression are typically done on those
networks, but simply taking a bunch of received wisdom about POTS
modems and assuming it applies everywhere would be foolish.

Finally, precompressing the payload may have other performance
effects, because you produce smaller TCP segments. Besides saving
somewhat on TCP and IP overhead (probably negligible), you may improve
pacing (particularly if the client or server has poorly-written code
that is vulnerable to things like Nagle/Delayed-ACK Interaction),
reduce stack overhead on both ends, etc.

Steve Souders works for Yahoo!'s performance team and has made many
experiments.

There's a huge body of literature on TCP/IP performance. A handful of
experiments by "Yahoo!'s performance team" might give some decent
general guidelines, but they're not much better ground for
generalization than the "modems compress" folklore is.

The real rule is that there is no set of rules that adequately covers
all situations. If you find there's a performance problem for a
particular case, you can investigate that and often improve it; and
your improvements may result in better performance for most or all of
your users. But blanket recommendations like "compress Javascript" (or
don't) are the litanies of the cargo cults.

--
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

[Jorge]

Re: Lots of noise about user agent strings

On May 29, 4:22 am, Peter Michaux <petermich...@g mail.comwrote:

>
If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not?

Looking at the Accept-Encoding header.

[VK]

Re: Lots of noise about user agent strings

On Jun 1, 12:20 pm, Jorge <jo...@jorgecha morro.comwrote:

On May 29, 4:22 am, Peter Michaux <petermich...@g mail.comwrote:
>
>
>

If you were a system administrator and you wanted to send gzipped
JavaScript files to save bandwidth, how would you determine which
browsers could accept gzipped files and which could not?

>
Looking at the Accept-Encoding header.

In the context of the discussion I dare to question what principal
difference one sees between altering over say (Gecko) about:config
User-Agent string and altering network.http.ac cept-encoding string? If
one doesn't have a trust to one chunk of info send by agent, why so
much trust to another chunk sent in the same request? ;-)
With a reliable stats proving the point of view, please ;-)

[Richard Cornford]

Re: Lots of noise about user agent strings

Peter Michaux wrote:

On May 29, 4:19 pm, Richard Cornford wrote:

>Peter Michaux wrote:

<snip>

>>If you were a system administrator and you wanted to send
>>gzipped JavaScript files to save bandwidth, how would you
>>determine which browsers could accept gzipped files and
>>which could not?

>>
>The HTTP Accept-Encoding header sent with the request would
>seem like the obvious place to start (as that is precisely
>what it is for).

>
I believe that the issue is that IE6 claims it can accept
gzip but in actual fact it cannot due to a decompression bug.

IE 6 absolutely can accept gzip encoding, else that would have been
spotted long ago and be very well known by now.

This bug may only apply to files over a certain size.

Are we in the realm of rumour and folk-law or are there demonstrable
facts behind this assertion? Such as the precise size of the (compressed
or uncompressed) files that are supposed to be a problem, a Microsoft KB
article about it, a test case created by someone who's analytical skills
run to real cause and effect identification?

Beyond my normal cynicism, one of the reasons that I suspect this is BS
is that at work we have a QA department that delights in trying to break
our web applications (which is, after all, their job) and one of the
ways they try to do that is by overwhelming the browser with huge
downloads. The HTTPS test servers are set-up to server gziped content
when they think they can and IE 6 is certainly is the test set of
browser used so not having seen any evidence of this being a problem
suggest that it is not (or the problematic files size is so very large
that there is no real issue).

This leads to the use of the user agent string.

<snip>

But you only have to see that other browsers send default UA headers
that are indistinguishab le from that of IE 6 to know that would be a
poor approach. If you had to make an assumption based on a request
header I would probably pick on IE's unusual Accept header. How many
other browsers would be willing to accept the set of Microsoft specific
formats that IE says it would prefer (say Word and Access documents)?
And even if some other browser said it could handle that content would
those types come out with the same relative q values as in IE 6's Accept
header? That doesn't entirely solve the problem because IE's Accept
headers can be modified, but it is better than looking at something that
is known to be deliberately spoofed by other browsers.

Richard.

[Richard Cornford]

Re: Lots of noise about user agent strings

VK wrote:

On Jun 1, 12:20 pm, Jorge wrote:

>On May 29, 4:22 am, Peter Michaux wrote:
>>

>>If you were a system administrator and you wanted to send
>>gzipped JavaScript files to save bandwidth, how would you
>>determine which browsers could accept gzipped files and
>>which could not?

>>
>Looking at the Accept-Encoding header.

>
In the context of the discussion I dare to question what
principal difference one sees between altering over say
(Gecko) about:config User-Agent string and altering
network.http.ac cept-encoding string? If one doesn't have
a trust to one chunk of info send by agent, why so
much trust to another chunk sent in the same request? ;-)

Who is proposing not trussing the User Agent header? The HTTP
specification defines it as an arbitrary sequence of characters that
does not even need to be consistent over time, and so as being something
that should not be treated as a source of information. And the proposal
being made here is to trust it to be precisely what it is defined as
being; not a source of information.

With a reliable stats proving the point of view, please ;-)

"Stats" are not capable of "proving" anything.

Richard.

[VK]

Re: Lots of noise about user agent strings

On Jun 1, 10:19 pm, "Richard Cornford" <Rich...@litote s.demon.co.uk>
wrote:

Peter Michaux wrote:

On May 29, 4:19 pm, Richard Cornford wrote:

Peter Michaux wrote:

<snip>

>If you were a system administrator and you wanted to send
>gzipped JavaScript files to save bandwidth, how would you
>determine which browsers could accept gzipped files and
>which could not?

>

The HTTP Accept-Encoding header sent with the request would
seem like the obvious place to start (as that is precisely
what it is for).

>

I believe that the issue is that IE6 claims it can accept
gzip but in actual fact it cannot due to a decompression bug.

>
IE 6 absolutely can accept gzip encoding, else that would have been
spotted long ago and be very well known by now.
>

This bug may only apply to files over a certain size.

>
Are we in the realm of rumour and folk-law or are there demonstrable
facts behind this assertion? Such as the precise size of the (compressed
or uncompressed) files that are supposed to be a problem, a Microsoft KB
article about it, a test case created by someone who's analytical skills
run to real cause and effect identification?
>
Beyond my normal cynicism, one of the reasons that I suspect this is BS
is that at work we have a QA department that delights in trying to break
our web applications (which is, after all, their job) and one of the
ways they try to do that is by overwhelming the browser with huge
downloads. The HTTPS test servers are set-up to server gziped content
when they think they can and IE 6 is certainly is the test set of
browser used so not having seen any evidence of this being a problem
suggest that it is not (or the problematic files size is so very large
that there is no real issue).

As a devil advocate I would suggest to your QA department to test IE6
SP1 w/o Q837251 patch installed ;-)
That is in reference to http://support.microsoft.com/kb/837251
But if they come back victorious you may point out that users not
updating their IE or Windows for a year and half do deserve every bit
of troubles they are getting as the result.

[Michael Wojcik]

Re: Lots of noise about user agent strings

VK wrote:

On Jun 1, 2:16 pm, Lasse Reichstein Nielsen <l...@hotpop.co mwrote:

>VK <schools_r...@y ahoo.comwrites:

>>In the context of the discussion I dare to question what principal
>>difference one sees between altering over say (Gecko) about:config
>>User-Agent string and altering network.http.ac cept-encoding string? If
>>one doesn't have a trust to one chunk of info send by agent, why so
>>much trust to another chunk sent in the same request? ;-)

The premises of your argument are false. The problem with user-agent
feature detection has never been that the user-agent string is
"untrustworthy" ; there is no trust relationship between the user agent
and the server, so that attribute does not apply. User-agent feature
detection is a broken mechanism because it makes incorrect inferences,
especially false negatives that restrict UAs from receiving content
they're perfectly capable of handling.

>Because users have, or have had, reason to fake the User-Agent string

>
Correction: not users, but some browser producers.

A bogus dichotomy. The user agent is an agent of the user.

User-agent values often are set by the user; it doesn't matter what
tool enables them to do so.

The only other
cases I am aware of are experienced programming involved users
removing some data from the distributer section of User-Agent string
in IE.

It is remotely possible that your experience does not cover the entire
set of applicable cases.

And do we have a site where User-Agent spoofing would help? Like with
this string welcome, with this - go away? I mean being reasonable so
on a browser no more than 6-7 years old?

GIYF. A trivial search turned up complaints about [1], for example.
Look at the Javascript used by that page, particularly the computation
of the variables is_ie and is_nav, and how they're used in functions
like displayAll().

The Browser Wars was the fight of two: nobody cared of some 3rd or
4th.

Except the people who did, of course. And the people who cared about
standards.

It is not fair to blame on developers that they didn't account
some possible neutral 3rd parties that will come someday from
somewhere.

Oh yes it is. That's the whole point of standards and
interoperabilit y, and those have always been explicit goals for the
web, just like most other Internet applications.

[1] http://www.trader.ca/search/default....goryid=1&CAT=1

--
Michael Wojcik
Micro Focus
Rhetoric & Writing, Michigan State University

[Jorge]

Re: Lots of noise about user agent strings

On Jun 1, 9:47 pm, Thomas 'PointedEars' Lahn <PointedE...@we b.de>
wrote:

My Spanish is not good enough, so I have to use an online translator.
Trying Google Translate, this leads to:
>
| Mislead the server to receive a file. Gz with which you can not
| do anything?
| Why Ibas to want to do that?
>
Which IMHO would beg the question if you confused "receive" and "send".
Unless, of course, the translation is incorrect.  However, since Englishand
Spanish are both Indo-European languages, I would assume the common root of
"recibir" and "receive" to be of meaning.

Yes, recibir and receive mean the same thing.

You fool the server:
you send the fake Accept-Encoding: gzip request header,
you receive the answer gzipped,
yet you don't know how to deal with gzips,
you are the browser.

HTH, Thomas.

Regards,
--Jorge.

[Joost Diepenmaat]

Re: Lots of noise about user agent strings

"Richard Cornford" <Richard@litote s.demon.co.ukwr ites:

Peter Michaux wrote:

>On May 29, 4:19 pm, Richard Cornford wrote:

>>Peter Michaux wrote:

<snip>

>>>If you were a system administrator and you wanted to send
>>>gzipped JavaScript files to save bandwidth, how would you
>>>determine which browsers could accept gzipped files and
>>>which could not?
>>>
>>The HTTP Accept-Encoding header sent with the request would
>>seem like the obvious place to start (as that is precisely
>>what it is for).

>>
>I believe that the issue is that IE6 claims it can accept
>gzip but in actual fact it cannot due to a decompression bug.

>
IE 6 absolutely can accept gzip encoding, else that would have been
spotted long ago and be very well known by now.
>

>This bug may only apply to files over a certain size.

>
Are we in the realm of rumour and folk-law or are there demonstrable
facts behind this assertion? Such as the precise size of the
(compressed or uncompressed) files that are supposed to be a problem,
a Microsoft KB article about it, a test case created by someone who's
analytical skills run to real cause and effect identification?

Now this is hear-say, since I haven't dealt with the problem myself (a
coworker of me did), but there appears to be some issue with some
versions of IE6 combined with some (microsoft, IIRC) HTTP proxy server
that does indeed send out accept-encoding headers for gzip while
messing up the download (possibly the proxy doesn't send on the
encoding headers, or maybe it adds accept-encoding headers when
they're not reliable; i'm not sure). AFAIK IE6 by itself works fine,
though, and win XP SP2 or installing IE7 also seems to fix the issue,
even with a proxy server.

--
Joost Diepenmaat | blog: http://joost.zeekat.nl/ | work: http://zeekat.nl/

[Peter Michaux]

Re: Lots of noise about user agent strings

On Jun 1, 11:48 am, VK <schools_r...@y ahoo.comwrote:

On Jun 1, 10:19 pm, "Richard Cornford" <Rich...@litote s.demon.co.uk>
wrote:
>
>
>

Peter Michaux wrote:

On May 29, 4:19 pm, Richard Cornford wrote:
>Peter Michaux wrote:

<snip>

>>If you were a system administrator and you wanted to send
>>gzipped JavaScript files to save bandwidth, how would you
>>determine which browsers could accept gzipped files and
>>which could not?

>

>The HTTP Accept-Encoding header sent with the request would
>seem like the obvious place to start (as that is precisely
>what it is for).

>

I believe that the issue is that IE6 claims it can accept
gzip but in actual fact it cannot due to a decompression bug.

>

IE 6 absolutely can accept gzip encoding, else that would have been
spotted long ago and be very well known by now.

>

This bug may only apply to files over a certain size.

>

Are we in the realm of rumour and folk-law or are there demonstrable
facts behind this assertion? Such as the precise size of the (compressed
or uncompressed) files that are supposed to be a problem, a Microsoft KB
article about it, a test case created by someone who's analytical skills
run to real cause and effect identification?

>

Beyond my normal cynicism, one of the reasons that I suspect this is BS
is that at work we have a QA department that delights in trying to break
our web applications (which is, after all, their job) and one of the
ways they try to do that is by overwhelming the browser with huge
downloads. The HTTPS test servers are set-up to server gziped content
when they think they can and IE 6 is certainly is the test set of
browser used so not having seen any evidence of this being a problem
suggest that it is not (or the problematic files size is so very large
that there is no real issue).

>
As a devil advocate I would suggest to your QA department to test IE6
SP1 w/o Q837251 patch installed ;-)
That is in reference tohttp://support.microso ft.com/kb/837251
But if they come back victorious you may point out that users not
updating their IE or Windows for a year and half do deserve every bit
of troubles they are getting as the result.

This must have been it. It is good to know the issue is gone in new or
updated browsers but the general problem still exists. The server
cannot feature test the client directly (at least not easily) and does
need to rely on the strings it is sent.

Peter

[Richard Cornford]

Re: Lots of noise about user agent strings

Peter Michaux wrote:

On Jun 1, 11:48 am, VK wrote:

>On Jun 1, 10:19 pm, Richard Cornford wrote:

>>Peter Michaux wrote:
>>>On May 29, 4:19 pm, Richard Cornford wrote:
>>>>Peter Michaux wrote:

<snip>

>>>I believe that the issue is that IE6 claims it can accept
>>>gzip but in actual fact it cannot due to a decompression bug.

<snip>

>>>This bug may only apply to files over a certain size.

>>

>>Are we in the realm of rumour and folk-law or are there
>>demonstrabl e facts behind this assertion? ...

<snip>

>That is in reference to http://support.microsoft.com/kb/837251

<snip>

This must have been it. It is good to know the issue is gone
in new or updated browsers but the general problem still exists.

The Microsoft KB article asserts that the issue was introduced in a
security update for IE, and then fixed in a patch, so the issue is with
IE installations that have had some updates but are not up to date, or
non-updated installations of versions released between the introduction
of the security update and the issuing of the patch. Microsoft don't
seem very keen to let the reader know which security update introduced
the issue (so we can know the length of the interval between its release
and the patch that fixed its bugs) or the size of the downloads in
question.

The server cannot feature test the client directly (at least not
easily) and does need to rely on the strings it is sent.

But the Accept Encoding string not the User Agent string.

Richard.