https proxy

[Paul Sweeney]

Does anyone know of a working (python) https proxy which allows viewing of
unencrypted data being sent from my browser to an https site?

I've worked my way through most on the list at
http://xhaus.com/alan/python/proxies.html, but while many claim to support
https, if you actually point your browser at the proxies, they work fine for
http, but not for https pages.

TIA

Paul Sweeney

[Simon Dahlbacka]

Re: https proxy

Paul Sweeney wrote:[color=blue]
> Does anyone know of a working (python) https proxy which allows viewing of
> unencrypted data being sent from my browser to an https site?[/color]

hmm, I thought the _purpose_ of using https was to make it relatively
impossible to view the unencrypted data being the "man in the middle"..

[Paul Sweeney]

Re: https proxy

Simon Dahlbacka wrote:[color=blue]
> hmm, I thought the _purpose_ of using https was to make it relatively
> impossible to view the unencrypted data being the "man in the middle"..[/color]

It's certainly not impossible, there are tools like Paros for java which do
the job, the browser sets up an http connection with the proxy (using the
proxy's built in certificate), and the proxy then sets up an https
connection with the destination server, but the data is unencrypted in the
proxy before being re-encrypted to send to the destination server.

What is (virtually) impossible is to intercept and do a "man in the middle"
attack on an existing connect. I don't want to intercept stuff on the net,
just see what the browser on my machine is sending/receiving

HTH :-)

[Peter Hansen]

Re: https proxy

Paul Sweeney wrote:[color=blue]
> Simon Dahlbacka wrote:
>[color=green]
>>hmm, I thought the _purpose_ of using https was to make it relatively
>>impossible to view the unencrypted data being the "man in the middle"..[/color]
>
> It's certainly not impossible, there are tools like Paros for java which do
> the job, the browser sets up an http connection with the proxy (using the
> proxy's built in certificate), and the proxy then sets up an https
> connection with the destination server, but the data is unencrypted in the
> proxy before being re-encrypted to send to the destination server.
>
> What is (virtually) impossible is to intercept and do a "man in the middle"
> attack on an existing connect. I don't want to intercept stuff on the net,
> just see what the browser on my machine is sending/receiving[/color]

It sounds like you want either to see the raw data stream (the
encrypted stuff), or you want to see the unencrypted data that
the browser would be sending if it weren't using https. It's
still unclear. If the latter, why not use Paros, since you seem
to know about it and how it works?

(The reason your request is unclear is because your first message
talks about seeing the "unencrypte d data being sent from [your]
browser to an https site" and yet obviously, as you know, there
is no unencrypted data going to the https site... But since you
certainly know this, it makes it unclear just which you are
requesting.)

-Peter

[Paul Sweeney]

Re: https proxy

Peter Hansen wrote:[color=blue]
> Paul Sweeney wrote:[color=green]
> > Simon Dahlbacka wrote:
> >[color=darkred]
> >>hmm, I thought the _purpose_ of using https was to make it relatively
> >>impossible to view the unencrypted data being the "man in the middle"..[/color]
> >
> > It's certainly not impossible, there are tools like Paros for java which[/color][/color]
do[color=blue][color=green]
> > the job, the browser sets up an http connection with the proxy (using[/color][/color]
the[color=blue][color=green]
> > proxy's built in certificate), and the proxy then sets up an https
> > connection with the destination server, but the data is unencrypted in[/color][/color]
the[color=blue][color=green]
> > proxy before being re-encrypted to send to the destination server.
> >
> > What is (virtually) impossible is to intercept and do a "man in the[/color][/color]
middle"[color=blue][color=green]
> > attack on an existing connect. I don't want to intercept stuff on the[/color][/color]
net,[color=blue][color=green]
> > just see what the browser on my machine is sending/receiving[/color]
>
> It sounds like you want either to see the raw data stream (the
> encrypted stuff), or you want to see the unencrypted data that
> the browser would be sending if it weren't using https. It's
> still unclear. If the latter, why not use Paros, since you seem
> to know about it and how it works?
>
> (The reason your request is unclear is because your first message
> talks about seeing the "unencrypte d data being sent from [your]
> browser to an https site" and yet obviously, as you know, there
> is no unencrypted data going to the https site... But since you
> certainly know this, it makes it unclear just which you are
> requesting.)
>
> -Peter[/color]

Ok, apologies, let me try to explain it more clearly.

I wish to access an https:// site using my browser and click around it. I
wish to see the http(s) GET urls and http(s) POST urls and data which are
being sent to the https server. If the site were an http:// site I would
use a tool like ethereal to see what was going on, but this data is sent
encrypted from the browser due to the https connection, so I'll just see the
encrypted data. So I need a tool to view the unencrypted form of the data
which is being sent encrypted (sorry for being unclear about the "unecrypted
data" in my last post, I hope this is better).

I have and could use a tool like Paros, but it is java and (need I say
more?):

a) I love Python (:-D) ...
b) ... seriously, I'd like to be able to play with the source code to record
the gets and posts for later replay in a python based retriever tool.

Thanks for your interest in this thread.

Paul Sweeney

[Peter Hansen]

Re: https proxy

Paul Sweeney wrote:
[color=blue]
> I wish to access an https:// site using my browser and click around it. I
> wish to see the http(s) GET urls and http(s) POST urls and data which are
> being sent to the https server. If the site were an http:// site I would
> use a tool like ethereal to see what was going on, but this data is sent
> encrypted from the browser due to the https connection, so I'll just see the
> encrypted data. So I need a tool to view the unencrypted form of the data
> which is being sent encrypted (sorry for being unclear about the "unecrypted
> data" in my last post, I hope this is better).[/color]

Based on what you say here, I think the answer is back to being
"impossible ". The encryption is going on in the browser. If
the browser thinks it's talking HTTPS, then it will encrypt
stuff and talk HTTPS. Nothing between the browser end of the
socket and the server end of the socket will be able to
decrypt it.

If you had a proxy that used its own certificate and established
an https connection when the client tried to connect with http
to something, then you could at least start the whole thing off
without encrypted data on the browser side. As soon as you tried
to click on an embedded https link, however, the browser would
try to establish an encrypted link, and the proxy wouldn't work
any more.

What about just writing a tool using urllib2 and basically
simulating the browser activity in Python? Then you can
bypass the whole issue, and have programmatic access to the
unencrypted data before or after it hits the encrypted socket
(depending on the direction of travel).
[color=blue]
> I have and could use a tool like Paros, but it is java and (need I say
> more?):
>
> a) I love Python (:-D) ...
> b) ... seriously, I'd like to be able to play with the source code to record
> the gets and posts for later replay in a python based retriever tool.[/color]

I love Python too, but that doesn't stop me from using a more
effective tool when the need arises. Surely something like Paros,
if it can do what you need (and I don't see how it can), also
has the ability to save the data to a file for later perusal,
even if the utility for retrieving it were in Python.

-Peter

[Dietmar Schwertberger]

Re: https proxy

In article <ce5rgo$ffk$1@t itan.btinternet .com>, Paul Sweeney
<URL:mailto:rev erse.ku.oc.isso lok@nothypgnal. delrest.co.uk> wrote:[color=blue]
> Does anyone know of a working (python) https proxy which allows viewing of
> unencrypted data being sent from my browser to an https site?
>
> I've worked my way through most on the list at
> http://xhaus.com/alan/python/proxies.html, but while many claim to support
> https, if you actually point your browser at the proxies, they work fine for
> http, but not for https pages.[/color]

I wrote a http proxy server some time ago to use Netscape/Firefox for
homebanking through the modem connection of my RISC OS computer (I don't
like connecting my Windows PC to the internet....).
I don't think a simple proxy server will help you at all as it just waits
for the browser connecting, receives a "CONNECT ..." request, builds the
requested connection and then just passes on the data between the browser
and the website until the connection is closed from either side ...

Anyway it's available from my homepage; it's in the RISC OS additions
archive - the name is SSLRelay.



Regards,

Dietmar

[Paul Rubin]

Re: https proxy

"Paul Sweeney" <reverse.ku.oc. issolok@nothypg nal.delrest.co. uk> writes:[color=blue]
> I wish to access an https:// site using my browser and click around
> it. I wish to see the http(s) GET urls and http(s) POST urls and
> data which are being sent to the https server. If the site were an
> http:// site I would use a tool like ethereal to see what was going
> on, but this data is sent encrypted from the browser due to the
> https connection, so I'll just see the encrypted data. So I need a
> tool to view the unencrypted form of the data which is being sent
> encrypted (sorry for being unclear about the "unecrypted data" in my
> last post, I hope this is better).[/color]

You could do that but I don't know of already-existing Python code
that does it. Basically it would be a man-in-the-middle attack against
the browser/server pair:

browser---(SSL)---proxy---(SSL)---server

There'd be two separate SSL sessions and the proxy would of course
have to present its own server certificate to the browser, which you'd
configure the browser to accept. The server application could take
various measures to detect this subterfuge and stop it from happening,
but I doubt any normal server apps actually bother to do that. It's
not a true security break, since the browser user has to accept the
proxy certificate, thereby giving "permission " for the interception.

[John Hazen]

Re: https proxy

* Peter Hansen <peter@engcorp. com> [2004-07-27 13:55]:[color=blue]
> Paul Sweeney wrote:
>[color=green]
> >I wish to access an https:// site using my browser and click around it. I
> >wish to see the http(s) GET urls and http(s) POST urls and data which are
> >being sent to the https server. If the site were an http:// site I would
> >use a tool like ethereal to see what was going on, but this data is sent
> >encrypted from the browser due to the https connection, so I'll just see
> >the
> >encrypted data. So I need a tool to view the unencrypted form of the data
> >which is being sent encrypted (sorry for being unclear about the
> >"unecrypted
> >data" in my last post, I hope this is better).[/color]
>
> Based on what you say here, I think the answer is back to being
> "impossible ". The encryption is going on in the browser. If
> the browser thinks it's talking HTTPS, then it will encrypt
> stuff and talk HTTPS. Nothing between the browser end of the
> socket and the server end of the socket will be able to
> decrypt it.[/color]

Although, if this is just for testing, I think you could trick the
browser by running a proxy on a different machine that accepts an HTTPS
connection, and creates another HTTPS connection with the destination
server. Then run your own DNS server that hands back bogus answers to
every request resolving to the proxy's IP address.

The browser will talk HTTPS to the proxy, thinking it's talking to the
target server. Of course, this will trigger the "certificat e doesn't
match IP address" type of warning, but I think most browsers will let
you continue after warning you. (If not, you could probably still get
around it with the gross hack of having the DNS server communicate with
the proxy, so the proxy could generate on the fly a certificate for the
server name that the browser is trying to reach. This would get you the
"certificat e can't be verified" type of error, which is definitely one
that you can continue after acknowledging.)

So, if it's just for testing and research, that might work. Obviously,
it's not a viable setup if you're trying to use it to snoop on the
conversations of unsuspecting users (which I'm not accusing you of, but
it's a much harder problem).

-John

[Paul Rubin]

Re: https proxy

john@hazen.net (John Hazen) writes:[color=blue]
> Although, if this is just for testing, I think you could trick the
> browser by running a proxy on a different machine that accepts an HTTPS
> connection, and creates another HTTPS connection with the destination
> server. Then run your own DNS server that hands back bogus answers to
> every request resolving to the proxy's IP address.[/color]

You can use /etc/hosts for that. No need for running your own DNS.
But why bother? Just configure the browser send requests through the
proxy.
[color=blue]
> The browser will talk HTTPS to the proxy, thinking it's talking to the
> target server. Of course, this will trigger the "certificat e doesn't
> match IP address" type of warning, but I think most browsers will let
> you continue after warning you.[/color]

Yes, or you can generate a cert with the right hostname in it, and
install it in the browser as a trusted cert.

To the OP, are you trying to debug a server-side app? If you're not
trying to send actual confidential data through the SSL connection and
if you can configure the server, it's probably simplest to just open a
non-SSL port, unless you're trying to debug something SSL-specific.

[Paul Sweeney]

Re: https proxy

Paul Rubin wrote:[color=blue]
> john@hazen.net (John Hazen) writes:[color=green]
> > Although, if this is just for testing, I think you could trick the
> > browser by running a proxy on a different machine that accepts an HTTPS
> > connection, and creates another HTTPS connection with the destination
> > server. Then run your own DNS server that hands back bogus answers to
> > every request resolving to the proxy's IP address.[/color]
>
> You can use /etc/hosts for that. No need for running your own DNS.
> But why bother? Just configure the browser send requests through the
> proxy.[/color]

agreed, no need to involve DNS, just set as proxy in browser connection
settings
[color=blue]
>[color=green]
> > The browser will talk HTTPS to the proxy, thinking it's talking to the
> > target server. Of course, this will trigger the "certificat e doesn't
> > match IP address" type of warning, but I think most browsers will let
> > you continue after warning you.[/color]
>
> Yes, or you can generate a cert with the right hostname in it, and
> install it in the browser as a trusted cert.[/color]

This is not a problem since it is ok to have to click on the certificate
warning box that will come up in the browser.
[color=blue]
>
> To the OP, are you trying to debug a server-side app? If you're not
> trying to send actual confidential data through the SSL connection and
> if you can configure the server, it's probably simplest to just open a
> non-SSL port, unless you're trying to debug something SSL-specific.[/color]

Unfortunately I don't have access to the target server, so this is not an
option (see you other post to this thread for a more description of the
problem)

Thanks

[Paul Sweeney]

Re: https proxy

Paul Rubin wrote:[color=blue]
> "Paul Sweeney" <reverse.ku.oc. issolok@nothypg nal.delrest.co. uk> writes:[color=green]
> > I wish to access an https:// site using my browser and click around
> > it. I wish to see the http(s) GET urls and http(s) POST urls and
> > data which are being sent to the https server. If the site were an
> > http:// site I would use a tool like ethereal to see what was going
> > on, but this data is sent encrypted from the browser due to the
> > https connection, so I'll just see the encrypted data. So I need a
> > tool to view the unencrypted form of the data which is being sent
> > encrypted (sorry for being unclear about the "unecrypted data" in my
> > last post, I hope this is better).[/color]
>
> You could do that but I don't know of already-existing Python code
> that does it.[/color]

Rats, that would seem to be the consensus on this, the nearest is the http
proxies which allow https to tunnel through them leaving the data encrypted.
At least they would do if this feature of any of them actually worked ;-)
[color=blue]
> Basically it would be a man-in-the-middle attack against
> the browser/server pair:
>
> browser---(SSL)---proxy---(SSL)---server
>
> There'd be two separate SSL sessions and the proxy would of course
> have to present its own server certificate to the browser, which you'd
> configure the browser to accept. The server application could take
> various measures to detect this subterfuge and stop it from happening,
> but I doubt any normal server apps actually bother to do that. It's
> not a true security break, since the browser user has to accept the
> proxy certificate, thereby giving "permission " for the interception.[/color]

A perfect description of the problem and possible solutions, thanks Paul,
glad you could make this out from my descriptions :-D

This is exactly what the java Paros tool does, but I'd like to have a Python
version which I can customise to do the things I need automatically. I'm
working on an automatic testing framework written in python which can
perform scripted actions on a target system, and want an easy way to
determine the urls of GETs and the urls (including query strong) and data of
POSTS performed under https connections. In order to replay them later. I'm
doing it with the target server owners permission, but don't have access to
fiddle with server configuration settings etc.

Just to emphasise again, I'm only trying to see the data that my browser is
sending/receiving, *not* intercept someone elses connection.

Any posts for suggested python software gratefully received. If I end up
writing one of these, I'll try to get agreement from my employer for it to
be open sourced since there seems to be an opening for such a tool, even if
there isn't much of a market for it!

[Paul Sweeney]

Re: https proxy

Dietmar Schwertberger wrote:[color=blue]
> In article <ce5rgo$ffk$1@t itan.btinternet .com>, Paul Sweeney
> <URL:mailto:rev erse.ku.oc.isso lok@nothypgnal. delrest.co.uk> wrote:[color=green]
> > Does anyone know of a working (python) https proxy which allows viewing[/color][/color]
of[color=blue][color=green]
> > unencrypted data being sent from my browser to an https site?
> >
> > I've worked my way through most on the list at
> > http://xhaus.com/alan/python/proxies.html, but while many claim to[/color][/color]
support[color=blue][color=green]
> > https, if you actually point your browser at the proxies, they work fine[/color][/color]
for[color=blue][color=green]
> > http, but not for https pages.[/color]
>
> I wrote a http proxy server some time ago to use Netscape/Firefox for
> homebanking through the modem connection of my RISC OS computer (I don't
> like connecting my Windows PC to the internet....).
> I don't think a simple proxy server will help you at all as it just waits
> for the browser connecting, receives a "CONNECT ..." request, builds the
> requested connection and then just passes on the data between the browser
> and the website until the connection is closed from either side ...
>
> Anyway it's available from my homepage; it's in the RISC OS additions
> archive - the name is SSLRelay.[/color]

Thanks for posting a response Dietmar.

It sounds like this works the same as the http proxies in my OP so it won't
unencrypt the data to make it viewable, but then it doesn't need to for your
app :-)

In case someone else is reading this and it sounds useful to them, I
couldn't figure the url of your website, so you may want to repost with it.

Paul

[Alan Kennedy]

Re: https proxy

[Paul Sweeney]
[color=blue][color=green]
>>I wish to access an https:// site using my browser and click around
>>it. I wish to see the http(s) GET urls and http(s) POST urls and
>>data which are being sent to the https server. If the site were an
>>http:// site I would use a tool like ethereal to see what was going
>>on, but this data is sent encrypted from the browser due to the
>>https connection, so I'll just see the encrypted data. So I need a
>>tool to view the unencrypted form of the data which is being sent
>>encrypted (sorry for being unclear about the "unecrypted data" in my
>>last post, I hope this is better).[/color][/color]

[Paul Rubin]
[color=blue]
> You could do that but I don't know of already-existing Python code
> that does it. Basically it would be a man-in-the-middle attack against
> the browser/server pair:
>
> browser---(SSL)---proxy---(SSL)---server
>
> There'd be two separate SSL sessions and the proxy would of course
> have to present its own server certificate to the browser, which you'd
> configure the browser to accept. The server application could take
> various measures to detect this subterfuge and stop it from happening,
> but I doubt any normal server apps actually bother to do that. It's
> not a true security break, since the browser user has to accept the
> proxy certificate, thereby giving "permission " for the interception.[/color]

Paul S,

Perhaps you could achieve what you want with Apache and mod_proxy?

http://httpd.apache.org/docs/mod/mod_proxy.html

Use (a HTTPS-configured) Apache to decode the requests with mod_proxy,
and simply forward all requests to another plain (HTTP) python proxy
of your choice. The ProxyRemote or ProxyPass directives could be used
for that.

On looking at the ProxyRemote documentation, it states "only 'http' is
supported by this module.'

However, a quick search turns up the following snippet, i.e. that
Apache mod_ssl provides HTTPS support for mod_proxy.

From the horses mouth, i.e. Ralf Engelschall: "when you want to
forward to a HTTPS target you need HTTPS support in mod_proxy.
That's only possible with mod_ssl (which enhanced mod_proxy for
HTTPS). Then you can do all: Accept HTTP and HTTPS and connect
to HTTP and HTTPS targets, i.e. create gateways in all
combinations."

http://www.lists.aldigital.co.uk/apache-ssl/msg02860.html

HTH,

--
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/contact/alan

[Alan Kennedy]

Re: https proxy

[Alan Kennedy]
[color=blue]
> Perhaps you could achieve what you want with Apache and mod_proxy?
>
> http://httpd.apache.org/docs/mod/mod_proxy.html[/color]

Just realised that the Apache 2 documentation contains more
information relating to proxying HTTPS content:-

http://httpd.apache.org/docs-2.0/mod/mod_proxy.html

HTH,

--
alan kennedy
------------------------------------------------------
check http headers here: http://xhaus.com/headers
email alan: http://xhaus.com/contact/alan