Using RD Gateway on 2008 r2 works, but RD without gateway also works?

[TSiqi]

I have a dedicated 2008 r2 server with Remote Desktop Services at a datacenter. I can connect thru Remote Desktop Connections.
I wanted more security so I installed Remote Desktop Gateway. The certificate is in place and I can connect thru the Gateway and have ensured that the connection is indeed thru the Gateway.
But I can still connect to the server by Remote Desktop Connections without using the Gateway.

So the more detailed question is how do I only allow access using the Gateway?

Thanks for any help with this!!