During the last weeks I have been assessing how Mobile operators mangle HTTP connections and today I start a series of posts to present the results.
One of the main differences between mobile and fixed operators is that the first have been providing the so called Valued Added Services (VAS), while the second, in most cases, are just fat pipes. The main VAS enablers are the former Wireless Application Protocol Gateways (WAP GW).
Ten years ago, WAP GW had the important role to convert the efficient and compact WTP/WSP to the standard and broadly used TCP/HTTP. Nowadays, when all (relevant) devices support WAP2.0, WAP GW are not much different from any enterprise Proxy. The main difference is that in order to support the VAS, WAP GW enrich the HTTP headers. There are different flavors of header enrichment: on one hand the most common behavior includes adding a user ID, like the mobile phone (MSISDN) for dedicated internal services allowing the VAS to identify the user, and most probably charge him/her. On the other, some operators would modify the HTTP headers for different reasons. Also some others would add extra information like bearer type, Proxy ID, device real IP, etc.
The methodology for the tests is simple. I have several GPRS/UMTS modems with the SIM cards of the operators that I want to analyse. I created a script that would initiate a connection through a given modem from one of the mobile operators and access a server I own that replies with the HTTP headers that reached the server. As I know the headers I input and the ones that reach the server I can compare them and note the results.
When doing the connection to the operator I do it in two ways. First I would use the so called WAP connection and connect trough the WAP GW. In this case I´m simulating a standard mobile phone like most Nokia, Motorola, Samsung, Sony Ericcson.... In the second connection I would use the so called WEB or INTERNET connection, simulating an Android, iPhone or standard PC with a data card. Although most people think that in the second case the user is accessing the internet directly, without going through a proxy we will see that it is not the case. Since the advent of the Androids and iPhones, which do not support an operator proxy, mobile operators have deployed transparent proxies (aka intercepting proxies) on the "Internet" connection, forcing also these devices to go trough them.
Having said that, lets take a look at some real examples from the french mobile operators: Orange, SFR and Bouygues. First of all, these are the original headers of my script:
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1
whereas when using the direct connection the headers were the same but the User Agent was:
User-Agent: HeaderValidator/1.1-dir
Below I add the headers as received in the server for the different connections. In red I highlight the unexpected changes, while in orange I marked the ones that where understandable for a proxied connection.
=== Orange France through WAP GW/Proxy ===
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1
X-Nokia-CONNECTION_MODE: TCP
X-NSN-PROTOCOL-TYPE: WAP2.0
X-Nokia-BEARER: GPRS
X-Nokia-PLTF: NBG3.1.3_PP
X-Wisp: wisp2
Via: 1.1, 1.1 proxy (proxy)
Cache-Control: max-age=259200
Connection: keep-alive
=== Orange France direct connection (INTERNET) ===
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1-dir
X-Wisp: wisp2
Via: 1.1 proxy (proxy)
Cache-Control: max-age=259200
Connection: keep-alive
=== SFR through WAP GW/Proxy ===Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1
X-vfprovider: SFR
X-vfstatus: 10
X-Nokia-BEARER: GPRS
serviceControlInfo: 18+=-
x-nokia-gid: 259107281784650
X-Nokia-CONNECTION_MODE: TCP
X-Nokia-gateway-id: NWG/4.1/Build79
X-Nokia-ipaddress: 10.187.207.132
Via: 1.1 proxy.cwg.net (proxy)
X-Forwarded-For: 10.187.207.132, 10.187.207.132
Cache-Control: max-age=259200
Connection: Keep-Alive
Pragma: no-cache
X-BlueCoat-Via: 28409D05D072730C
=== SFR direct connection (INTERNET) ===
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1-dir
X-vfprovider: SFR
X-vfstatus: 10
X-Nokia-BEARER: GPRS
serviceControlInfo: 18+=-
x-nokia-gid: 259107281784650
X-Nokia-CONNECTION_MODE: TCP
X-Nokia-gateway-id: NWG/4.1/Build79
X-Nokia-ipaddress: 10.205.32.74
Via: 1.1 proxy.cwg.net (proxy)
X-Forwarded-For: 10.205.32.74, 10.205.32.74
Cache-Control: max-age=259200
Connection: Keep-Alive
Pragma: no-cache
X-BlueCoat-Via: DFF4C47D046B9013
=== Bouygues through WAP GW/proxy ===
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
Host: m.mobiclip.net
User-Agent: HeaderValidator/1.1
=== Bouygues direct connection (INTERNET) ===
TE: deflate,gzip;q=0.3
Connection: TE, close
Accept: text/html,text/vnd.wap.wml,application/vnd.wap.html+xml,application/xhtml+xml,application/vnd.wap.xhtml+xml,text/x-wap.wml,text/x-hdml,text/vnd.sun.j2me.app-descriptor,application/java-archive,application/octet-stream,,image/png,image/gif,image/jpg,image/jpeg,*/*
Accept-Charset: iso-8859-1,utf-8
Accept-Language: en-us,en;q=0.5
User-Agent: HeaderValidator/1.1-dir
In France we can see there is one operator, Bouygues that preservers the HTTP headers intact, probably having a better performance than SFR and Orange which add a lot of useless information.
Looking at the "new" headers added by Orange and SFR, the first question that arises is, do we really need to know the information below? What for?
X-NSN-PROTOCOL-TYPE: WAP2.0
X-Nokia-PLTF: NBG3.1.3_PP
X-Wisp: wisp2
Via: 1.1, 1.1 proxy (proxy)
X-vfprovider: SFR
X-vfstatus: 10
x-nokia-gid: 259107281784650
X-Nokia-CONNECTION_MODE: TCP
X-Nokia-gateway-id: NWG/4.1/Build79
X-Nokia-ipaddress: 10.205.32.74
Via: 1.1 proxy.cwg.net (proxy)
X-Forwarded-For: 10.205.32.74, 10.205.32.74
X-BlueCoat-Via: DFF4C47D046B9013
What is the purpose of wasting GW memory and CPU cycles adding this? Do we really need to know our device uses WAP2.0 or TCP? Or the Nokia GW ID or the Blue-Coat proxy ID? What for? As we will see in the future posts the variety of data across the main mobile operators is so big that we can conclude that this HTTP noise can´t be of use for anyone.
The surprising part is that, especially in the case of SFR, all the headers are also added by the transparent proxy in Internet connection. While in the case of Orange we see that the number of headers has been dramatically reduced.
I would assume that SFR is adding the transparent proxy "in front" of the standard Nokia GW, routing all Internet connections through it and later through the BlueCoat. Good for Nokia, as that sounds like they would need a lot of HW and licenses!!. While Orange France seams to follow a more sensible approach redirecting the Internet connections to the "WISP2" proxy which might be in charge of the useful header enrichment.
That´s all for now. Next time we´ll have a look at how the Italian Mobile Operators deal with the header enrichment.
Let's Kerberos
7 months ago
Thank you for your article.
ReplyDeleteI'm using Orange in france and encounter problems with synchronization with Exchange due of proxy.
When i use a different access point internet-entreprise instead of orange it works well.
So proxy behaviour is different with access point "orange" and access point "internet enterprise".
this is good picture
ReplyDelete19 This is very smart tutorial...thank you for sharing
This whole research is amazing! Is there a way to contact you? I have an operator-related question that I've been desperately trying to find an answer to.
ReplyDeleteHi,
ReplyDeleteyou can contact me at xufxuf @ gmail . com
Thank you for your article.
ReplyDeleteWhen i traceroute from Orange 3G, i go through ips like 10.187.1.x, 10.187.3.x and so on even if i have a public ip address like 90.94.x.y ??
ReplyDeleteI have the same behaviour even on APN internet-entreprise with a Business Everywhere contract.
Is that even possible ? What's the explanation ? I go through a proxy ?
That is perfectly normal and you can see the same behaviour in severla ISPs. Internal router might have private IPs. As long as the edge routers of the ISP know you are beyond the 10.187.1.x packets will flow from the external sites to your device and the other way arround. A traceroute can´t be used to detect transparent proxys.
DeleteThanks for the explanation, much appreciated.
DeleteDo you think going through a 10.x network which is private can impede vpn connectivity when the LAN you're trying to connect to has a 10.x address plan too ?
Using native mobile vpn clients through orange 3G, i can connect to a l2tp/ipsec or even pure ipsec vpn so it seems ok but everytime i try to transfer a file via ftp, the transfer stalls after few seconds and timeout occurs after a while.
Is it due to the double connexions (control/data) nature of FTP or some NAT problem instead or Orange causing trouble ?
Thanks for the tips
If your device has a public IP, which most probably is the case when using the "Internet APN", I don´t think you are being NATed. The fact your packets go through routers with private IPs should be transparent. FTP Active mode is a hassle always as it requires the server to open a connection to the client. Perhaps Orange do not allow externally initiated connections. Try using Active mode instead. More info: http://slacksite.com/other/ftp.html
DeleteThanks again for bearing with me. I check and yes i have both a public Orange address ala 90.94.x.Y (often) checked by local ip tool and whatismyip.fr (gives me back same ip address so no proxy i guess) and a private vpn address. I use ftp passive mode same result.
ReplyDeleteTo be more specific, ftp goes just fine when i connect via Orange 3G internet-entreprise without vpn to a public ftp site and put or retrieve a file. But the ftp server endpoint address is not in the 10.x range because it's a public ftp site.
NOw, the data connection does only stall when i use vpn (through Orange 3G APN internet-entreprise) to connect to a private ftp server located on our remote LAN with 10.x address hence my question about going through a 10.X Orange private network to end up via vpn to my remote 10.x LAN causing problems ?
Single connection traffic like surfing the intranet, pop , smtp and synchronizing sql ce database work fine through Orange internet-entreprise and our vpn but maybe it's our vpn server which is bugging. Bye
To close the topic, i solved my problem, it's the l2tp ipsec client which is bugged on my ES400 motorola. I tried it on another Windows mobile device (HTC Touch pro 2) and it works on internet-entreprise despite having Orange's 10.x addresses along the way and my target LAN on a 10.x subnet too.
ReplyDeleteSo nothing to do with Orange...