Introducing the Netalyzr Android app
CACM article on "bufferbloat"
A new release, celebrating 400,000 sessions!
Power outage at ICSI
Network outage at ICSI
Updated analysis of Paxfire-related search hijackings
Netalyzr co-wins the FCC's Open Internet Research Challenge
Netalyzr reveals ISPs hijacking users' web search queries
Does your operator leak your private data?
Over the course of 2014 we observed more than 27,000 installations of our Netalyzr Android app. Thank you for your support! Your Netalyzr sessions allowed us to study more than 450 mobile operators in 128 countries. Our analysis has revealed widespread deployment of HTTP proxies in cellular networks. Nearly 60% of Netalyzr sessions running on cellular networks encountered proxies, compared to a mere 14% of the sessions run on wired and WiFi sessions.
Cellular network operators generally enforce the use of proxies on customer traffic. This means that any privacy violation, security vulnerability, or performance degradation caused by these proxies affects the proxy's entire user base. We have noticed several examples of severe privacy violations (some of which widely reported in the press) as well as practices that affect service quality, such as image transcoding and content modification.
In this article we focus on the practice of HTTP header injection by proxies, used by a wide range of mobile operators. HTTP headers convey meta-information about a data transfer and remain invisible to the user. The following table summarizes the set of sensitive HTTP headers we have identified.
|Header||Purpose and Implications|
|LBS-EventTime||Unknown timestamp. Probably related to location-based services.|
|LBS-ZoneID||Unknown purpose. Probably related to location-based services.|
|MSISDN||Subscriber phone number.|
|x-up-3gpp-imeisv||Device IMEI. Identifies the device uniquely.|
|x-up-nai||Email-like name that identifies the user and operator.|
|x-up-calling-line-id||Subscriber phone number.|
|x-up-vodacomgw-subid||Subscriber phone number.|
|Network-related and operational headers|
|o2gw-id||Unique identifier of the client's GSM gateway.|
|Proxy-Connection||Experimental header created by Netscape to force persistent connections. Ignored by most servers.|
|vf-za-trust||Private IP address of the subscriber.|
|WAP-Connection||Defines application-layer protocol. e.g., Stack-Type=HTTP.|
|WISP-A||Unknown. Probably related to Wireless ISP Roaming, defining roaming agreements between operators.|
|X-BlueCoat-Via||Unique identifier for a Bluecoat proxy.|
|X-EE-Client-IP||Private IP address of the subscriber.|
|X-Forwarded-For||Private IP address of the subscriber.|
|x-gateway||The GSM gateway and its location.|
|X-Nokia-BEARER||Nokia-specific header. Reports the cellular technology providing network to the customer: e.g., GPRS, UMTS, etc.|
|X-Nokia-CONNECTION_MODE||Nokia-specific header. Defines transport-layer protocol (e.g., TCP or UDP).|
|X-Nokia-Gateway-Id||Nokia-specific header. Identifies the model and version of the gateway.|
|X-nokia-gid||Nokia-specific header. Identifies the model and version of the gateway.|
|X-Nokia-ipaddress||Nokia-specific header. Private IP address of the subscriber.|
|X-Nokia-MaxDownlinkBitrate||Nokia-specific header. Defines downlink bitrate for the client.|
|X-Nokia-MaxUplinkBitrate||Nokia-specific header. Defines uplink bitrate for the client.|
|X-Operator-Domain||Operator domain name.|
|X-Orange-RAT||Unknown. Integer value. May be used to define the Radio Access Type.|
|X-Orange-Roaming||Reports if the device is roaming or not.|
|X-Proxy-ID||Unique identifier of the proxy.|
|x-up-3gpp-rat-type||Unknown. Integer value. May be used to define the Radio Access Type.|
|x-up-3gpp-sgsn-mcc-mnc||Identifies the mobile operator and the GSM gateway.|
|x-up-bearer-type||Reports the cellular technology providing network to the customer: e.g., GPRS, UMTS, etc.|
|x-up-forwarded-for||Likely reports the private IP address of the subscriber.|
|x-up-sgsn-ip||IP address of the SGSN.|
|X-vfstatus||Integer value. Unknown.|
|x-vodafone-3gpdpcontext||Unknown. May be used to define the state of the PDP context.|
|x-vodafone-roamingind||MCC and MNC of the operator.|
|X-Wisp||Unknown. Probably related to Wireless ISP Roaming, defining roaming agreements between operators.|
The next table shows for each country and operator which interesting HTTP headers we have observed, and during which timespan.
|Operator||Header||First seen||Last seen|
|United Arab Emirates|
The type of headers added by each operator varies. The more worrisome cases uniquely identify the subscriber, their location, their IP address, and even their phone number. Some headers, particularly X-ACR (by AT&T), X-VF-ACR (by Vodafone Netherlands and Vodacom South Africa) and X-UIDH (by Verizon), contain operator-generated UUIDs (unique identifiers) to identify the subscriber. Such tracking-enablers (or "perma-cookies") are associated with advertising programs deployed by mobile operators. This article explains one such program, Verizon Selects. Until recently AT&T had a similar scheme. These programs allow the operators' partner companies to uniquely identify subscribers and monetize mobile subscriber information, e.g. via more targeted advertisements. Since the proxies inject these headers into each HTTP request users issue and any web server can record them, third-party content providers hooked into the visited website can likewise use the information to track users. While AT&T stopped adding these headers soon after the media exposure, Verizon still continues the practice at the time of this writing.
The problem is not constrained to the US. Operators such as Ooredoo in Algeria and Orange in Jordan directly append the MSISDN (the phone number stored in the SIM card) to each HTTP request. Depending on operator-defined settings, Nokia's proxies leak also this information, as seen in the mobile operator Smart (Philippines) using the header X-NOKIA-MSIDN.
Vodacom in South Africa (part of Vodafone group) appears to be the most egregious case of privacy violation. While we currently possess only one session from this provider, the above table summarizes the plethora of tracking headers they employ. We have identified the IMEI, a unique identifier of the user's device, in the x-up-3gpp-imeisv header, the subscriber's phone number in the x-up-calling-line-id and x-up-vodacomgw-subid headers, and an email-like unique name of form email@example.com in the x-up-nai header.
Operators such as T-Mobile in Germany, EE (Everything Everywhere) and O2 in the UK, Vodacom in South Africa, and SFR in France also reveal other aspects of the client-side infrastructure. Proxies widely use headers such as X-Forwarded-For (which is part of the HTTP standards), X-EE-Client-IP, or vf-za-trust to report the IP address devices use on their uplink. Headers such as x-gateway and o2gw-id (used by O2 and its MVNO GiffGaff in the UK) also include the client's location. While these headers might not compromise user security, they could serve as substitutes for user-visible HTTP cookies. We have also observed proxies explicitly defining the maximum uplink and downlink speed for a given device. Pelephone in Israel uses X-Nokia-MaxUplinkBitrate and X-Nokia-MaxDownlinkBitrate for this purpose. TIM in Italy adds Proxy-Connection to force persistent connections in an attempt to replace the Connection header.
Finally, many operators report the software vendor of their proxy to the external world. We observe that Bluecoat provides proxies to Vodafone Qatar, SFR in France, and 3 in Ireland. These add X-BlueCoat-Via headers to inform endpoints of the proxy's presence, replacing the standard Via header. Other operators such as EE in the UK, SFR in France (as well as MVNOs running on top, such as Leclerc and Prixtel), and Pelephone in Israel also reveal the hardware vendor, proxy model, build version, and the cellular technology providing the service to the device with the headers X-Nokia-Gateway-Id and X-Nokia-BEARER.
HTTP header injection can compromise privacy without the user's knowledge. A recent Wired article recommends Netalyzr as a way to identify such leaks, and we encourage users to periodically use our service to check whether your operator leaks unwanted information about you.
As of today, the Netalyzr results summary explicitly alerts you to the presence of HTTP headers that proxies insert into your web traffic and that we deem onerous. Look for "Sensitive proxy-introduced HTTP headers" in the HTTP results section.
If you find leakage affecting you, consider contacting your operator's technical support; some operators allow subscribers to opt-out from advertising programs, as in the case of Verizon, and in other cases, customer complaints and media exposure can generate effective pressure to cease such practices.
You can find more information about ISP HTTP and DNS manipulations, and different business relationships between mobile operators, in our tech report. Download Netalyzr for Android for free (and ad-free) from Google Play. We welcome your feedback and suggestions.
As always, thanks for your support!
Introducing the Netalyzr Android app
Today we're very happy to announce the release of Netalyzr for Android.
The Android app works for WiFi and cellular connections and includes the same spectrum of tests that you know from our applet and command-line interface. It also provides configurability of the tests conducted and lets you conveniently access and share the results of sessions previously conducted on the device.
We have big plans for the app and welcome your feedback and suggestions. As always, thanks for your support!
CACM article on "bufferbloat"
Recently ICSI researcher and Netalyzr co-developer Nick Weaver participated in a disussion with Vint Cerf, Van Jacobson, and Jim Gettys of bufferbloat, the tendency of network equipment to employ excessively large buffers that impose unnecessary latencies on network traffic. Netalyzr measures these buffer sizes, and the findings we reported in our IMC'10 paper have attracted significant interest in the community.
The roundtable discussion formed the basis of a feature in the February edition of the Communications of the ACM. The resulting article, BufferBloat: What's Wrong With the Internet?, is now online.
We have just released a substantial Netalyzr update. First and foremost, we've made Netalyzr a whole lot faster. After analyzing the test durations experienced by our users, we increased parallelization in test executions and optimized a number of timeouts, all with the goal of reducing execution time. Netalyzr now runs approximately twice as fast and a session typically completes in under two minutes.
We have also improved the level of control you have over the amount of information displayed on the test results page. You can now expand and collapse details of individual tests, instead of entire test categories. In the initial results display, this ability lets us show details only for tests that exhibited irregularities or problems, while keeping all other results collapsed. See the help page for more information.
Several users reported that Netalyzr's bandwidth test at times crashes their DSL modems, causing a temporary loss of network connectivity. This outage could cause Netalyzr sessions to stall at the result upload stage. We now attempt result uploads several times while waiting for connectivity to resume. You may have noticed that Netalyzr tries to use the UPnP protocol to identify the model and manufacturer of your gateway device. In the future, this information, combined with tracking outages induced by bandwidth tests, will allow us to identify particularly fragile devices.
For researchers we now provide better ways to label Netalyzr sessions and obtain the resulting set of result data. Netalyzr has long provided a command-line interface, but no convenient way to obtain test results for subsequent processing. In addition to the regular HTML summary we now also provide the session results in JSON format. To obtain a session's results in JSON, use this URL schema:
Here's our example session in JSON. Note that we're still in the process of improving the details reported for individual tests. We'd love to hear your feedback on this data format—discuss it with us on our mailing list.
If you're a researcher providing customized ways for people to run Netalyzr, you can now arrange to have all resulting sessions automatically tagged with a string identifying your experiment and retrieve the list of resulting sessions in near-real-time, likewise in JSON format. If you're interested in this feature, please contact us.
As we're approaching the 500,000 session mark, we'd once again like to thank our users for their ongoing support and use of the Netalyzr service. We couldn't do this without you!
A new release, celebrating 400,000 sessions!
Today we pushed out a round of updates to the Netalyzr codebase. Most importantly, we dropped the HTTP content test that downloads the EICAR test virus. Even though this file is completely harmless, some anti-virus systems take it just as seriously as real malware. When detecting the file, these systems cut off all of Netalyzr's connectivity, causing a failure of the test session to complete. Many thanks to our users for alerting us to this problem—it's a great example of the unexpected things we encounter during the tests.
In terms of new features, we have added initial testing of DNS root server behavior. Netalyzr now checks whether it can actually reach all DNS root servers and whether querying them works as intended. We have also beefed up our regression testing and as a consequence fixed a number of result rendering glitches in the test reports.
A few weeks ago the Netalyzr session counter crossed the 400,000 mark. We would like to take this opportunity for a sincere Thank You to all of our users for continuing to run Netalyzr! We always welcome your feedback and suggestions at firstname.lastname@example.org.