By Guy Podjarny

SPDY is awesome. It’s the first real upgrade to HTTP in 10+ years, it tackles high latency mobile networks performance issues and it makes the web more secure. SPDY is different than HTTP in many ways, but its primary value comes from being able to multiplex many requests/responses from client to server over a single (or few) TCP connections.

Previous benchmarks tout great benefits, ranging from making pages load 2x faster to making mobile sites 23% faster using SPDY and HTTPS than over clear HTTP. However, when testing real world sites I did not see any such gains. In fact, my tests showed SPDY is only marginally faster than HTTPS and is slower than HTTP.

Why? Simply put, SPDY makes HTTP better, but for most websites, HTTP is not the bottleneck.

The Bottom Line

If you don’t have time to read the full details, here’s the quick summary.

I tested the top 500 websites in the US (per Alexa), measuring their load time over HTTPS with and without SPDY, as well as over HTTP. I used a Chrome browser as a client, and proxied the sites through Cotendo to control whether SPDY is used. Note that all the tests – whether HTTP, HTTPS or SPDY – were proxied through Cotendo, to ensure we’re comparing apples to apples.

The results show SPDY, on average, is only about 4.5% faster than plain HTTPS, and is in fact about 3.4% slower than unencrypted HTTP. This means SPDY doesn’t make a material difference for page load times, and more specifically does not offset the price of switching to SSL.

I started this test because I found previous tests to be bad representations of the real world. This test is therefore different in several ways:

-> I only enabled SPDY for 1st party content.
Website owners don’t control 3rd party domains and how they’re delivered.
-> I combined 1st party domains, but not 3rd party domains.
Most previous tests flattened the page into a single domain by creating static copies of pages, which is an artificial environment where SPDY
thrives.
-> I did not use a client-side proxy, but rather reverse-proxied the website.
Using a client side proxy again creates one client/proxy connection where all requests are multiplexed, which is beneficial to SPDY but not
realistic.
-> I tested real world websites, with all their warts.
This includes many domains on the page, unoptimized page, inefficient backends, etc. Most other data I’m aware of is either from the highly
optimized Google websites or from static copies of websites, which eliminates many real world bottlenecks.
I’ll let you decide if these differences make it a better test or a worse test, but it helps understand why the results are different.

There could be many reasons why SPDY does not help, but the two that stand out are:

1. Web pages use many different domains, and SPDY works per domain. This means SPDY can’t reduce connections or multiplex requests across the different domains (with some exceptions), and its value gets diminished.
2. Web pages have other bottlenecks, which SPDY does not address. For example, SPDY doesn’t prevent scripts from blocking downloads of other resources, nor does it make CSS not block rendering. SPDY is better than HTTP, but for most pages, HTTP is not the bottleneck.

The Test

For this experiment, I needed a set of websites to test, a client that supports SPDY and reverse-proxy that support SPDY.

For the websites, I chose the top 500 websites in the US, as defined by Alexa. The percentage of porn sites on that list is a bit alarming, but it’s a good representation of websites users browse often.

For a proxy, I used the Cotendo CDN (recently acquired by Akamai). Cotendo was one of the early adopters of SPDY, has production-grade SPDY support and high performance servers. Cotendo was used in three modes – HTTP, HTTPS and SPDY (meaning HTTPS+SPDY).

For a client, I used WebPageTest’s Chrome agent (with Pat Meenan’s help). WebPageTest automats a real Chrome browser (version 18 at the time of my tests), and through that supports SPDY. Note that Chrome randomly disables SPDY on 5% of browser runs, but WebPageTest disables this sampling. I measured each page 5 times, over 4 different network speeds, including Cable, DSL, low-latency mobile and high latency mobile.

Since some websites use multiple 1st party domains, I also used some Akamai rewriting capabilities to try and consolidate those domains. Roughly speaking, most resources statically referenced in the HTML were served through the page’s domain. This helped enable SPDY for those resources and consolidate some domains.

Lastly, since time of day and Internet events can skew results, I repeated the test 3 times, twice during the day and once overnight. In total I ran 90,000 individual page loads, or 30,000 per mode, more than enough for statistical accuracy.

The Results

The main result was that SPDY didn’t make the websites faster. Many different views of the data repeated this result:

>> SPDY was only 4.5% faster than HTTPS on average
>> SPDY was 3.4% slower than HTTP (without SSL) on average
>> The median SPDY acceleration over HTTPS was 1.9%
>> SPDY was faster than HTTPS in only 59% of the tests
>> SPDY is only 2.1% faster than HTTPS when comparing the average load time of each URL/scheme, across batches and network speeds
>> SPDY’s acceleration over HTTPS was 4.3%, 6.3% and 2.8% in each of the three test batches

See Summary in Pic

Link: http://www.guypo.com/technical/not-as-spdy-as-you-thought/