Understanding broadband speed measurements

Understanding broadband speed measurements

Steve Bauer David Clark William Lehr Massachusetts Institute of Technology

Abstract

Broadband speed has emerged as the single most commonly cited metric for characterizing the quality of broadband offerings. However speed measurements for the same service can vary significantly. These differences arise from a complex set of factors including different test methodologies and test conditions. For any testing methodology, teasing apart the end-to-end tests and attributing performance bottlenecks to constituent parts is technically challenging. While the broadband access network can be the bottleneck, significant bottlenecks arise in home networks, end users' computers, and server side systems and networks. Consequently, inferences regarding how ISP delivered speeds compare with their advertised speeds need to be undertaken with careful attention to the testing methodologies employed. Many testing methodologies are inappropriate for the purposes of assessing the quality of a broadband network.

1. Executive Summary1

Speed is the single most important metric of interest in characterizing the "quality" of broadband service. Perceptions about broadband quality inform regulatory policy, end-user behavior (e.g., broadband subscriptions), investments in complementary assets (e.g., content and applications), as well as the marketing, traffic management, and provisioning decisions of network operators. Consequently, it is important to understand how speed is, and should be, measured.

The goals for this paper are several. First, we explain the complexity of measuring broadband speeds. There are multiple definitions of "speed" that are potentially of interest: is it a measure of potential throughput or capacity or is it as a measure of average speed as experienced by endusers? Is the focus the broadband access service or end-to-end performance? Is the goal to

1 We would like to acknowledge the support of participants in the MIT MITAS () and CFP () research programs. Any opinions expressed and errors are solely the responsibility of the authors.

Page 1 of 38

diagnose a potential fault or to benchmark performance? Is the interest in a single broadband connection, a geographic market, or some larger population?

Second we document key methodological differences that account for some of the variability in popular speed measurement data sources. While it is not surprising that there is significant variability in broadband speed measurements across providers, geographic markets, access technologies, and time, it is surprising how much variation can result from methodological differences in how speed is measured. Understanding these methodological differences is important to making valid inferences from the measurement data.

For example, while a principal motivation for many in looking at speed measurements is to assess whether a broadband access ISP is meeting its commitment to provide an advertised data service (e.g. "up to 20 megabits per second"), we conclude that most of the popular speed data sources fail to provide sufficiently accurate data for this purpose. In many cases, the reason a user measures a data rate below the advertised rate is due to bottlenecks on the user-side, at the destination server, or elsewhere in the network (beyond the access ISP's control). A particularly common non-ISP bottleneck is the receive window (rwnd) advertised by the user's transport protocol (TCP). In the NDT dataset we examine later in this paper, 38% of the tests never made use of all the available network capacity.

Other non-ISP bottlenecks also exist that constrain the data rate well below the rate supported by broadband access connections. Local bottlenecks often arise in home wireless networks. The maximum rate of an 802.11b WiFi router (still a very common wireless router) is 11 Mbps. If wireless signal quality is an issue, the 802.11b router will drop back to 5.5 Mbps, 2 Mbps, and then 1 Mbps. Newer wireless routers (e.g. 802.11g/n) have higher maximum speeds (e.g. 54 Mbps) but will similarly adapt the link speed to improve the signal quality. End-users also can self-congest when other applications or family members share the broadband connection. Their measured speed will be diminished as the number of competing flows increase.

Each of the different testing methodologies we examined provides insights into network performance. We concluded however that the Ookla/Speedtest approach ? which typically results in higher measured data rates than the other approaches reviewed ? was the best of the currently available data sources for assessing the speed of ISP's broadband access service. One of the key differences that accounts for this is that the Ookla/Speedtest tools utilize multiple TCP connections to collect the measurement data which is key to avoiding the receive window limitation if a test is done from a common desktop computer. The Ookla/Speedtest tests are also much more likely to be conducted to a server that is relatively close to the client running the test.

We expect that new hardware based speed measurements, such as those being conducted now by the FCC in collaboration with Samknows (which ran a similar study for Ofcom in the UK), will produce very useful results.2 The Samknows hardware platform eliminates many of the potential

2 See (accessed 21 June 2010).

Page 2 of 38

bottlenecks (e.g. noisy wireless links and receiver window bottlenecks) that can interfere with network measurements. However, such hardware based testing will not eliminate the importance of other testing methodologies. Orders of magnitude more data can be gathered by the existing non-hardware-based tests. We also expect that some of the same tests will be conducted in both the hardware and browser based environments (e.g. we expect some of the hardware testing platforms will leverage the M-labs infrastructure running the NDT test.)

Precisely because differences do matter, it is important to understand methodological details of the tests being conducted. Unfortunately uncovering the testing details is not always easy. There are marked differences in the quality of testing documentation and disclosure. Some of the organizations made adjustments to their methods and documentation in response to our queries. This highlights a kind of Heisenberg Uncertainty Principle as it applies to this sort of research: because speed data matters, purveyors of such data may adjust the methods and published information dynamically and, potentially, strategically.

We recognize that the debate over speed is, itself, strategic and engages the interests of all market participants ? policymakers, users, and ISPs alike. While we do not believe there is a single method for measuring and reporting speeds that is appropriate for all contexts, we recognize the analytic and cost benefits from narrowing the range of methods and from moving toward collective agreement on best practices. Having a few generally accepted methods for measuring broadband speeds and having clear documentation to support published measurements will aid data aggregation, analysis, and interpretation.

Furthermore, although speed matters and will continue to do so in the future to all market participants interested in assessing the quality of broadband services, it is not all that matters. There is a risk that the focus on speed or a particular way of measuring or reporting broadband speeds might distort market behavior. We believe promoting a more nuanced understanding of how and why speed matters offers the best defense against such adverse distortions (e.g., what applications are most likely to benefit from faster broadband services? What additional mechanisms may be available to isolate speed bottlenecks at different points along the end-toend path? What other technical attributes should be regularly measured and reported to better characterize broadband service quality?).

In writing this paper, our intent is to contribute to this public debate and to render more accessible some of the speed measurement issues. Our hope is that progress may be made via a market-mediated process that engages users, academics, the technical standards community, ISPs, and policymakers in an open debate; one that will not require strong regulatory mandates. Market efficiency and competition will be best served if there is more and better understood data available on broadband speeds and other performance metrics of merit (e.g., pricing, availability, and other technical characteristics).

Page 3 of 38

2. Introduction3

Broadband speed has emerged as the single most commonly cited metric for characterizing the quality of broadband offerings. There are now a number of sites and organizations that measure the speed (and other characteristics) of a user's broadband service (e.g., , Akamai, ComScore, M-Labs, Google's Youtube service, or the FCC's own speed tests). The data generated by such tests is often aggregated and reported in the trade press and in government reports and plays a role both in policy formulation and in decision-making by individual consumers.4 Consequently, how speeds are measured and how the data is interpreted can have important implications for market and policy outcomes.

In this paper, we explain why the resulting speed measurements at times vary significantly. For instance, Google's Youtube service reports an average download speed of 3.83 Mbps for the United States while the reports a 7.71 Mbps average.5 The differences can be even more pronounced at smaller geographic granularities. For Comcast in the Boston area, reports average download speeds of 15.03 Mbps while Youtube reports a 5.87 Mbps average.6 Differences in the methodologies account for most of the discrepancy. The proper interpretation is not that one test is "right" or "wrong" but rather the different tests provide different insights into the end-to-end performance under different workloads.

For any testing methodology, teasing apart the end-to-end tests and attributing performance bottlenecks to constituent parts is technically challenging. While the broadband access network can be the bottleneck, significant bottlenecks arise in home networks, end users' computers, and server side systems and networks. The dynamics and settings of TCP (the dominant transport protocol on the Internet) also play a significant role in determining the resulting speed that is measured. There is also an important question about systematic biases in user initiated speed tests. Potentially users are running those tests in a diagnostic fashion when they are experiencing problems.7 The main point is that inferences regarding how ISP delivered speeds compares with

3 This paper is intended to be accessible to non-experts in Internet technology, but space constraints presume a level of technical sophistication. To help bridge the gap, this paper is accompanied by web-based resources for acronyms and further details. See 4 The recent U.S. National Broadband Plan ( ) for instance notes both current results and recommends expanding the set of broadband performance measurements. 5 The exact numbers from each of these services varies over time. These results were downloaded on March 16, 2010 from and respectively. 6 Results download on March 16, 2010 by computers in the Boston area from and . It is unclear how closely geographic areas align for different services reporting results from smaller geographic regions like states and cities. 7 For some networks such as broadband satellite networks or wireless broadband networks which often have a smaller number of test samples in a given data set, we suspect that tests run in an attempt to diagnosis a problem could account for a significant percentage of tests. However, we are personally familiar with many expert and nonexpert users that simply like to measure their speed for fun.

Page 4 of 38

their advertised speeds need to be undertaken with careful attention to the testing methodologies employed. Many testing methodologies are inappropriate for the purposes of assessing the quality of ISP access networks.

Measuring and understanding the performance of the broadband access networks, though, remains an important policy question. We examined in detail a number of speed tests, using packet traces to understand what factors shaped the outcome and how the test was performed. We discuss a number of these cases, and argue that the Ookla/Speedtest test methodology8 is more likely than the other tests we examine to correspond to the speed of an access link for common network usage patterns. We explain why the Ookla/Speedtest methodology often produces higher estimates of speed then other tests. We discuss limitations and some of the technical rationales underlying the tradeoffs that must be made in this type of testing.

We conclude with policy recommendations that emphasize that while speed remains an important statistic for evaluating the quality of broadband and will likely remain so, appropriate metrics for evaluating the performance of broadband services should consider additional characteristics as well. Moreover, we expect that the relevance of raw speed measurements ? separate from more nuanced context-dependent considerations ? will become increasingly less relevant as the speed of the typical broadband offering becomes ever faster. We also call for better documentation of speed testing methodologies from all organizations so that third-party validation of measurements is easier.

3. Defining what is meant by broadband speed

We first have to understand exactly what is meant by the "speed of broadband". The casual interpretation of speed is that it indicates "how fast you can go". More speed, everything else being equal (particularly price), is always better. But beyond this casual interpretation of speed, lie a variety of divergent meanings. For example, there is the speed implied by the way the broadband access service is configured (its "capacity"), the speed that is marketed or understood by consumers ("advertised"), or the speed that is actually experienced by users ("achieved"). We put these terms in quotes because we are using them as imprecise shorthand.

3.1. Provider configured broadband speeds

In the advertisements of broadband providers, upload and download speeds provide information about how the link between the customer's location and the broadband provider is configured. It generally corresponds to configuration settings that are set in network equipment such as cable and DSL modems and the routers or other network equipment, and are generally intended to give an indication of the maximum or peak data rates that a customer may experience. More typically, however, these advertised data rates are taken to imply something meaningful about the

8

Page 5 of 38

experience of using the broadband service. The challenge therefore is how to define and measure "speeds" that are both representative of the user experience on the broadband network and are robust to the messy measurement conditions of the real world.

Figure 1: Network diagram showing the most important devices and links connecting broadband users with destinations on the Internet. Broadband providers service demarcation points are between points 2 and 5 generally. Most of the speed tests we examine in this paper measure the speed between 1 and 6. Diagram source: "Federal Communications Commission Request for Quotation for Residential Fixed Broadband Services Testing and Measurement Solution."

The speed advertised and set by a broadband provider (i.e. the speed between 4 and 5 in Figure 1 above) is an important metric for characterizing the broadband service and does provide a useful first-level indicator of the expected "speed" of traffic carried by the network. However, it may not be directly indicative of what a user will actually experience. There are numerous reasons why differences arise. The most commonly assumed reason for measuring speeds lower than the configured speed is that broadband access networks rely on multiple users sharing the available network capacity (particularly the link between 4 and 5 in Figure 1 in the case of cable and between 3 and 4 in the case of DSL). Because end-user demands are unlikely to be perfectly correlated in time, the traffic of multiple users can share network capacity, thereby substantially reducing the total costs of providing network services. Assuming no problems elsewhere, a user whose connection is configured to run at a peak rate of 15 Mbps can realize that speed across parts of the shared infrastructure if enough other users are not simultaneously utilizing the network.9 The

9 This assumes that the applications or services they are utilizing actually attempt to use all capacity available on their connection. Video on the web today generally does not transfer at the highest speeds possible. (This includes video transmitted over TCP.)

Page 6 of 38

throughput during the peak usage hours depends upon how many users are configured to share (a design decision) and how many of them are simultaneously active (user behavior). This sharing is fundamental not just for broadband networks, but the Internet as a whole.10 The potential aggregate demand from senders and receivers on the Internet vastly exceeds the capacity of the network. Sharing saves money and allows everyone to get more than they would on their own for a given price. As we demonstrate in this paper, however, congestion arising from infrastructure sharing with multiple users is not the most common reason that speeds lower than the provider configured speed are measured. This may change over time as other sources of performance impairments are fixed.

The configured "up to" speed advertised by a provider is also not actually the maximum speed that one can measure on a network. Higher-than-expected measurements can occur for a number of reasons. The "line speed" for a broadband service can be faster than the configured speed.11 The line speed is the speed at which an individual packet is sent on the wire or fiber and is a property of the physical and link layer technologies employed. A packet in a DOCSIS 2.0 network will be sent at something around 38 Mbps for instance.12 Short bursts of packets can also be sent closer together than would be implied by the configured speed.13 Consequently, the speed that may be measured over very short time intervals is complicated by non-trivial scheduling algorithms employed by the lower link-layer protocols.

However, even over longer intervals, the speed measured may be more than the data rate that is advertised. Access networks can remove the configured speed limits for periods of time thus allowing the users to send at higher rates. Some cable broadband networks employ what goes by the trade name of "Powerboost."14 While Powerboost is advertised as a boost in the configured sending rate to a certain configured higher level, the Powerboost speeds often exceed even that

10 For a more complete discussion of Internet congestion and why the observance of congestion, in itself, is not an indication that a network is configured or managed inappropriately, see Bauer, Steve, David Clark, and William Lehr (2009), "The Evolution of Internet Congestion," paper prepared for 37th Research Conference on Communication, Information and Internet Policy (),A rlington,V A ,Septem ber 2009 (available at: ). 11 The exception would be services like 100mbps broadband that is available in some countries where the service is actually defined by the line speed itself. 12 DOCSIS is the Data Over Cable Service suite of international telecommunication standards that define interface requirements for offering broadband services over cable modems. The DOCSIS standards are developed by CableLabs, an industry research consortium (). Most of the cable systems currently in operation have implemented DOCSIS version 2.0, although a number of operators are presently rolling out DOCSIS 3.0 systems that support higher data rates and additional functionality. 13 Lakshminarayanan, K., Padmanabhan, V. N., and Padhye, J. 2004. Bandwidth estimation in broadband access networks. In Proceedings of the 4th ACM SIGCOMM Conference on internet Measurement(Taormina, Sicily, Italy, October 25 - 27, 2004). IMC '04. ACM, New York, NY, 314-321. 14 See (accessed April 22, 2010).

Page 7 of 38

advertised limit by non-trivial amounts.15 This is important to recognize because some studies have attempted to infer the subscribed speed tier by assuming the maximum measured speed is less than or equal to 110% of the Powerboost level.16 Tests we have conducted on our personal broadband connections have demonstrated this assumption does not hold.

So, if the configured peak or "up to" speed is not adequate for characterizing the speed of the network, how should speed be measured? Academics have long been interested in how to understand what a network is capable of supporting. The research literature focuses on a variety of different measurements one might want to make from the edge of a network. (We emphasize edge-based measurements because the research literature emphasizes how to infer or verify network characteristics with probing or measurements taken at the edges. Some of what one might want to know is known by the providers but may not be directly observable by thirdparties.)

A distinction is generally made between the following properties of a network that one might want to measure.17

1. Capacity is a measure of the total traffic carrying capability of a link or path in a network. The end-to-end capacity is the minimum link capacity along a path. See Figure 2 (A) below. Link 3 in that figure has the most capacity but the end-to-end capacity would be determined by the capacity of link 2. The capacity is expressed as the amount of traffic the link can carry over a particular time interval (e.g., megabits per second).

2. Available bandwidth18 is a measure of how much capacity is unused in a link or along a path. The available bandwidth along a path is the minimum available bandwidth of the set of links along a path. See Figure 2 (B) below. Link 1 has the most available bandwidth but the end-to-end available bandwidth is determined by link 2. One can look at a link utilization graph and observe that the total capacity was 100 Mbps but the peak usage was 45 Mbps and conclude that the available bandwidth, or spare capacity, was 55 Mbps.

3. Bulk transfer capacity is a measure of the amount of data that can be transferred along a network path with a congestion aware transport protocol like TCP. See Figure 2 (C)

15 For example one of the author's home cable network service includes a 12 mbps service plan with Powerboost up to 15 mbps. Measuring with tools like iperf and however we can regularly measure speeds in excess of 20 mbps for data transfers of 10 ? 20 MB. 16 This was part of the methodology employed by Comscore and relied upon by the FCC. 17 See for instance R.S.Prasad, M.Murray, C.Dovrolis, and K.C.Claffy. Bandwidth Estimation: Metrics, Measurement Techniques, and Tools. IEEE Network, 2003. 18 Network operators may be more likely to refer to "available bandwidth" as the "spare capacity" on a link or along a path. The "bulk transfer capacity" might be more likely to be discussed in terms of "throughput".

Page 8 of 38

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download