Methodology

The network tests conducted by umlaut, part of Accenture, and connect are widely accepted as the de-facto industry standard and for being highly objective. With a further refinement of the crowdsourcing methodology already known from umlaut‘s accredited mobile network tests, it became also possible to analyze relevant performance KPIs of fixed-line services.

Comprehensive crowdsourcing

The results of this test are based on a comprehensive analysis of crowd-sourced data which is performed by umlaut, Part of Accenture and headquartered in Aachen, Germany. For the data collection, umlaut has integrated a background diagnosis process into thousands of popular Android apps. If one of these applications is installed on the end-user’s phone and the user authorizes the background analysis, data collection takes place in the background during smartphone use. Samples are generated in specific inter-vals (from one second up to 15 minutes) and sent daily to umlaut‘s cloud servers, where the data is further processed. By filtering the network access technology to those samples collected via Wi-Fi (in contrast to mobile network connections) and identifying the network operator, the collected samples can be limited to fixed-line connections. Using a complex set of rules and comprehensive checks, umlaut hardens the validity of the evaluations. Among other steps, data recorded when the smartphone battery is low is filtered out, as are measurements when the transfer volume is too low or extremely low data rates indicate interference. Thus, the influence of the mobile devices is likely to be small. The Wi-Fi speeds achievable on current smartphones are usually significantly higher than the total data rates observed, the influence actual Wi-Fi link speed on the measurement results can be neglected.

Passive Data Rates

The passive gathering of the data rates observed for downloads and uploads take place in the background while the user‘s employ everyday applications on their devices such as web browsing, streaming or gaming. In order to classify the observed speeds, umlaut has defined four application-related speed classes: “HD Video“ requires 5 Mbps, “UHD Video“ requires 20 Mbps and “Highspeed“ requires 50 Mbps.For the typically lower rates of data uploads, only the speed classes “HD Video“ (min. 5 Mbps) and “UHD Video“ (min. 20 Mbps) are considered. The observed passive download speeds make up 10% of the total result, the upload speeds contribute 5% to the total result. 

Active Data Rates

In addition to the passive measurements, which register the throughputs requested by the apps running in the foreground, umlaut also conducts active measurements of upload and download data rates once a month. They determine the amount of data that could be transferred in 3.5 seconds. For the determined mea-surement values, our scoring considers the average data rate, the P10 value (90% of the values higher than the specified threshold, a good approximation of the typical minimum speed) and the P90 (10% of the values higher than this threshold, a view at the peak values). This measurement typically approaches the maximum possible throughput of a considered fixed line. The determined active download speeds make up 37.5% of the total score, the active upload tests contribute 17.5% of the total result.

Latency

The latency measurements are carried out every 15 minutes  – “pings“ are performed in direct succession to the connection tests. Their results are also assigned to an application-related class: Roundtrip times of less than 50 ms qualify a sample for “Standard Gaming“ and less than 20 ms for “Highend Gaming“. If the latency is faster than 10 ms, the sample is counted as “Ultra Low Latency” (ULL) which would qualify for near real-time applications.The tables in this report show the percentage of the examined connections that were able to achieve the required threshold values in the different classes or did even better. The assessment of latency makes up 25% of the total score.

Stability

Based on the determined data rates and additional browsing and connection tests, umlaut also examined when a broadband connection could be used at all. The averaged and weighted results define the percentage of transaction success. The stability assessment makes up 5% of the total score.