Benchmarking

The network tests covered inner-city areas, outer metropolitan and suburban areas. Measurements are also taken in smaller towns and cities along connecting highways. The combination of test areas is selected to provide representative test results across the populations. The test routes and all visited cities and towns are shown in every report. Our drive-test cars can be equipped with the latest smartphones for the simultaneous measurement of voice and data services in 4G and 5G.

One smartphone per operator in each car is used for the voice tests, setting up test calls from one car to another. The walktest team also carries one smartphone per operator for the voice tests. The audio quality of the transmitted speech samples are evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through injection of 100 KB of data traffic (HTTP downloads). Contact us for more information.

As a new KPI in 2019, we also evaluate the so-called Multirab (Multi Radio Access Bearer) Connectivity. This value denominates whether data connectivity is available during the phone calls.

Data performance is measured by using one mobile phone per operator in each car. For the web tests, they access web pages according to the widely recognized Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) is used. In order to test the data service performance, files of 5 MB and 2.5 MB for download and upload are transferred from or to a test server located in the cloud. In addition, the peak data performance is tested in uplink and downlink directions by assessing the amount of data that is transferred within a seven seconds time period. The evaluation of YouTube playback takes into account that YouTube dynamically adapts the video resolution to the available bandwidth. So, in addition to success ratios and start times, the measurements also determine average video resolution. All the tests are conducted with the best-performing mobile plan available from each operator.

For the collection of crowd-data, we have integrated a background diagnosis processes into thousands of diverse Android apps. If one of these applications is installed on the end-user’s phone, data collection takes place 24/7, 365 days a year (with the user approval). Reports are generated for regularly and sent daily to our cloud servers. We focus on the user experience, avoiding that our software cause any disruption on the user day-by-day. Our data collection is compliant with the GDPR, since we do not include any personal user data. Interested parties can deliberately take part in the data gathering with the specific ”U get“ app. This unique crowdsourcing technology allows us to collect data about real-world experience wherever and whenever customers use their smartphones.

For the assessment of network coverage, we lays a grid of 2 by 2 kilometres over the whole test area. The “evaluation areas“ generated this way are then subdivided into 16 smaller tiles. To ensure statistical relevance, we require a certain number of users and measurement values per operator for each tile and each evaluation area. Above that, we now distinguish urban and non-urban areas in our crowd evaluations – respecting that the coverage with mobile services is usually higher in urban areas than in rural surroundings.

We are investigating the data rates that were actually experienced by each user. For this purpose, we determine maximum download and upload data rates per user within 15 minutes slices. These values are then aggregated per evaluation area in 4-week-timeslices.

Also called “operational excellence“, this parameter indicates the number of “service degradations“ – events where data connectivity is impacted by a number of identified anomalies with sufficient severity. To judge this, the algorithm compares similar timeframes on similar days in a window around the day and time of interest. The algorithm looks at large scale anomalies on a network-wide level and ensures that individual users‘ degradations such as a simple loss of coverage due to an indoor stay or similar reasons can not affect the result.

Using our experience in the field of network-testing of mobile networks we verify and certify the performance of mobile services. The umlaut service certification is the ideal tool for the independent assessment and attestation of mobile networks and the services they offer. This is done by means of independent measurements and audits of service-quality. The successful verification of the service statement is recorded in a certificate.

The operator is entitled to use the certificate for its own communications purposes and use the „Best in Test“ seal in case of being awarded.

Our independent Public Benchmarks are designed to evaluate and objectively compare the performance and service quality of mobile networks all around the world. They are published in several countries on an annual basis in order to inform the end users about the best services available and to give the operators the opportunity to demonstrate both the strengths of their respective networks and improvements in performance or coverage over time. These improvements show the positive impact of our public benchmark activities on the development of the country’s networks. The end customers can take informed decisions on this basis.

Based on the results of an extensive and comprehensive series of tests focusing on the performance-capability of UMTS and LTE networks, the benchmarks highlight the performance of mobile networks, including the latest technologies deployed by each operator and their benefits to consumers.