Benchmarking
is our territory and the de facto industry standard
Our top priority is to fairly and transparently assess the global development of networks, to push their quality and performance with our benchmarks and ultimately achieve improvements for every customer.
The network tests covered inner-city areas, outer metropolitan and suburban areas. Measurements are also taken in smaller towns and cities along connecting highways. The combination of test areas is selected to provide representative test results across the populations. The test routes and all visited cities and towns are shown in every report. Our drive-test cars can be equipped with the latest smartphones for the simultaneous measurement of voice and data services in 4G and 5G.
One smartphone per operator in each car is used for the voice tests, setting up test calls from one car to another. The walktest team also carries one smartphone per operator for the voice tests. The audio quality of the transmitted speech samples are evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through injection of 100 KB of data traffic (HTTP downloads). Contact us for more information.
We also evaluate the so-called Multi-RAB (Multi Radio Access Bearer) Connectivity. This value denominates whether data connectivity is available during the phone calls.
Data performance is measured by using one mobile phone per operator in each car. For the web tests, they access a mix of national and international web pages according to the widely recognized Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) is used. In order to test the data service performance, files of 10 MB and 5 MB for download and upload are transferred from or to a test server located in the cloud. In addition, the peak data performance is tested in uplink and downlink directions by assessing the amount of data that is transferred within a seven seconds time period. The evaluation of YouTube playback takes into account that YouTube dynamically adapts the video resolution to the available bandwidth, covering both typical user experience scenarios of HD (1080p) videos as well as testing the networks’ capability to deliver 4K UHD content. So, in addition to success ratios and start times, the measurements also determine average video resolution. All the tests are conducted with the best-performing mobile plan available from each operator.
For the collection of crowd-data, we have integrated a background diagnosis processes into thousands of diverse Android apps. If one of these applications is installed on the end-user’s phone, data collection takes place 24/7, 365 days a year (with the user approval). Reports are generated for regularly and sent daily to our cloud servers. We focus on the user experience, avoiding that our software cause any disruption on the user day-by-day. Our data collection is compliant with the GDPR, since we do not include any personal user data. Interested parties can deliberately take part in the data gathering with the specific ”U get“ app. This unique crowdsourcing technology allows us to collect data about real-world experience wherever and whenever customers use their smartphones to consume data services or place phone calls.
For the assessment of broadband network coverage, we are looking at both the geographical aspect of coverage as well as the individual experience of the user. The individual experience is captured by looking at the time users spend connected to the 4G or 5G broadband technology, while the geographical coverage metrics pre-aggregate data into a rectangular grid of Evaluation Areas (EA) which are then checked for the geographical reach of broadband technology as well as the consistency of getting access, provided the EA meets the min. requirements for statistical reliability.
Out of the crowdsourced data, we are investigating the data rates that were actually experienced by each user. For this purpose, we determine maximum of observed download and upload data rates per user within 15 minutes slices. In addition, we can configure the execution of active download, upload, web and video tests in our user base to complement the passive speed observation.
Framework
Our yearly update brings significant upgrades of our methodology framework in 2021.
umlauts mobile Benchmarking is concentrating on recent developments like 5G user experience, Video experience, Reliability and new developments in our crowdsourcing data, such as voice call evaluation.
The Fixed Broadband Benchmarking introduces active testing components to the existing set of evaluation criteria like download speed, upload speed and latency for broadband technologies across copper, cable and full fibre. Our unique scoring methodology includes a worldwide ranking on country level.
Markets
Today, more than 200 mobile networks in more than 120 countries are being evaluated by our unique scoring methodology & framework.
Latest reports:
2022/06 Audit Report
Vodafone New Zealand, Hamilton
pdf, 4 MB
2022/04 Audit Report
Vodafone New Zealand, Southland
pdf, 4 MB
2022/04 Audit Report
Vodafone New Zealand, Taranaki
pdf, 5 MB
2022/04 Mobile Certificate
Yettel Serbia
pdf, 3 MB
2022/05 5G Audit Report
pdf, 2 MB
2021/12 Mobile Certificate
Safaricom Kenya
pdf, 2 MB
2022/04 OpenRAN Audit Report
Rakuten Mobile Japan
pdf, 2 MB
2022/04 Audit Report
Vodacom
pdf, 3 MB
2022/03 Mobile Certificate
Vodafone New Zealand
pdf, 3 MB
2022/03 Mobile Certificate
Vodafone Romania
pdf, 2 MB
2022/03 Mobile Certificate
Yettel Bulgaria
pdf, 2 MB
Please use the interactive world map below to filter all operators, tests and certificates by clicking on the button in the top right corner. You can also zoom in and out using the button in the top left corner of the world map. If you click on the individual countries, a pop-up window will open, allowing you to access the respective country teasers and to find and download all reports and certificates.
Taking network testing to the next level
Our public benchmarks as well as the certificate benchmarks help network operators to demonstrate how well they deliver wireless connections to consumers, business users and enterprises while revealing the areas of improvement.
Using our experience in the field of network-testing of mobile networks we verify and certify the performance of mobile services. The umlaut service certification is the ideal tool for the independent assessment and attestation of mobile networks and the services they offer. This is done by means of independent measurements and audits of service-quality. The successful verification of the service statement is recorded in a certificate.
The operator is entitled to use the certificate for its own communications purposes and use the „Best in Test“ seal in case of being awarded.
Our independent Public Benchmarks are designed to evaluate and objectively compare the performance and service quality of mobile networks all around the world. They are published in several countries on an annual basis in order to inform the end users about the best services available and to give the operators the opportunity to demonstrate both the strengths of their respective networks and improvements in performance or coverage over time. These improvements show the positive impact of our public benchmark activities on the development of the country’s networks. The end customers can take informed decisions on this basis.
Based on the results of an extensive and comprehensive series of tests focusing on the performance-capability of UMTS and LTE networks, the benchmarks highlight the performance of mobile networks, including the latest technologies deployed by each operator and their benefits to consumers.
You don´t find your market or need more information? Contact us!
