Benchmarking Methodology & Framework
As the de-facto industry standard, our benchmarking methodology focuses on customer-perceived network quality and covers a wide range of mobile services. It allows a technical analysis that is unprecedented in its level of detail.
Our yearly update brings significant upgrades of our methodology framework in 2021, concentrating on recent developments like 5G user experience, Video experience, Reliability and new developments in our crowdsourcing data, such as voice call evaluation. The framework is based on a rule set that is used in all countries to provide a fair, transparent and independent evaluation of all networks and allows for true international comparability.
The network tests covered inner-city areas, outer metropolitan and suburban areas. Measurements are also taken in smaller towns and cities along connecting highways. The combination of test areas is selected to provide representative test results across the populations. The test routes and all visited cities and towns are shown in every report. Our drive-test cars can be equipped with the latest smartphones for the simultaneous measurement of voice and data services in 4G and 5G.
One smartphone per operator in each car is used for the voice tests, setting up test calls from one car to another. The walktest team also carries one smartphone per operator for the voice tests. The audio quality of the transmitted speech samples are evaluated using the HD-voice capable and ITU standardized so-called POLQA wideband algorithm. In order to account for typical smartphone-use scenarios during the voice tests, background data traffic was generated in a controlled way through injection of 100 KB of data traffic (HTTP downloads). Contact us for more information.
We also evaluate the so-called Multi-RAB (Multi Radio Access Bearer) Connectivity. This value denominates whether data connectivity is available during the phone calls.
Data performance is measured by using one mobile phone per operator in each car. For the web tests, they access a mix of national and international web pages according to the widely recognized Alexa ranking. In addition, the static “Kepler” test web page as specified by ETSI (European Telecommunications Standards Institute) is used. In order to test the data service performance, files of 10 MB and 5 MB for download and upload are transferred from or to a test server located in the cloud. In addition, the peak data performance is tested in uplink and downlink directions by assessing the amount of data that is transferred within a seven seconds time period. The evaluation of YouTube playback takes into account that YouTube dynamically adapts the video resolution to the available bandwidth, covering both typical user experience scenarios of HD (1080p) videos as well as testing the networks’ capability to deliver 4K UHD content. So, in addition to success ratios and start times, the measurements also determine average video resolution. All the tests are conducted with the best-performing mobile plan available from each operator.
For the collection of crowd-data, we have integrated a background diagnosis processes into thousands of diverse Android apps. If one of these applications is installed on the end-user’s phone, data collection takes place 24/7, 365 days a year (with the user approval). Reports are generated for regularly and sent daily to our cloud servers. We focus on the user experience, avoiding that our software cause any disruption on the user day-by-day. Our data collection is compliant with the GDPR, since we do not include any personal user data. Interested parties can deliberately take part in the data gathering with the specific ”U get“ app. This unique crowdsourcing technology allows us to collect data about real-world experience wherever and whenever customers use their smartphones to consume data services or place phone calls.
For the assessment of broadband network coverage, we are looking at both the geographical aspect of coverage as well as the individual experience of the user. The individual experience is captured by looking at the time users spend connected to the 4G or 5G broadband technology, while the geographical coverage metrics pre-aggregate data into a rectangular grid of Evaluation Areas (EA) which are then checked for the geographical reach of broadband technology as well as the consistency of getting access, provided the EA meets the min. requirements for statistical reliability.
Out of the crowdsourced data, we are investigating the data rates that were actually experienced by each user. For this purpose, we determine maximum of observed download and upload data rates per user within 15 minutes slices. In addition, we can configure the execution of active download, upload, web and video tests in our user base to complement the passive speed observation.
You need more information? Contact us!
