Samsung not alone in manipulating benchmarking software

Various smartphones

The practice of manipulating benchmark software is more widespread than initial reports let on. Samsung was caught gaming benchmark scores on the Galaxy Note 3 by detecting when benchmark software was running. Not surprisingly, Samsung denied any wrongdoing. It now appears that Samsung is not the only manufacturer manipulating benchmarks. AnandTech, which first caught Samsung gaming Galaxy S4 scores this past summer, has now found that other manufacturers are engaged in similar practices.

Among the manufacturers found to be gaming at least some benchmark suites on at least some of their devices are ASUS, HTC, and LG. Specific devices include the ASUS Padfone Infinity, HTC One, HTC One mini, LG G2 and various Samsung devices including the Galaxy Tab 3 and Galaxy Note 10.1 (2014).

It should also be noted that a number of devices that AnandTech looked at do not appear to have any such benchmark gaming tweaks. Among those are the Google Nexus 4, Google Nexus 7, Motorola Moto X and NVIDIA Shield.

Further invalidating any benchmarking scores is the fact that the code optimizations are different between manufacturers and even between devices.

Sadly, AnandTech does not see the practice stopping any time soon. Instead, it is likely to get harder to detect:

The unfortunate reality is this is all going to get a lot worse before it gets better. We wondered what would happen with the next platform release after our report in July, and the Note 3 told us everything we needed to know (you could argue that it was too soon to incite change, perhaps SGS5 next year is a better test). Going forward I expect all of this to become more heavily occluded from end user inspection. App detects alone are pretty simple, but what I expect to happen next are code/behavior detects and switching behavior based on that. There are thankfully ways of continuing to see and understand what’s going on inside these closed platforms, so I’m not too concerned about the future.

As we have pointed out before, benchmarks should not be the basis upon which one buys a smartphone or tablet. But they are a useful tool that allows us to compare devices under a certain set of controlled conditions. Sadly, it now appears that we will no longer be able to do so with any confidence.



Source : AnandTech