For those that are not already aware of the issue, the folks at Futuremark seem to be struggling to keep a consistent product in the latest 3DMark benchmark. In particular, Fire Strike. Sometime around the release of Time Spy things started getting screwy with Fire Strike and now it seems with every Fire Strike GUI version update the effect is progressively decreasing benchmark scores, and specifically the physics portion of the benchmark.
Kudos to @Papusan for noticing this months ago and asking me to have a look at it. He has been going back and forth with Futuremark about the problem and it seems they are either ignoring him or perhaps they do not view it as a high priority issue. Or, maybe because most people running Fire Strike are not observant enough to notice, care, or ask questions they feel they don't need to fix it.
Some people might say you cannot compare results across benchmark software versions, but that shouldn't hold water here. There is a leaderboard and searchable database of results that basically every benching enthusiast and PC reviewer relies on, and if there is not a very high degree of consistency between GUI versions the results in their database will become irrelevant, as will their leaderboard. The search filter does not have a field to filter by GUI version, so we can expect the results from the database and leaderboard to be increasingly misleading, inaccurate and unreliable over time. This certainly is not a desirable thing for what is supposedly the current defacto standard in PC benchmarks.
You will notice from the examples posted below that with each new version of Fire Strike the scores get lower and lower. These examples are consecutive runs on the same day, same machine, and identical CPU and GPU settings. The only thing that changes is Fire Strike benchmark results degrade with newer versions. We need Futuremark to understand and correct this.
Here is a similar example from @Papusan: http://www.3dmark.com/compare/fs/11036017/fs/11035883
If you agree this is a problem and want it to be fixed, please complain to Futuremark and let them know they need to put the brakes on and not do anything else with 3DMark until they have this mess under control. Gimmicky features are one thing, but inconsistent benchmark results makes 3DMark unreliable.
If you would like to do your own testing to validate the issue before contacting Futuremark, older versions of 3DMark are available for download from theweb site.
In case you're not good at simple math, here is a visual aid to show what the fuss is about.
We would like to acknowledge that a representative of Futuremark has responded promptly to this article and provided an email address for those interested in communicating with them about the issue. We appreciate the accountability and responsiveness.
We sincerely are grateful for Futuremark's responsiveness. I provided additional test results to Mr. Kokko to corroborate the findings of @Papusan and they have released an update that is expected to resolve the issue. See the message from James below for more details.
14 hours ago, Futuremark_James said:
Hello. James from Futuremark here again.
We've confirmed that there was an issue with the GUI, and we're in the process of rolling out an update (3DMark v2.2.3509) that should fix the scoring discrepancy.
With this update, overall scores increase slightly by up to 0.3%. Scores from the Physics and CPU parts of benchmark tests may improve by up to 2.5%. These changes bring the scores from 3DMark v2.2.3509 back in line with results from earlier versions that did not have the GUI issue.
For context, it is normal for 3DMark scores to vary by up to 3% between runs since there are some factors in a modern, multitasking operating system that cannot be completely controlled. So again, all credit to @Papusan for noticing the problem and bringing it to us.
To get the update, just open 3DMark and you should get a notification with the option to install it. The Steam version and Steam demo have also been updated.
On 12/13/2016 at 0:19 PM, Futuremark_James said:
Hi. James from Futuremark here.
We've been looking into this today, and I'd like to share what we've found.
The Fire Strike workload has not changed at all since 2013. This means that Fire Strike scores should not have changed across app versions either.
We've confirmed that running 3DMark from the command line gives consistent scores across all versions. Unfortunately, it does look like there is an issue when running recent versions from the GUI. We see the same ~2.5% difference in Physics test scores across GUI versions that @Papusan reported to us.
We believe we have found the bug in the GUI, but we need to run some more tests to be sure.
@Mr. Fox, the differences that you are seeing in your results are much larger, and it is not clear why. We would be grateful if you could contact us at firstname.lastname@example.org so we can go through some troubleshooting steps with you.
Thank you, @Papusan, for bringing this to us. I am sorry that we have been slow to respond. I understand how frustrating that is.
I'll post here again when we have more info to share.