Jump to content
Latest News
  • Apologies for the downtime. We had to update our backend and theme.
  • We will continue pushing updates.
  • Apologies for the downtime. We had to update our backend and theme.
  • We will continue pushing updates.
  • Sign in to follow this  

    Gradual Degradation of 3DMark Fire Strike Results Produces Unreliable Results


    Mr. Fox

    For those that are not already aware of the issue, the folks at Futuremark seem to be struggling to keep a consistent product in the latest 3DMark benchmark. In particular, Fire Strike. Sometime around the release of Time Spy things started getting screwy with Fire Strike and now it seems with every Fire Strike GUI update the effect is lowered benchmark scores, and specifically the physics portion of the benchmark.

     

    Kudos to @Papusan for noticing this months ago. He been going back and forth with Futuremark about the problem and it seems they are either ignoring him or don't care. Maybe because most people are not observant enough to notice or care.

     

    Some people might say you cannot compare results across benchmark software versions, but that shouldn't hold water here. There is a leaderboard and searchable database of results and if there is not a high degree of consistency between GUI versions the results in their database will become irreleant, as will their leaderboard. The search filter does not have a field to filter by GUI version, so we can expect the results from the database and leaderboard to be misleading, inaccurate and unreliable.

     

    You will notice from the examples posted below that with each new version of Fire Strike the scores get lower and lower. These examples are consecutive runs on the same day, same machine, and identical CPU and GPU settings. The only thing that changes is Fire Strike benchmark results. We need Futuremark to understand and correct this.

     

    http://www.3dmark.com/compare/fs/11047304/fs/11047179/fs/11047154

     

     

    Here is a similar example from @Papusanhttp://www.3dmark.com/compare/fs/11036017/fs/11035883

     

    If you agree this is a problem and want it to be fixed, please complain to Futuremark and let them know they need to put the brakes on and not do anything else with 3DMark until they have this mess under control. Gimmicky features are one thing, but inconsistent benchmark results makes 3DMark unreliable.

     

     


    This post has been promoted to an article


      Report Article
    Sign in to follow this  


    User Feedback


    6 hours ago, Mr. Fox said:

    For those that are not already aware of the issue, the folks at Futuremark seem to be struggling to keep a consistent product in the latest 3DMark benchmark. In particular, Fire Strike. Sometime around the release of Time Spy things started getting screwy with Fire Strike and now it seems with every Fire Strike GUI update the effect is lowered benchmark scores, and specifically the physics portion of the benchmark.

     

    Kudos to @Papusan for noticing this months ago. He been going back and forth with Futuremark about the problem and it seems they are either ignoring him or don't care. Maybe because most people are not observant enough to notice or care.

     

    Some people might say you cannot compare results across benchmark software versions, but that shouldn't hold water here. There is a leaderboard and searchable database of results and if there is not a high degree of consistency between GUI versions the results in their database will become irreleant, as will their leaderboard. The search filter does not have a field to filter by GUI version, so we can expect the results from the database and leaderboard to be misleading, inaccurate and unreliable.

     

    You will notice from the examples posted below that with each new version of Fire Strike the scores get lower and lower. These examples are consecutive runs on the same day, same machine, and identical CPU and GPU settings. The only thing that changes is Fire Strike benchmark results. We need Futuremark to understand and correct this.

     

    http://www.3dmark.com/compare/fs/11047304/fs/11047179/fs/11047154

     

     

    Here is a similar example from @Papusanhttp://www.3dmark.com/compare/fs/11036017/fs/11035883

     

    If you agree this is a problem and want it to be fixed, please complain to Futuremark and let them know they need to put the brakes on and not do anything else with 3DMark until they have this mess under control. Gimmicky features are one thing, but inconsistent benchmark results makes 3DMark unreliable.

     

     


    This post has been promoted to an article

    Thanks bro Fox

     See https://www.techinferno.com/index.php?/articles/frontpage_news/software_news/gradual-degradation-of-3dmark-fire-strike-produces-unreliable-results-r94/&do=findComment&comment=154385

     

     

    Share this comment


    Link to comment
    Share on other sites


    Create an account or sign in to comment

    You need to be a member in order to leave a comment

    Create an account

    Sign up for a new account in our community. It's easy!

    Register a new account

    Sign in

    Already have an account? Sign in here.

    Sign In Now


  • Similar Content

    • By Mr. Fox
       
      For those that are not already aware of the issue, the folks at Futuremark seem to be struggling to keep a consistent product in the latest 3DMark benchmark. In particular, Fire Strike. Sometime around the release of Time Spy things started getting screwy with Fire Strike and now it seems with every Fire Strike GUI version update the effect is progressively decreasing benchmark scores, and specifically the physics portion of the benchmark.
       
      Kudos to @Papusan for noticing this months ago and asking me to have a look at it. He has been going back and forth with Futuremark about the problem and it seems they are either ignoring him or perhaps they do not view it as a high priority issue. Or, maybe because most people running Fire Strike are not observant enough to notice, care, or ask questions they feel they don't need to fix it.
       
      Some people might say you cannot compare results across benchmark software versions, but that shouldn't hold water here. There is a leaderboard and searchable database of results that basically every benching enthusiast and PC reviewer relies on, and if there is not a very high degree of consistency between GUI versions the results in their database will become irrelevant, as will their leaderboard. The search filter does not have a field to filter by GUI version, so we can expect the results from the database and leaderboard to be increasingly misleading, inaccurate and unreliable over time. This certainly is not a desirable thing for what is supposedly the current defacto standard in PC benchmarks.
       
      You will notice from the examples posted below that with each new version of Fire Strike the scores get lower and lower. These examples are consecutive runs on the same day, same machine, and identical CPU and GPU settings. The only thing that changes is Fire Strike benchmark results degrade with newer versions. We need Futuremark to understand and correct this.
       
      http://www.3dmark.com/compare/fs/11047304/fs/11047179/fs/11047154
       
       
      Here is a similar example from @Papusan: http://www.3dmark.com/compare/fs/11036017/fs/11035883
       
      If you agree this is a problem and want it to be fixed, please complain to Futuremark and let them know they need to put the brakes on and not do anything else with 3DMark until they have this mess under control. Gimmicky features are one thing, but inconsistent benchmark results makes 3DMark unreliable.
       
      If you would like to do your own testing to validate the issue before contacting Futuremark, older versions of 3DMark are available for download from the TechPowerUp.com web site. 
       
      In case you're not good at simple math, here is a visual aid to show what the fuss is about.
       

       
      Update 12/13/2016:
      We would like to acknowledge that a representative of Futuremark has responded promptly to this article and provided an email address for those interested in communicating with them about the issue. We appreciate the accountability and responsiveness. 
      Update 12/15/2016:
      We sincerely are grateful for Futuremark's responsiveness. I provided additional test results to Mr. Kokko to corroborate the findings of @Papusan and they have released an update that is expected to resolve the issue. See the message from James below for more details.

      View full article
    • By Mr. Fox
      For those that are not already aware of the issue, the folks at Futuremark seem to be struggling to keep a consistent product in the latest 3DMark benchmark. In particular, Fire Strike. Sometime around the release of Time Spy things started getting screwy with Fire Strike and now it seems with every Fire Strike GUI update the effect is lowered benchmark scores, and specifically the physics portion of the benchmark.
       
      Kudos to @Papusan for noticing this months ago. He been going back and forth with Futuremark about the problem and it seems they are either ignoring him or don't care. Maybe because most people are not observant enough to notice or care.
       
      Some people might say you cannot compare results across benchmark software versions, but that shouldn't hold water here. There is a leaderboard and searchable database of results and if there is not a high degree of consistency between GUI versions the results in their database will become irreleant, as will their leaderboard. The search filter does not have a field to filter by GUI version, so we can expect the results from the database and leaderboard to be misleading, inaccurate and unreliable.
       
      You will notice from the examples posted below that with each new version of Fire Strike the scores get lower and lower. These examples are consecutive runs on the same day, same machine, and identical CPU and GPU settings. The only thing that changes is Fire Strike benchmark results. We need Futuremark to understand and correct this.
       
      http://www.3dmark.com/compare/fs/11047304/fs/11047179/fs/11047154
       
       
      Here is a similar example from @Papusan: http://www.3dmark.com/compare/fs/11036017/fs/11035883
       
      If you agree this is a problem and want it to be fixed, please complain to Futuremark and let them know they need to put the brakes on and not do anything else with 3DMark until they have this mess under control. Gimmicky features are one thing, but inconsistent benchmark results makes 3DMark unreliable.
       
       

      This post has been promoted to an article
    • By guilhrme
      Hello,
      I have an Alienware 18 (2013 model, A09 Bios), with two GTX 780m SLI (modded bios), Core i7 4930MX (Stock - Overclock enabled on the Bios Settings but in Default values), 32Gb Ram 1600mhz.
      My friend has an very identical hardware, but his Alienware is a previous model, with a 3940MX Extreme Processor. The GPUs are identical, with modded bios too. Both systems with a 330W PSU.
      Running 3dkmark, we realized that GPU scores has a significative difference.
      After disabling Overclock option on the Bios, my scores grew a little more, but the difference between the systems continues.
      Results: Result
      Guys, do you think that can be my processor getting the power from the GPU's? If no, what could be?
      Thank you.
×

Important Information

By using this site, you agree to our Terms of Use.