Benchmark Methodology

[Stevenxowens792]

Please allow me to explain myself when it comes to benchmarking.

I believe when it comes to testing games or apps in reference to benchmarking we have two types of benchmarking.

1. Static - This means the test case which provides the benchmark is the same each time and provides a very repeatable expected results. These type of benchmarks usually dont show real in game performance but give you a good general overview of performance. Examples of this include 3dmark11, 3dmark vantage, 3dmark 06, and any in game benchmarks such as Just Cause 2 and Metro 2033.

2. Dynamic - This type of benchmark is the result of recording performance in game with tools such as FRAPS which record frames per second and the time during the bench. These can be taken during gameplay and represent an accurate representation

of performance during gaming and this includes multiplayer. The difficulty is that these can be somewhat variable and other factors such as network latency can impact performance.

Best wishes,

StevenX

[svl7]

Yeah, I agree... especially comparing fraps benches is difficult as the framerate will not only depend on the map, but also the way you're playing.

In some games measuring the fps while playing online doesn't make any sense, e.g. Black Ops Zombies... if there's a player in your field of view with a huge ping it will drastically drop your fps, at least that's what I experience.

Btw, does anybody know how huge (small) the impact of fraps on the performance is? (Benching, not recording)

[Stevenxowens792]

Minimal at best. What i do is run fraps at low priority. Zero impact.

[Brian]

With regards to benchmarking methods, that's what we also employ. However, with "dynamic" benchmarking, we use single player (e.g. bad company 2) at a certain scene so that the results are as close to each other as possible. We typically run 3 FRAPs tests and then average those out.

[Stevenxowens792]

Brian- I understand. Many folks want to know performance in MP. One of the first vids I did was the m11x R1 playing BFBC2 MP map, 32 man, on low to prove that you could play the game as others didn't believe it. I had FRAPS displaying rates and I took the video at 720p with a digital camera.

It's always challenging to get benchmarks in MP and again I completely understand why you would choose to bench SP during parts of maps that can be replicated.

Stevenx

[Nando]

so, lets do a list with a benchmarks tools:

• 3DMark 06: 3DMark06 is a PC benchmark suite designed to test the DirectX performance of your graphics card.
  
• PCMark Vantage: PCMark Vantage is a PC benchmark suite designed for Windows Vista offering one-click simplicity for casual users and detailed, professional grade testing for industry, press and enthusiasts.
  
• 3DMark Vantage: 3DMark Vantage is the industry standard performance benchmark for DirectX 10 gaming PCs. It includes two graphics tests, two CPU tests and six feature tests
  
• HWiNFO32: professional hardware information and diagnostic tools supporting latest components, industry technologies and standards.
  
• Fraps: Fraps is a universal Windows application that can be used with games using DirectX or OpenGL graphic technology.
  

[mw86]

Yes FRAPS is the only way to test and bench games that do not have in-game benchmarks built in. Finding a key point in game to start recording frames per second is the best way we can... but who can be sure that the exact points we try to start the bench on are the same point the next person starts recording frames per second?