Metrics – Scalar Limitations and Ignorance

This post is really about problems with the way the player base measures results and just a few issues that I end up having with them.  This is more of a rant/brainstorming post so beware of rambling and possible correct and incorrect use of mathematics.

Tools to measure performance, not success but performance.

  • Gear Score and Clones – this type of tools tries to assign a value to a player based on the gear they are wearing at a point it time.  Some of the more advance tools also compare iLvl and the stats on it to the spec in question.
  • The earlier form of measurement was the DPS meter again sever limitations especially when looking at healing.
  • Combat Log Parsers more advanced Meters.

I am sure there are many other tools that I am not naming directly here but every tool that I know of either measures information out of the combat log or looks at what characters are wearing.

For example consider the following diagrams.

In this diagram we show left to right time increasing and in most fights we hope the total damage done is positive so we see and increase in total damage as time gets larger.  So dps would be Y/X for the duration of the graph.

In this diagram we have Total damage and # of creatures.  As more creatures are added and we have a multi-target ability the total damage increases as we add more creatures to the mix.

However the reality is more complex than these simple diagrams.  In the diagram to the right we see 3d version of the previous 2 and we see that if you use AoE and increase # of creatures over time you will get more damage overall and if you boil this down to 1 scalar number DPS you are loosing all context of what the Scalar even means.

This is just a very simple demonstration of how most tools we use to evaluate performance end up giving us meaningless data because we use the output single dimensional value to evaluate players while removing all context.

For example on the boss targeting dummy in Ironforge my mage without blowing cooldowns will average about 5K DPS as fire or arcane on a single target.  Now when in the Lich King fight in phase 1 I will tab target everything up every 30 or so seconds and put living bomb on them.  During this phase while casting Scorch/Fire Blast/Insta Pyro on Burning Streak Proc and Blizzard near the end will rock 10-13k DPS.

So now which is the correct number to say as my DPS?

Now my WoW-Heroes score at time of writing is 3061.  Is this my metric of performance.  The answer is no it just states what my gear and enchants are worth.  It show my potential maximum, but not what I will or will not do, in an given context.

Now these diagrams are just as oversimplified when looking at 2 into the 3rd dimensional diagram because there are many more factors to add further context to the problem of evaluating other players or your own performance.

Other factors to consider.

  • FPS – How much interface lag are you experiencing when rendering graphics.
  • Latency – What is your Ping/Round Trip time to the server.  Remember DPS is Damage per Second so that .2 second round trip time is .2 seconds where commands are delayed and slower casts/key-presses = few abilities being triggered over a period of time and therefore less damage being done.
  • Interface/Add-on Lag – How much memory and processor time are you add-ons to track people using the wrong level of spell or the latest Peggle game causing your responsiveness to go down while rendering information not important to what you are doing.
  • Was your kill/heal sniped.  Were you in the middle of casting your “IWin” spell or heal and someone else killed the mob or healed the target so your damage or heal = 0 and goes into overkill/over-heal.
  • Do you have a lot of AoE spells that are part of your normal rotation?  Do you lack multi target attacks?

So I bring all of this up so that hopefully going forward in Cataclysm Blizzard and or the Player-base can come up with new ways for us to evaluate ourselves and others and not loose as much context while doing so.

Along these lines it would be useful to have repeatable solo quests that award multi level achievements on a character and or account wide basis that reflect scalar achievement results on different skills used in solo and group play.  Therefor the achievement will bring a common context to go with the score and therefor the score becomes a vector rather than a scalar.

Posted in World of Warcraft