PEF/REV: A Multi-dimensional bug tracking metric
Traditional one dimensional bug tracking suffers from a lack of directive capability. If each bug is rated "ReallyBad", "Bad", or "Trivial", no clear direction exists for which ones to fix first. If you concentrate on all the "ReallyBad" bugs, you may fix bugs that only 2% of your users encounter 1% of the time and it breaks other code. You may not get to "Bad" bugs that 90% of the users encounter 90% of the time.
Each bug is rated with a PEF and a REV number.
- User's View: PEF - the sum of a rating for Pain, Effort, and Frequency.
How painfully is this bug to a user. 1 - low pain, 3 - fairly painful, 9 - very painful
How much effort does it take a user to work around the bug. A rating of 1 requires little effort, 3 means a fair amount of effort, and 9 implies a major amount of work.
How often does a user encountered this bug. 1 - infrequently, 3 - fairly often, 9 - frequently
The PEF represents the user's experience with the bug. A PEF of 27 means its really painful, is very hard to work around, and happens all the time. A PEF of just 3 means its not very painful, doesn't require much effort, and happens infrequently. All the really high PEF should be candidates to be fixed soon.
- Programmer's View: REV - the sum of a rating for Risk, Effort, and Verifiability
How risky is this fix to the system. 1 - little risk of collateral damage from the fix, 3 - some risk to other components, 9 - this fix touches lots of code and its very risky to the entire system.
How much effort does it take a programmer to fix this bug. A rating of 1 requires little effort, 3 means a fair amount of effort, and 9 implies a major amount of work.
If we fix this bug, can we easily verify that its fixed? 1 - its easy to verify, 3 - we can verify with some amount of work, 9 - its really hard to verify this bug is fixed.
Each bug also gets a REV number. A REV of 27 means its a very risky fix which may endanger other code, it will take lots of programmer effort to fix it, and its going to be difficult to verify the fix. High REV number bugs are bad.
The ideal bug (from a programmer's view point) is a PEF of 27 and a REV of 3. The worst case scenario is a PEF of 27 and a REV of 27.
Using the PEF/REV system you can gain more control over releases by specifying a target goal. This goal might be that we are going to fix all bugs with a PEF over 21 and REV under 21. (This goal should not be immutable. Certain bugs will need to get fixed that are not in this category, and others we may not fix that fall into this criteria. The team should decide on deviations from the policy as a conscious decision.)