I run the Quality Assessment for a large group of estate agents and we regularyly calibrate our quality (at least once a month). We use a Yes/No/NA form which have points allocated to each, giving an overall percentage score.
<60% is a fail
61% - 74% is an average
75% - 96% is a good
96% > Outstanding.
This way, when Quality is reported to seperate sales team, we can compare it to competing sales teams on the sales floor and also, when the sales floor report is issued at the end of working month, teams can see where they are in relation to quality compared to other teams doing the same job but for a different region for example.
This particular approach works really well for us here, as its a simple Yes/No or N/a so we dont have the issue with what one might feel is a 3, another may feel is a 5. Another benefit to using this, is when we are feeding back to agents they motivate themselves to beat their % score the next month (often because colleagues scores are higher then theirs.