Qualitative Metrics by Mel Schnapper, Ph



The Why, What and How of Qualitative Metrics

By Mel Schnapper, Ph.D.

Founder and Chair of Project Management Institute (PMI)

Metrics Specific Interest Group (SIG)

“It’s either quantifiable or it’s not measurable”

“It’s either quantifiable or it’s subjective

“ We have to quantify it in order to measure it”

Most, if not, all who just read the statements above, will:

1. Agree with them,

2. Equate the word “measure” with the word “quantify”, and, consequently,

3. Avoid and/or neglect the explicit qualitative dimensions of measuring your Project Management progress.

Even staff with a Quality Management-related role, are not likely to have rigorous metrics of quality that are qualitative! in nature.

This article will establish the foundation of qualitative metrics, illustrate how we use them anyway! And then show how to use them in the Project Management world as illustrated by the qualitative metrics of a Director of a Project Management Office.

There is an almost total lack of qualitative metrics in the PM world. The “3Ms of Performance: Measure, Manage and Magnify” is a methodology where qualitative metrics are as rigorous and as precise as quantitative metrics. In fact, much of the 3Ms methodology reflects how most people think about most of their lives, but at a level that is implicit and seldom articulated. The exceptions are in conflict resolution or crisis situations when individually or as parties in a dispute, people are forced to spell-out, in detailed fashion, the metrics of the resolution for mutual understanding, agreement and commitment.

Therefor, after some examples from the PM world, there will also be examples from the non-PM world, to see the kind of rigor and precision provided by qualitative metrics.

The Qualitative Metrics of a Project Management Office

The 3Ms profile of a Director of a Project Management Office (PMO) whose metrics in the performance arena is an appropriate mix of quantitative and qualitative metrics. The 3Ms profile is the performance metrics or objective by which this position would be measured.

This mix is derived from the process and template of the 3Ms where (at any level – individual, team, project, function or even at the dashboard or corporate objectives level), clients first define what the value of their contribution is. The word “value” means “the delivery of products and/or services that meet the expectations of the purchaser. Value is the metrics/quality of the deliverable and is judged as being at least equal and/or fair for the price/fee/salary being paid.

Another dimension of developing qualitative variables is to develop distinguishing standards to tease out the qualitative metrics of any performance outcome. These standards and their definitions are:

Satisfactory = which is defined as: meeting customer expectations, competent, delivering value or whatever words used in your environment,

Very Good = which is some degree of “better, faster, cheaper” with an added value of at least 10% as defined by the customer, and

Excellent = which is some many multiples of Very Good or a creative/conceptual breakthrough beyond Very Good.

Unsatisfactory = Anything less than or missing that is defined by Satisfactory

Instead of the rigorous four standards above, standards are supposedly defined by the use of the many tools of PM. These tools are typically focused on risk management, health checks (typically with use of green, amber, red) levels of “health” which are themselves aggregated ambiguous labels of other and more, lower level ambiguities.

The question (to an individual or to a team) is framed thusly:

In order to obtain at least a performance level of “Satisfactory”; what are the metrics that would determine this? Satisfactory is defined as earning your money (for an employee their paycheck, for a consultant their contracted fee). Whether the money was “satisfactorily” earned, is judged based on metrics defined by the client or company (project on schedule, under budget, results desirable).

The answers are typically:

Timeliness- is it on time?.

Financial - is it within budget?

Customer satisfaction - does it meet the customer’s expectation?

Efficient– did it maximize customer resources?

Effective – how well does it work?

Problem Resolution – there was minimum conflict and time needed to resolve issues at the level where they exist and needing to be “escalated” or moved upward within the reporting hierarchy of authority.

Examples of PMO Qualitative Standards The same logic occurs with the qualitative standards of a project management health check.

Satisfactory (2) = Project Management healthchecks are delivered on a monthly basis and complete

Very Good (3) = healthchecks are delivered bi-monthly (for greater timeliness to correct)

Excellent (4) = “create an online application enabling all project management data to be inputted at ANY time to be calculated in a healthchecks database or real-time monitoring which eliminates the separate task of doing healthchecks. The concept of healthchecks becomes like breathing, gravity, etc.

Now you will see the concepts above included in the PMO example below: In this case it is the 3Ms profile of a Director of the Project Management Office (PMO).

I. Key Result Areas or KVA with weightings (%)

Build PMO (40)

Direct PMO (40)

Staff Development (20)

II. Performance Objectives (with weightings, metrics, standards)

Build PMO 40%

20%: Issue initial program management process/templates: (4) by June 30; (3) by July 15; (2) by Aug 30; (1) Not issued in third quarter.

20% Staff PMO = (4) >90%; (3) >50% of identified PMO; (2) Roles and responsibilities of PMO resources documented. (1) Unspecified.

Direct PMO 40%

20%: PMO project lifecycle = (4) No project moves forward without following PMO processes. (3) All required documents are completed on targeted projects.; (2 Complete pilot project by 10/1/02; (1) later than 10/2/02

20%: Create master schedule = (4) All project interdependencies identified; (3) all projects on schedule included; (2) Key targeted projects included; (1) unspecified.:

Staff Development 20%

10%: PMO compliance to 3Ms process = (4) Performance rewards based upon 3Ms process; (3) PMO team 3M's externally aligned; (2) PMO team members all have 3Ms profiles; (1) unspecified.

10%: PMO staff learning = (4) 100% of learning goals completed; (3) PMO team members complete 50% of learning objectives; (2) All PMO team members have learning objectives (1) unspecified.

From another project involving security, here are more examples of qualitative metrics with standards of performance for the areas of: virus protection, hiring security staff, clearance and physical.

Virus:

1 = Opened with damage

2= Opened and disabled

3 = Not opened

4 = Never gets in

 

Hiring Security Staff:

1 = No security related certification

2 = Certified Information Systems Security Professional (CISSP)

3 = CISSP and CPP (spelled out)

4 = has held a role with corporate security accountability

 

Clearance of all employees (pre-determined appropriate clearances for different levels): This might be a PO that ALL C-level are Top Secret, etc.

1 = No clearance

2 = Confidential

3 = Secret

4 = Top Secret

 

Physical Entry:

 

1 = Entry with loss and/or damage

2 = Entry with no loss

3 = Failed attempted entry

4 = Deterred from entry

To further support the rigor that qualitative metrics have, here are a few non-PM examples. .

Example #1: The Foreign Service Institute (FSI) of the US State Department measures linguistic fluency for nine distinguishable standards of linguist fluency ranging from 0 = “knowing nothing” to 5 = “native speaker competency and accent”. Amongst certified testers, the interjudge correlation of ratings is .9, as proof that amongst judges who judge grammar, syntax and even the nonverbal accoutrements of the linguistic behavior, all the variables are known and accounted for. One example of qualitative metrics.

Example #2: Olympic performances that are balletic (ice skating, ring tossing, etc.) demonstrate that 10 judges from 10 different countries will rate this kind of performance 90% of the time, with a deviation or interval of 1 point about >80% of the time, with a deviation of only .5. And yet, the performance itself is about form, rhythm, poise, and level of difficulty (with its own metrics!)

Example #3: For a fast food restaurant, the metrics of the value for the $2.99 one pays is:

• Have an accurate order within 5 minutes,

• The food is hot,

• At least one napkin, plastic knife, fork, spoon,

• Food in a bag or a cardboard tray

• The food is conveniently arranged so that you can grab each item, even while you drive and are on the cell phone at the same time!

All the above are a mix of qualitative and quantitative metrics: quantitative = the price ($2.99); timeliness = within 5 minutes; # of eating utensils = at least one of each.

The qualitative metrics are: Hot, cardboard tray, convenience of access to the food while driving. Other example of qualitative metrics are from wine tasting, ballroom competition, restaurant reviews, movies, theatre, art, literature and music, where the metrics are qualitative, and amongst, experts in those respective fields, agreement, or in statistical terminology, interjudge correlation is near 1. Interjudge correlation being the degree of agreement between different judges

The way that people describe their life and their relationships, with spouse, lovers, children, parents, colleagues, is also a mix of qualitative and quantitative metrics. A child is doing “well” in school where the term “well” is a cumulative qualitative based on judgement of: grades, friendships, completion of homework, participation in extra-curricular activities, etc. So the term “well” is an aggregated metric that summarizes a myriad of qualitative and quantitative metrics. And, in fact, the examples (1-5) of qualitative metrics above are what people usually access to describe your work and social life.

The “3Ms of Performance: Measure, Manage and Magnify” is a total performance management system and presented often as a PMI-sponsored workshop at the Chapter and National levels. Mel can be reached at: mel@.

................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download