The Meaning of Metrics

Establishing, maintaining, and interpreting meaningful metrics has become an emerging industry issue. This topic has risen to prominence based on an article (1) and a Federal Register notice (2) that explored the question of what types of metrics should be applied to pharmaceutical operations, giving meaningful insight to their overall quality and compliance.

Susan J. Schniepp

There is no set requirement on what metrics a company should track to measure their overall performance. Each company should determine which metrics to track based on their operations, number of facilities they operate and where they are located, what types of products they manufacture, and what type of culture exists in their places of business.

DETERMINING WHICH METRICS TO TRACK

When establishing a metrics program, companies should evaluate numerous data input points including, but not limited to, product quality attributes, manufacturing site performance, people metrics, and quality system metrics. For product-quality metrics, companies should consider reporting on batch-specific data such as trending drug product, drug substance, and stability-test results against customer complaint rates. Indirect product quality metrics could include environmental monitoring, water trend results, and yield rates. When establishing site metrics the company could look at inspection history including internal audit findings and maintenance history such as equipment age versus defect failure rates. People metrics should consider ongoing job-specific training and education, skills and experience assessments, and employee turnover rate by job function and site. Quality systems metrics might look at change control, investigation root-cause trends, and release-testing cycle times.

The metrics chosen must be meaningful and written to provide a clear analysis of ongoing activities. It is important for operations and quality to agree on the metrics and how to report them to management to avoid overreaction to the data. It is not sufficient to simply report the data. The interpretation of the data is of crucial importance because it may include a root-cause analysis of its own.

EXAMPLE METRIC

Let's examine a simple metric and explore the hidden unintended behavior it might encourage:

Metric: Time from completion of manufacturing to approval of batch records.

Goal: All batch records are completed in 30 days or less after manufacturing.

Realistically, not all batches will be able to be released in 30 days or less for a variety of reasons including the fact that some complex investigations into root cause may take longer to resolve than the allotted 30 days. When considering how to report this metric, the organization should consider all possible reasons for achievement or non-achievement of the goal. This includes, in essence, a root-cause analysis to interpret the meaning of the metric. Evaluation of the cause and effect relationships are necessary before determining whether or not to revise the goal. If the goal is met most of the time with a few exceptions the data might indicate the batch release system is operating as intended. If the 30-day period is exceeded on a regular basis, the organization needs to consider why the 30 days are exceeded. Some of the questions to be asked might include:

• Were there too many errors in initial submission?

• Are people unable to prioritize their work?

• Were the records too complex?

Asking and answering these questions may offer solutions that can be used to streamline the batch release process so the 30 days can be consistently achieved.

CONCLUSION

Careful thought and consideration should be exercised when determining what to measure, how often to measure, how to interpret and communicate the data, and what the expectation is for using the data to drive positive change. Management needs to be cognizant of the fact that whatever metrics are chosen to be reported, they must be developed, evolved, and adjusted over time to maximize their impact on driving positive change.

When choosing a metric it is important that the architects of the metric are aware of unintended consequences that may inadvertently drive negative behavior. Management attempting to incentivize achievement of the goal such as offering a financial award if the goal is achieved, for instance, may lead to inappropriate behaviors that do not address the real issue. In these cases, it is generally not the metric that will drive the behavior but rather use of behavioral rewards. Reward for achievement rather than analysis of the real underlying causes will not lead to sustainable positive change. When managed properly, metrics are an important tool to help drive positive change and quality process improvements.

Susan J. Schniepp is vice-president of quality and regulatory affairs at Allergy Laboratories and is a member of the BioPharm International Editorial Advisory Board.

REFERENCES

1. J. Woodcock and M. Wosinska, Clinical Pharmacology & Therapeutics, 93 (2) (February 2013), www.nature.com/cpt.

2. Docket No. FDA-2013-N-0124, Food and Drug Administration Drug Shortages Task Force and Strategic Plan; Request for Comments, Federal Register, 78 (29) (February 12, 2013).