THE MEANING OF SEVERITY RANKINGS: HOW TO ASSESS POTENTIAL PATIENT HARM
Participants also struggled to understand the meaning of severity rankings. The ICH Q9 guideline instructs the industry to
consider harm in terms of patients. For some examples, however, like the cell culture step discussed in the workshop, it is
difficult to envision any problem—from contamination to aggregation—that wouldn't be resolved before the product reaches the
patient. Some cited improper glycosylation, but others pointed out that for many products, particularly monoclonal antibodies,
even carbohydrate structures would not necessarily affect product safety or efficacy.
The solution, many speakers said, is to use in-process product quality as a surrogate for patient harm when assigning severity
"The first time we piloted a risk assessment during our implementation of ICH Q9, we attempted to use a severity scoring criteria
that included harm to patients, and the highest severity score included serious injury or death," said Weese. "That didn't
work for upstream assessments begun in the middle of Phase 2, because in many cases, you just don't know how a potential failure
in a unit operation might cause direct harm to a patient. But we can estimate the potential impact on product quality attributes." As a result, he said, they decided it made more sense to use
the potential impact on a product quality attribute as a surrogate for potential harm to patients.
The connection to patient risk would be clearer, some said, if clinical staff participated in risk assessments. "We include
clinical staff when evaluating the need for product recalls, so I would think companies could benefit from including clinical
staff when assessing risks," said Kevin O'Donnell of the Irish Medicines Board. "Clinical staff do see things that we don't
in terms of risk to patients."
A few people responded that clinical staff participate in an earlier stage in the process. "We get input from clinicians in
an earlier step, when we are determining critical quality attributes," said Kathy Francissen of Genentech.
Others questioned whether any clinician could really be expected to predict the patient impact of a manufacturing change.
"Unless you have a very strong event, you are not going to get a clear linkage, because of extreme number of variables involved,"
said Wassim Nashabeh of Genentech.
Steven Kozlowski of the US Food and Drug Administration's Center for Drug Evaluation and Research (CDER) asked if anyone has
attempted to tackle this problem through data mining, using multivariate data analysis to compare clinical outcomes or adverse
event reporting to product attributes, perhaps by linking those data to lot numbers. "If we did this, would we find clusters
of data that would be useful?" he asked.
Stefanie Pluschkell of Pfizer said her company is trying to do just that. "It's complex, because the databases [of adverse
events and product lots] are not always directly linked," she said, adding that it also can be difficult to determine which
specific adverse events to look at, because they can be diverse. "But I am reasonably hopeful that this effort will be useful,"
SCIENTISTS AND SUBJECTIVITY
A number of participants also questioned the real value of risk-ranking scores, given their subjectivity. "Now may be the
time to improve the science of our decision-making, to increase our confidence in the results of these risk assessment exercises,"
said O'Donnell of the Irish Medicines Board. "There are other disciplines, such as experimental psychology, that we should
look at. Much research work has been carried out and that research can be helpful to us when we are carrying out quality risk
"Subjectivity will not disappear, and we should not strive for that," cautioned Rohin Mhatre of Biogen Idec. "The whole process
is very subjective, from deciding how many experiments to run, to which parameters to study. We should try to build a good
rationale for our risk assessments, but not aim for objectivity."
Nancy Waites of the FDA's Center for Biologics (CBER) agreed. "We're all scientists and we don't like ambiguity," she said.
"But you can't think of everything. You need to be comfortable with not knowing everything."
Terry Ocheltree of CDER also concurred. "I don't think anyone at FDA expects you to remove all subjectivity," he said. "What
is important is to have the tools and use them correctly. But that's not to say that everyone must run risk assessments in
the same way."
THE RIGHT TEAM AND THE RIGHT FACILITATOR
Various speakers stressed the importance of assembling the right team to conduct a risk assessment. "The scoring is as good
as who is in the room," said Amgen's Weese, emphasizing the importance of including staff from all key departments. "Our rules
say that if certain people are not present, we will not proceed," he added.
But don't underestimate the importance of a good facilitator, several people said. "It's not just a question of involving
the right subject matter experts, but also about the competence of the facilitator," said Keith Webber of CDER.
In particular, managing the people and personalities involved can be tricky. "Often we will have one very outspoken person
who dominates, especially if that person is an expert in the topic under discussion," said Weese. "We need to ensure we hear
from everyone else."
Vince Hamner of Talecris agreed, adding that those who disagree don't always voice their views. "Sometimes you get someone
who sits back and doesn't say anything, then suddenly speaks up at the end," he said.
A good facilitator, Weese added, can help the group reach decisions, particularly when doubt leads a group to score everything
conservatively. "If every risk number is extremely high, we are not achieving anything," he said.
But many also pointed out that there are few people in the biotech industry trained in risk management. Some suggested turning
to the medical devices industry, because medical device manufacturers have been conducting risk assessments for decades.