Figure 7. Change-point analysis of potency targeting. Data shows a decrease in the percent from target at lot 31. The potency
targeting improved from 9.9% to 5.8%. This change has a confidence level of 97% and a confidence interval of 15–45.
In the second case study, the parameter that was being monitored was potency targeting. Different potencies of active ingredient
are produced depending on the needs of the customer. To meet these needs, it is important that the actual potency be as close
as possible to the potency level required by the customer (referred to as the target). The effectiveness of potency targeting is measured by how far the actual potency is from the target potency (percent from
target). A recent improvement project sought to improve potency targeting. Process- and assay-related factors are both responsible
for the variation in targeting accuracy. In this project, assay improvements were implemented and a change-point analysis
was conducted on lots that bracketed the time of implementation of the improvements (Figure 7). Actual improvements were implemented
at lot 17, and the results of the change-point analysis indicate that a shift in the mean from 9.9% to 5.8% occurred at lot
31. The confidence interval for this change is from lots 15 to 45, which encompasses when the actual change occurred.
Table 2. T-test results from the percent recovery data
The statistical significance of changes identified by change-point analysis also can be verified by conducting a two-sample
t-test. This was done for the two examples described above, and the results confirmed the statistical significance of the change
(Tables 2 and 3). For each example, two t-tests were conducted. One compared data before and after the change point identified by change-point analysis, and the other
compared data before and after the actual lot when the process change occurred. The p-value for all the t-tests was <0.05, indicating that with at least 95% confidence, the means are not equal. The t-tests confirmed the change-point analysis results that a shift in the mean occurred.
Table 3. T-test results from potency targeting data
Because a normal distribution is an assumption for a t-test, it should only be used to verify the statistical significance of changes when data are normally distributed. Also,
the t-test by itself does not identify when changes occur. It can only be used to confirm a hypothesized point of change, and as
such is not a substitute for change-point analysis when used for problem solving and trending.
If the timeframe of a process change corresponds to the timeframe of a shift in the output of a measured parameter, one can
conclude that the process change may have caused the shift. Correspondence doesn't necessarily prove a cause-and-effect relationship
between the process change and the shift in output, however. The analysis should be supplemented with process knowledge, other
statistical analysis, or scaled-down experimentation to more definitively demonstrate cause and effect.