The pharmaceutical industry does not have a good track record of applying sound statistical principles to the investigation of out-of-specification (OOS) results. Recently, Steven Kuwahara presented an article on the history of the OOS problem highlighting some statistical deficiencies.

  In this article, we present some additional statistical principles. Failure to apply these principles causes unnecessary investigations, nearly guarantees recurrence, and wastes valuable time and resources. Control charts can help scientists and managers avoid these pitfalls.

In many factories and laboratories, the prevalent mindset is that if the test result under examination remains within specifications, not only does nothing more 

 to be said about that result, but also that nothing more 

 be said about it. This fundamental error leads to statistically significant signals that ought to warn of impending trouble being ignored. In turn, these overlooked or disregarded signals and disturbances eventually result in out-of-specification (OOS) results that should have been avoided. They also lead to an unnecessary increase in variation that, in accordance with Little's Law, damages not only quality, but also productivity.

 Any approach that blinds people to statistically valid signals and warnings is poor science. Control charts, which is a method of separating random from non-random variation, can help scientists and managers avoid these pitfalls.  

When a product batch is tested and is found to be out of specifications, the prevailing practice in the pharmaceutical industry is to study the faulty batch—independent of previous batches—to determine the cause of the OOS result. This can often lead to erroneous causal relationships, an incorrect determination of the root cause of the OOS result, and corrective action that almost guarantees future recurrences. The result is an increase in operational and compliance costs.  

Driving this approach to OOS testing is the "go, no-go" mindset so prevalent in the industry.  One real problem with the "go, no-go" mindset is that an OOS result must be recorded before any action is taken. Scientists tend to ignore the statistical signals. 

The top chart in Figure 1 shows the batch results against specifications and the lower chart is a Shewhart control chart (the moving ranges chart is omitted for clarity). 

Figure 1.  Two views of batch release results: batch release charts with specification and control limits. The top chart is the batch release chart versus specifications. The bottom chart is the same data as a Shewhart control chart. 

On two occasions, batches have tested as OOS. Because this is an isolated event, the first OOS batch (batch 18) and the circumstances leading to its production should be studied to determine the root cause of the problem and to take corrective action.  

However, to take similar action on the second OOS batch (batch 61) would be a mistake. Certainly, it is OOS, and an investigation must take place. Although batch 61 reported as OOS, the actual change to the process occurred at batch 41. The control chart shows two systems with a shift upwards in the process mean at batch 41. The second OOS result is at a random point in a stable system. Studying that batch alone will not reveal the true cause of the failure. 

To understand what changed and when, it is better to use a control chart. In this case, after the rise in the process mean at batch 41, it was only a matter of time until a random point fell beyond specifications. There are no statistically significant differences for any of the data after batch 41, including batch 61, which failed release testing.  

Figure 2 shows data from a chemical cleave step in a biologic process.

 The top chart shows the data with limits set at plus or minus three standard deviations. No points are observed beyond these 3 σ limits. Assuming the data are reasonably continuous, less than 1% of any data set can be expected to fall beyond 3 σ limits, regardless of how unstable the system may be, and irrespective of the functional form of the distribution involved. This is a central problem associated with the use of 3 σ limits. As the process is upset by special causes of variation or by drifts in the process mean, the 3 σ limits will grow without bound, making the process under examination look stable even when it is not.

Figure 2.  Two views of chemical cleave data: batch release charts set at three sigma and using Shewhart limits. The top chart is a run chart with specifications set at plus or minus three sigma. The bottom charts are the same data with Shewhart control limits set and process shifts incorporated (moving range chart is omitted from the Shewhart chart). 

The lower charts in Figure 2 show control limits calculated and placed according to Shewhart's methods, and much more information is available. It is important to note that in an individual point and moving range control chart, all the control limits are developed from the average of the moving ranges, rather than from the individual data themselves. It is the special relationship between the moving ranges chart and the control limits that allow control limits to be developed that are not significantly widened by upsets to the system, as so often happens with 3 σ limits.

 The estimate for σ using Shewhart's methods is nearly half of that calculated by the more usual root mean square deviation function found in calculators and spreadsheets. 

The process in Figure 2 is unstable. It is not behaving in a predictable manner and sooner or later will record an OOS event. The chart created using Shewhart's methods exposes several special causes and changes to the process mean. These signals provide clues for detecting changes to both manufacturing and analytical processes in an attempt to reduce variation and improve quality and productivity.  In this case, the Shewhart chart has identified where something changed the process average, and where upsets have introduced special cause events. Investigating these signals will lead to a better understanding of the process, and to proper identification of causality as well as the appropriate corrective action. 

The more traditional "average plus or minus three sigma" chart found no signals in the data. The Shewhart control chart found seven.  

The approach pioneered by Shewhart suggests that key variables be tracked in real time on a control chart.

 In this way, disturbances or changes to the process can be identified and corrected, often before an OOS result is recorded.  

One common analytical issue concerns effective comparison of the data against specifications. This becomes tricky when replication is used to combat analytical variability. According to FDA's guidance on Investigating Out-of-Specification (OOS) Test Results for Pharmaceutical Production, all OOS results must be investigated.

 FDA defines OOS results as, "all test results that fall outside the specifications or acceptance criteria established in drug applications, drug master files, official compendia, or by the manufacturer."

 It is acceptable to average multiple results when the sample is homogeneous or when the assay has a large amount of variability. It is not acceptable to average multiple results when testing for content uniformity or when the averaging will hide the variability between individual results in an investigation. Replicates must not be compared to the specification; only the final reportable result, generated by averaging all replicates that meet the acceptable limit for variability, should be compared to the specification. Consider the following set of analytical data for a representative assay method. This is a high-performance liquid chromatography (HPLC)-based method that determines the concentration of a therapeutic protein. As part of a variability reduction effort, it was determined that weighing was a significant source of variability (because of material hygroscopicity) along with the HPLC columns. Because the resultant concentration was used as part of the formulation process, it was important to reduce the variability to ensure a robust formulation process. Therefore, the analytical control strategy was:  

Three replicate weighings; each was solubilized individually to make a set of three solutions.

Each solution was injected once on two separate HPLC units.

All six replicates (three from each HPLC unit) were averaged together to get a single result.

The specification was applied to the final reportable result, but not to the individual replicates. (Note: the assay was not used for content uniformity.) This was the correct application of statistical principles. 

Analytical variability is a fact. It is impossible for two things to be identical. Ultimately, they may be similar enough so that the differences between them are rendered meaningless, but they are still different. No two HPLCs ever generate the same results; no two analysts ever prepare a sample the same way. In this world of variation, replication is used to reduce the analytical variability in the data generated to assist the production unit with generating quality product.  

For the case studies listed below, let us apply a specification of not less than (NLT) 25.0; OOS results will be indicated in bold. In each case study, a set of replicates for the assay will be shown. The more replicates generated, the closer the approximation should be to the true value.  

 "True" average concentration is 25.1. Using the random number generator in Microsoft Excel, the following six replicates were generated: 

Final value (rounded to the spec) → 25.1 

As seen in the data, Replicate #3 on HPLC #1 was OOS, despite a low relative standard deviation (%RSD) of 0.34%. (The actual %RSD of the assay was closer to 0.68% as determined by re-evaluation data. An OOS investigation into this result would be a waste of time because the OOS data were actually just a random event for a lot with a potency value close to the specification. Based on the analytical variability, it was likely that one of the replicates would be outside of the specification. Further, the point of the assay control strategy was to reduce the analytical variability by averaging across the critical sources of variability.  

 "True" average concentration is 24.9. Using the random number generator in Microsoft Excel, the following six replicates were generated: 

In this case, the "true" value is OOS. However, only three of the replicates (Replicate #1 on HPLC #1 and Replicates #4 and #5, which were run on HPLC #2) are OOS despite the "true" value and the average being OOS. Once the result is rounded appropriately, the result meets the specification. 

 An outlier has been substituted into the data set. The following six replicates were generated (except for the first data point): 

Articles

Features

Improving your quality operations by using sound statistical principles.

Avoiding the Pain of Out-of-Specification Results

The scale-up of processes from bench to commercial scale is challenging because of uncertainties about the physiochemical environment experienced by the cells at large scale. Process modeling tools can be useful in facilitating the scale-up and technology transfer of microbial and mammalian processes by ensuring that the process requirements can be met by the facility capabilities. 

Economics of scale and significant improvements made in the understanding of bioprocesses, design of equipment, and construction of facilities, have driven commercial production of biotech molecules to larger scales in the last decade. Microbial and mammalian commercial processes are routinely performed at 10,000 L or greater amongst the major biotech companies and contract manufacturing organizations. However, the scale-up of processes from bench to commercial scale is challenging, with uncertainties about the physicochemical environment experienced by the cells at large scale. Some of the scale-dependent bioreactor differences are not readily influenced. As such, these differences need to be fully understood to determine the potential impact on culture performance. 

 series and focuses on process modeling tools that can be useful in facilitating the scale-up and technology transfer of microbial and mammalian processes. Issues addressed include mixing and gassing characteristics, pressure differences, and heat transfer rates (fermentation). The underlying principles are presented along with examples from existing biotech processes.  

Bioreactor mixing is required to deliver process requirements for mass transfer of dissolved oxygen and carbon dioxide (in cell culture), and to aid in convective heat transfer for control of temperature (in fermentation). In addition, mixing strives to achieve bioreactor liquid homogeneity with respect to cell density, gas dispersion, and chemical gradients (pH and nutrients). In the case of mammalian cells, however, excessive mixing by impeller agitation or gas sparging may result in cell damage because of mechanical hydrodynamic shear (e.g., microscale eddies) or interfacial shear by bubble bursting events.

 As a result, a balance between cellular requirements, cellular sensitivities, and bioreactor capability is key to defining a comparable operating space that ensures consistent process performance between scales. 

Gradients in pH are influenced by bulk mix times, the type and concentration of titrant used (acid or base), pH control strategy, pH probe location, and titrant addition location. To determine the impact of pH gradients on cell culture performance, it is important to understand the magnitude of the pH excursion with regard to deviation from set-point and the excursion duration. Once this is established, the potential impact on process performance can be evaluated and resolved as required. The magnitude and duration of the pH gradients can be measured directly by control simulations in buffer solutions at large scale.

 However, this can be challenging with regard to pH probe location and probe response time.  

Computational fluid dynamics (CFD) can be used to model pH gradients with bolus titrant additions, and reasonable agreement with wet testing has been observed for base addition at 2,000 L (Figure 1). Initial discrepancies in the magnitude of the pH deviations over the first 22 seconds between wet testing and CFD are attributed to differences in measured and assumed locations for base addition and pH probe placement. Although CFD can be applied to assess bioreactor design capability and process scale-up, care should be taken to assess CFD predictions against the model assumptions and verified with wet testing.  

Figure 1.  Comparison of transient pH profiles from the wet test and computational fluid dynamics (CFD) simulation 

A solution to minimize pH gradients and prevent over-addition of titrant is to improve bulk mixing times by increasing the agitation speed in view of the cell sensitivities to hydrodynamic shear and foaming. Another best practice is to direct base addition to a well-mixed bioreactor location, ideally subsurface (adjacent to impellers) In large bioreactors, the distance between the online control probe (typically located in the impeller region) and base addition location (typically at liquid surface) can lead to differences in measured and actual pH values at the titrant addition. Optimization of the titrant addition strategy (pulse frequency and volume) with respect to bulk mix times may also  be required. Finally, the pH control strategy in relation to pH "dead band" or "controller dead zone" implementation may require fine-tuning to prevent acid-base cycling. Some other important considerations with pH control are sample handling, offline pH measurement, and probe standardization. Inconsistent practices with scale-up and process transfer may contribute to differences in pH control and process performance.  

Although dissolved oxygen (DO) generally is  kept constant between scales, attention to DO probe zero and span calibration are required to fully represent process conditions. The relationship between DO saturation (%) and partial pressure (mm Hg) should be taken into account between sites and scales. The impact of bioreactor hydrostatic pressure at large scale also needs to be considered on dissolved gasses (DO and pCO

). Depending on the mixing characteristics, liquid height, and location of the controlling DO probe, the actual DO (and pCO

) experienced by the cells will vary depending on their location in the bioreactor. In combination with agitation power input per unit volume (P/V), the superficial gas velocity will influence the oxygen mass transfer rate (k

a). At typical agitation speeds with nonclumping mammalian cells, cell culture processes are considered more sensitive to interfacial shear by bubble bursting events than bulk hydrodynamic shear or microscale turbulence (Kolmogorov eddy size). Superficial gas velocities should be modeled to provide sufficient kLa at the selected agitation speed but not so overly aggressive as to incur cell damage, excessive gas stripping, or foaming.  

For high cell density microbial processes, additional concerns exist around oxygen consumption and heat removal. Oxygen uptake rate (OUR) has a strong correlation with the heat generation rate and carbon evolution rate (CER). This is because of the relationship between cellular respiration and metabolism in aerobic microorganisms and the high cell densities that allows these values to be accurately determined. These data are important whenever a process is transferred into a new facility or increased in scale so that the process requirements (peak OUR, peak heat generation rate) can be assessed against the facility capabilities (maximum oxygen transfer rate, maximum heat removal rate).  

As mentioned above, selection of appropriate mass transfer conditions is important to achieve consistent cell culture and fermentation performance across sites and scales. During scale-up, for most situations the tank geometry and hardware (impellers, sparger, baffles, etc.) are not subject to change. Therefore, the focus is on defining appropriate agitation and sparge gassing conditions to achieve consistent performance across sites and scales.

For industrial-scale cell culture bioreactors, mixing times are typically on the order of three minutes or less. Mixing times of this order of magnitude have been sufficient to achieve consistent performance across scales for many industrial cell culture fed-batch processes. They have historically been estimated from impeller discharge rates using an equation such as

 is the mixing time, V is the liquid volume, N is the agitation speed, D is the impeller diameter, N

 is the impeller flow number (provided by the impeller manufacturer), and k is a constant related to the number of volume turnovers required to achieve homogeneity. Typically, five volume turnovers are assumed to result in homogeneity, but the actual impeller pumping rate is higher than the impeller discharge rate because of momentum transfer to the surrounding fluid. Therefore a multiplier, k, of three is often used. More recently, it has been suggested, based on turbulence theory, that the mixing time should be independent of impeller type

 is the tank diameter (m), D is the impeller diameter (m), P is the impeller power input (W), V is the liquid volume (m

). More accurate mixing times can be calculated using more complex models, for example, those based on CFD as described earlier. 

Shear rate is another agitation-related parameter that is often evaluated, especially for cell culture applications. Because mammalian cells lack a cell wall, they are more susceptible to shear damage than microbial cells. Impeller tip speeds can be correlated to maximum impeller shear rates, and therefore, constant tip speed has been suggested as a scaling criterion for mammalian cell culture agitation.

 However, it has been shown by multiple investigators that impeller shear rates commonly used for cell culture applications are orders of magnitude below the shear rates required to cause cell damage.

 Therefore, use of tip speed as a primary agitation scaling parameter is not recommended.  

Power input (P/V)or mean specific energy dissipation rate is a parameter that is more commonly used for scaling agitation across sites or scales. The mean energy dissipation can be calculated from: 

 is the mean energy dissipation rate (W/kg), P is the power input (W), V is the liquid volume (m

 is the impeller power number. Mean impeller energy dissipation rates typically used for mammalian cell cultures range from 0.001 to 0.050 W/kg (1 to 50 W/m

 A consequence of scaling agitation at constant energy dissipation is that the length scale of turbulent eddies is expected to be constant. The Kolmogorov microscale of turbulence is related to energy dissipation through the following equation: 

in which λ is the Kolmogorov length scale (cm), υ is the kinematic viscosity (cm

When the length scale for turbulent eddies approaches the length scale of the cells in culture, cell damage may occur. This theory has been applied to microcarrier cultures, in which cell death can be correlated to eddy lengths that are smaller than the microcarrier diameter, and to suspension cell cultures, where cell aggregate sizes were found to correlate with eddy length.

 It should be noted that energy does not dissipate uniformly over the entire bioreactor volume, and that local energy dissipation rates may differ significantly compared to the mean energy dissipation rate. Despite this caveat, the mean energy dissipation rate is most often used in agitation scaling calculations. 

Figure 2.  Impact of agitation speed on product titer in a scale-down production bioreactor model  

Case Study: Scale-Up of a Mammalian Cell-Culture Unit Operation 

In a case study involving technology transfer of a biotech process, we evaluated the impact of agitation speed on cell culture performance. Agitation speed was varied in a small-scale model bioreactor over a range that encompassed both the tip-speed and energy dissipation rate calculated for the full-scale bioreactor. As shown in Figure 2, we found that product titer correlated with agitation speed across this range. The cause for the titer correlation was found to be specific productivity, rather than cell growth. When the small scale model data are compared to manufacturing scale data as shown in Figure 3, it is evident that the specific productivity matched across scales for the agitation scaled by energy dissipation (275 rpm), whereas the specific productivity was lower for the constant tip speed condition (500 rpm).   

Figure 3.  Cellular specific productivity at manufacturing scale compared to a scale-down model operated with tip-speed agitation scaling (500 rpm) and energy dissipation scaling (275 rpm). Statistical analysis by Dunnett's method shows that scaling by energy dissipation more accurately reflects the manufacturing scale performance versus scaling by tip speed. 

Further, on visual observations of microscopic cell count images (Figure 4), cell clumping observed when agitation was scaled by constant mean energy dissipation (275 rpm) was more consistent with the manufacturing scale than when agitation was scaled by tip speed (500 rpm). These results are consistent with observations reported in the literature that the size of cell clumps is related to the Kolmogorov eddy length.

 The turbulent eddies serve to disrupt cell aggregates that are comparable in size to the turbulent length scale. Because the eddy length scale is a function of energy dissipation rate as described above, cell clumping is expected to correlate with energy dissipation rate. A possible explanation for the increased cellular specific productivity is that the cells in moderate sized clumps are more productive than individual cells. Another possibility is that cell counts are less accurate for the highly clumped cultures, and the apparent changes in cell-specific productivity could be related to changes in cell density. Regardless, it is clear from these results that the best match to the manufacturing scale data was achieved with a constant energy dissipation rate scaling strategy, rather than constant tip speed scaling. 

Figure 4.  The impact of agitation speed on harvest day cell aggregation. Cell clumping (middle figure) was more consistent with the manufacturing scale (left figure) when agitation was scaled by constant mean energy dissipation than when agitation was scaled by tip speed (right figure).  

In addition to facilitating liquid/ liquid mixing, agitation conditions can impact gas/liquid mass transfer. Oxygen is supplied and carbon dioxide (CO

) removed from most industrial cell cultures through sparging of gas bubbles through the bioreactor broth. The bubble gas/liquid mass transfer is a function of agitation conditions and the characteristics of the sparge gas inlet. The impact of these parameters on bubble gas/liquid mass transfer is most often described in terms of the mass transfer coefficient, k

in which P/V is the impeller power input normalized to liquid volume, Q is the sparge gas flow rate and k, α, and β are empirically fitted parameters. The fitted parameters can vary as a function of the sparge inlet characteristics, the most important of which may be orifice size. The size and number of air inlet orifices determine the superficial gas velocity and the resulting bubble size. Bubble size impacts the surface area to volume ratio of the bubble, and consequently the mass transfer efficiency. Smaller bubbles have a greater surface area relative to volume and hence they lead to more efficient mass transfer per volume of gas. Despite their mass transfer efficiency, very small (" 1 mm) bubbles are not favored for use in industrial mammalian cell culture for multiple reasons. A significant problem associated with sparging of mammalian cell cultures is cell death because of high shear, which occurs primarily because of bubble rupture at the liquid surface.

 Surfactants such as Pluronic F-68 are typically added to cell culture media to prevent cells from sticking to bubbles as they rise through the liquid, but damage can still occur to cells in the vicinity of bubble ruptures.

 Larger bubbles have lower maximum energy dissipation rates associated with bubble ruptures, which is consistent with literature observations that larger bubbles are less damaging to cell cultures.

 Further, small bubbles lead to less effective removal of carbon dioxide. Because of their high mass transfer efficiency, volumetric gas flow rates required to control dissolved oxygen at set-point are minimized, and hence, the bubbles saturate with a higher concentration of CO

. Accumulation of carbon dioxide can have a negative impact on the cell culture.

 By using larger bubbles, higher volumetric gas flows are required to control dissolved oxygen, and CO

 Finally, smaller bubbles lead to generation of more stable foam layers, and hence higher concentrations of antifoam may be required. For these reasons, larger bubbles (on the order of 5 mm) are preferred for industrial cell culture.

HEAT AND OXYGEN TRANSFER IN BIOREACTORS FOR HIGH CELL DENSITY MICROBIAL PROCESSES

Oxygen update rate (OUR) is defined as the rate of oxygen that is consumed by a given volume of fermentation broth, and this variable fluctuates over the course of a fermentation process depending on many factors such as the metabolic state of the cells, the level of oxygen saturation in the culture, and the volume of the culture itself relative to the cell density. Carbon dioxide evolution rate (CER) is defined as the rate of CO

 evolution from a carbon nutrient source (usually glucose in fermentation processes) and can be used to determine the level of glucose utilization in the culture relative to cell density and the state of cellular metabolism through comparison with OUR. For most fermentation processes, the desired ratio of CER to OUR (RQ) is approximately 1.0. This correlates to six CO

 molecules produced for every six oxygen (O

) molecules that are consumed during the aerobic metabolism of a six-carbon sugar such as glucose. An RQ value that is noticeably lower than 1.0 indicates that the cells are using a different nutrient source for energy. In most industrial fermentation situations, it is desirable to maintain the RQ above 0.75 for the duration of the culture to ensure that the metabolic state of the cells is consistent. OUR and CER can be readily calculated using the following expressions:

into the fermenter in standard liter per minute (SLPM); Q

 is the volumetric flow rate of air into the fermenter in standard liter per minute (SLPM); V

 is the molar volume of Air in the inlet air (L/mol); X

). The inlet gas flow rates and the molar concentrations of the outlet gases can be measured by a volumetric gas flow meter or an in-line mass spectrometer. Derivation of equations 6 and 7 is based on the assumption that the water content in the gas does not affect the composition of the gas or the accuracy of the mass spectrometer reading.  

Total heat generation rate in a bacterial fermentation system is the sum of all of non-negligible sources of heat generation and heat loss that typically consist of cellular metabolism, agitator motion, evaporation, feed addition, and gas addition. The following formula can be used to calculate the net rate of heat generation (H

 is the heat generation because of cellular metabolism, H

 is the heat generation because of mechanical motion of the agitator(s), H

 is the heat loss because of addition of feed held at room temperature, and H

 is the heat loss because of addition of supply air/O

 can be directly correlated to OUR and has been shown to account for approximately 80% of the net heat generation (i.e., heat generation from agitator motion accounts for ~20% of heat generation). However, when all major sources of process-related heat loss are taken into account, H

Because culture temperature is typically controlled at a constant setpoint during a fermentation process, H

. Hence, metabolic heat generation rate can be measured fairly accurately by calculating the amount of heat removed from the cooling jacket (H

The rate of heat removal through the cooling jacket and cooling coils can be determined from the outputs of the temperature control system on the fermenter. In a commonly used control configuration, the temperature of water recirculated through the jacket or coils is regulated by an incremental addition of chilled water. The water in the jacket is warmed up through contact with the fermenter wall, and after it exits the jacket, a regulated amount of chilled water is added to cool down the water. It then enters the jacket again through a loop, acting as a heat exchange fluid to remove as much heat as needed from the fermenter to control the temperature at the setpoint. Using this setup, the heat removal rate can be calculated using the following formula:  

Process-modeling tools can ensure smooth tech transfer.

Modeling of Biopharmaceutical Processes—Part 1: Microbial and Mammalian Unit Operations

This article is the second in a four part series on essential statistical techniques for any scientist or engineer working in the biotechnology field. This installment presents statistical methods for comparing sample means, including how to establish the correct sample size for testing these differences. The difference between one-sample, two-sample, and z-test also are explored. 

In hypothesis testing, we must state the assumed value of the population parameter called the null hypothesis. The goal of hypothesis testing is to verify if the sample data is part of the population of interest. You either have sufficient evidence to accept the null hypothesis or reject it—you do not prove it. The significance level or p-value indicates the likelihood that the sample comes from the population of interest. Statisticians usually use a p-value of 0.05 as the cutoff for statistical significance. In other words, a p-value less than 0.05 is sufficient evidence to reject the null hypothesis. Typically, the null hypothesis is a statement about the value of the population parameter. For example, μ = 100 versus μ ≠ 100. A one-sided test means we are testing the null hypothesis of either less than or greater than. A two-sided test means we are testing the null hypothesis of less than and greater than. 

The one-sample t-test is used to compare a sample mean to a hypothesized population mean. The hypothesis can be either a one-sided or two-sided test. Usually, the population variance is unknown requiring use of the t-distribution, which takes into account the uncertainty in estimating the sample variance. The t-distribution is tabled by confidence level and degrees of freedom. For the one-sample t-test, the degrees of freedom are the number of observations used to estimate the sample standard deviation minus one. The formula for the one-sample t-test is as follows: 

in which X-mean is the sample mean, μ is the theoretical population mean, s is the sample standard deviation, and n is the sample size used to estimate the mean and standard deviation.   

 is greater than the tabled value from the t-distribution, the sample mean is statistically different than the population mean (μ). An example of a one-sample t-test would be comparing protein concentration for a particular batch to a theoretical protein concentration. Table 1 shows an example of a two-sided one-sample t-test for protein concentration. The hypothesis is that the lot is not statistically different than 30 (μ = 30). The mean of the six vials was not statistically different than the theoretical value of 30 (p = 0.223). The 

 of 1.355 did not exceed the tabled value for a 95% confidence level with five degrees of freedom of 2.571. 

Table 1.  An example of a two-sided one sample t-test for protein concentration. The hypothesis is that the lot is not statistically different than 30 (Î¼ = 30). 

The two-sample t-test is used to compare two different sample means. Similar to the one-sample t-test, the hypothesis can be either a one-sided or two-sided test. The t-distribution is required here because instead of population mean, two samples are being compared. For the two-sample t-test, the degrees of freedom are the number of observations for sample 1 (n

) plus the number of observations for sample 2 (n

) minus two. The formula for the two-sample t-test is shown in the following equation:

 are the standard deviation of each mean, and n

 is greater than the tabled value from the t-distribution, the two sample means are statistically different. An example of a two-sample t-test would be comparing the existing lot to a previous lot of material. Table 2 shows an example of a one-sided two sample t-test for purity. The hypothesis is that the new lot is not less pure than the old lot (New ≥ Old). The mean of the new lot is statistically less pure than the mean of the old lot (p = 0.003). The 

 of 3.49 exceeds the tabled value for a 95% confidence level with 10 degrees of freedom of 1.812. Notice that the tabled value of a one-sided 95% confidence level is the same as the two-sided 90% confidence level. 

Table 2.  An example of a one-sided two- sample t-test for purity. The hypothesis is that the new lot is not less pure than the old lot (New â¥ Old). 

The z-test is probably the most misused and misunderstood statistical test in biopharmaceutical organizations. The misconception is that if the sample size is large enough (n > 30) then the z-test is appropriate. The z-test is restricted for only those situations where the population variance is known. Because it is practically impossible to know the population variance and computers can calculate the t-distribution for any sample size, it is preferred to use the t-test. If it is truly necessary to use the z-test, the formula is: 

in which X-mean is the sample mean, μ is the theoretical population mean,   σ is the population standard deviation, and n is the sample size used to estimate the mean. 

Notice that the formula for the z-test is similar to the formula for the one-sample t-test. The only difference between the two equations is the estimation of the standard deviation (square root of the variance). The t-test uses the sample standard deviation and the z-test uses the population standard deviation. The normal distribution values do not require a sample size because the standard deviation is known and, therefore, the tabled value is the same regardless of sample size. Table 3 shows a comparison of the z-values to the t-values for sample sizes of 30, 60, and 120. The t-distribution, though close to the normal distribution, does not converge to normal distribution until the sample size exceeds 120. 

Table 3.  A comparison of the z-values to the t-values for sample sizes of 30, 60, and 120.  

Most people use a p-value equal to or less than 0.05 as the criteria for rejecting the null. The probability of rejecting the null hypothesis when the null is true is called Type I error. The more critical error is the Type II error where you accept the null hypothesis when the null is actually false. Because most hypothesis testing in the biopharmaceutical industry sees its greatest use in comparing a previous lot to the new lot or comparing a sample to a known value, accepting the wrong answer can be detrimental.  Luckily, there are ways to minimize the risks of making a wrong decision in hypothesis testing. 

For example, increasing the sample size can minimize the risk. Alternatively, increase the difference needed to be considered statistically different will reduce the risk. The normal distribution can be used to get a rough estimate for the correct sample size. Software such as Minitab and JMPuse a noncentral t-distribution to calculate the sample size. The following equation gives the normal approximation to the sample size calculation: 

 is the number of samples to be calculated, S is the sample standard deviation, Δ is the difference to detect, Z

 is the Z-value for the α error (a two-sided 0.05 α would be 1.645) and Z

 is the Z-value for the β error (a two-sided 0.10 α would be 1.281). 

The α risk is the probability of rejecting a good lot; this is sometimes called the producer risk. The β risk is the probability of accepting a bad lot; this is sometimes called the consumer risk.  

If the sample size gets too large or the variability is too small, a hypothesis test might conclude a statistical difference, when the difference observed is not clinically relevant.  

One proposed method that combines both the statistical rigor of hypothesis testing and the appropriateness of meaningful differences is to set a minimum difference that must be obtained to be considered different. The correct selection of the minimum difference value is still being debated in the statistical and scientific community.  

Although a powerful statistical method, hypothesis testing can lead to false conclusions if applied incorrectly. Whenever possible, use the t-distribution over the z-test and normal distribution because the population standard deviation is never known for sure. Using the correct sample size and power analysis can lead to robust comparisons, especially if a minimum difference is prespecified.  

is the president of Statistical Outsourcing Services, Olney, MD, 301.325.3129, 

steven@statisticaloutsourcingservices.com

Columns and Departments

How to use hypothesis correctly, and understanding the difference between one-sample, two-sample, and z-test.

The Power of Hypothesis

The growing acceptance and adoption of disposable technologies raises the question, What is the cost impact of these technologies? This question sounds simple but hides complexity, in terms of what is included in the cost evaluation and the evaluation methods used.  

Disposable technologies have the potential to significantly reduce plant complexity and initial start-up capital costs.

 The benefits of disposable technologies, however, extend beyond reducing start-up costs, and also can include:  

reduced time to market, based on a quicker build and validation of facilities

reduced capital investment, resulting from reduced capital infrastructure

improved flexibility in operation, in terms of managing process and product change

As a result, many companies now wish to understand the broader cost impact of disposables. The critical question is, What is the best evaluation method?  

To evaluate costs, a robust cost model is required, in which the methodology and assumptions are transparent. Such a model will provide managers with better insight into the key cost drivers of the manufacturing process and the sensitivity of the overall cost of goods to changes in these key parameters. A good model helps evaluate the cost impact of implementing different technologies and the effect of process changes, such as increasing product titers and yields, and can be validated with financial accounting data.  

Some of the commonly used methods for evaluating projects are net present value (NPV), internal rate of return (IRR), return on investment (ROI), and cost of goods sold (COGS). In biopharmaceutical manufacturing, the COGS model is by far the most commonly used method. Although the COGS model has the merit that most people in the industry understand it, it is not the most rigorous or sophisticated method. For example, a COGS model should not be used when there is a need to understand the interplay among the expenditures, timing, and project risk. 

The other three methods, NPV, IRR, and ROI, all provide useful tools for decision-making, particularly for capital investment and project approval. NPV and IRR are particularly useful for evaluating potential long-term projects because they account for the upfront investment required as well as the timing of future cash flows and the risk and opportunity cost of the project. Given that disposables can delay and reduce capital expenditure, these methodologies are able to capture that benefit together with the cash flows associated with manufacturing operations.  

The NPV methodology is the best technique to analyze alternative technologies and manufacturing strategies, because it accounts for the impact of delays in expenditures and properly accounts for the time value of money.  

No matter which manufacturing cost model is chosen, it is important that certain management accounting techniques, such as lifecycle cost analysis and activity-based costing, be incorporated into the model to account for the significant effect manufacturing efficiency has on the cost of goods. For example, the number of successful production runs per year and the cost of facility downtime and batch failure often have a much greater effect on overall manufacturing costs than  changes in raw material or labor costs. Also, one should exercise care in comparing cost figures from different sources. Common problems when comparing data relate to consistency in methods and assumptions, especially when it comes to handling capital costs. 

Once the optimum cost analysis model has been identified, the next step is to define the purpose of the analysis, for example, technology evaluation, supplier evaluation, or operations optimization. 

Table 1.  Operating sequence for a typical reusable column  

The purpose to some extent determines both the detail and the scope of the analysis. If, for example, one wished to evaluate the operational effectiveness of an individual operation, such as buffer preparation, then it is often appropriate to look at the operation in isolation. It is also important, however, to consider all the affected systems. In the following example, we compare a disposable chromatography cartridge as a replacement for a conventional chromatography column in flow-through mode (Tables 1 and 2). This example compares the Sartobind module (Sartorius Stedim Biotech, Goettingen, Germany) with a reusable column processing 6.5 kg of monoclonal antibody (MAb) per batch in a flow-through anion column.

Table 2.  Operating sequence for a typical disposable column 

The simple approach would be to compare the two operations in terms of the stage throughput, labor, column costs, other consumables, and material costs. To properly compare these two options, however, the scope of the cost model needs to be wider. It should also address the impact on the buffer preparation and buffer hold operations and on maintenance and quality operations. It can also raise other questions. For example, Does the disposable column need a chromatography skid? By taking a broad approach and considering all the factors, it is possible to show the full impact of disposables. The results of our example are illustrated in Figure 1, in which the dramatic reduction in capital and materials is evident.  

Figure 1.  A comparison of the costs of the conventional and Sartobind disposable membrane chromatographic technologies, according to four major categories: capital charges, consumables, materials, and labor. 

Taking the analysis one step further, to understand the overall impact of disposable technologies on manufacturing costs it is necessary to build a cost model that covers the entire manufacturing process. A comprehensive analysis makes it possible to assess a particular technology on the overall COGS and thus helps determine whether the technology should be pursued. For example, referring to the previous analysis, does the membrane chromatography significantly reduce overall COGS, or should alternative technologies, such as storing buffers in bags, be pursued instead? Another benefit of a model that covers the whole process is that if constructed correctly, it should provide a framework for "what if" analyses to examine the cost impact of changes in  

scale of operation (e.g., bioreactor volumes and titers) 

process options and improvements (e.g., changes in materials, efficiencies, titers)

WHICH UNIT OPERATIONS PROVIDE THE GREATEST SAVINGS

When assessing the impact of disposables on manufacturing costs, it is interesting to consider that for a stainless-steel MAb manufacturing operation at large scale, most of the capital investment is in support infrastructure (e.g., preparing buffers, media, utilities, clean-in-place [CIP], and steam-in-place), followed by the bioreactors and then by downstream processing. Based on some capital breakdown figures presented in 1999, these areas amounted to about 52%, 32%, and 16%, respectively of capital investment.

 In addition, if one considers that most (between 55% to 85%) of the water used in a stainless-steel biotech facility relates to cleaning reusable equipment (i.e., CIP), then it follows that technologies that reduce the extent of the support infrastructure (e.g., buffer and media preparation and the holding of buffer media and product-containing solutions) will reduce overall costs. Based on capital dominance, one would expect the technology with the next largest impact to be bioreactors. Therefore, one can propose the technologies with the most potential to reduce MAb operating costs to be: 

1. holding bags and associated fluid management 

2. mixing systems for solution preparation (non-aseptic) 

The most mature technology listed above is item 1. The benefits of fluid-handling technologies have been demonstrated in practice, with data in recent papers showing that if the scale fits, storing solutions and products in disposable bags can significantly reduce both operating and capital costs.

 For example, operating costs can be reduced by about 17% and capital costs by about 40% (based on a 1,000-L scale perfusion MAb process).

I hope I have given some insight into how cost models are being used to evaluate the use of disposables technologies in biopharmaceutical manufacturing. I believe that the current focus on COGS models is underplaying the benefits of disposables technology. Typically, disposable technologies result in quicker builds of new facilities or manufacturing suites and reduced capital expenditures. The financial analysis should capture these benefits by accounting for this delay in spending (thereby reducing the project risk). The best method for accounting for these benefits is through the use of NPV analysis.  

is the managing director of Biopharm Services, Chesham, Bucks, UK, +44 1494 793 243, 

 1. Sinclair A, Monge M. Biomanufacturing for the 21st century: Designing a concept facility based on single-use systems. Bioprocess Int. 2004 Oct supp;26–31.  

 2. Lim J, Sinclair A, Kim D, Gottschalk U. Economic benefits of single-use membrane chromatography in polishing: a cost of goods model. Bioprocess Int. 2007 Feb; 5(2):48–56. 

 3. Van Reiss R. Genentech. IBC IBC Bioeconomics Conference; 1999; Washington, DC. 

 4. Lim J, Sinclair A. Process economy of disposable manufacturing. Process models to minimize upfront investment. Amer Pharm Rev. 2007 Sept/Oct;10(6):114–21. 

 5. Sinclair A, Monge M. Concept facility based on single-use systems, part 2. Leading the way for biomanufacturing in the 21st century. BioProcess Int. 2005 Oct supp;51–55. 

The current focus on cost-of-goods (COGS) models is underplaying the benefits of disposables technology in biopharmaceutical manufacturing. The best method for accounting for the benefits of reduced and delayed capital expenditures is through the use of NPV analysis.

How to Evaluate the Cost Impact of Using Disposables in Biomanufacturing

The selection of a contract manufacturing organization (CMO) can be a real challenge for many drug manufacturers. The situation is even more critical when clinical material is concerned, because of stringent regulations that must be followed while process development may still be in progress. 

Thus, it is particularly important to understand the most critical practical aspects that must be considered when selecting a CMO for the production of clinical material.  

The selection of a CMO usually is based on an audit conducted at the facility where the material is to be produced. The audit should include verification that the premises are adequate to manufacture the product at the scale required for the clinical trial, an evaluation of the quality systems in place, a review of personnel skills, and a review of the available equipment.  

A few of the key areas to audit at a CMO include its containment strategy, its cleaning procedures, its change control process, its engineering department, and its management strategy. 

The CMO's containment strategy should be reviewed, particularly if more than one manufacturing suite is available, and if both mammalian cells and bacteria or yeasts are used in the same facility.  

The material, personnel, and waste flows should be examined to detect any possible risk of (cross-) contamination. This should include both verifying the operability of physical barriers, such as pressure cascades and airlocks, as well as the use of appropriate operation procedures designed to reduce the risks of (cross-) contamination. 

Validated cleaning procedures must be developed for each product that will be handled by the CMO. The equipment used in biopharmaceutical processes should be dedicated to the type of host used (mammalian, bacterial, plant, etc.) to avoid any risk of cross-contamination.  

The contract between the client and the CMO should specify who is responsible for developing the cleaning procedures and who is responsible for the preparation of the cleaning validation protocols.  

Because process development may not be completed when clinical material needs to be produced, one of the most important characteristics of a contract manufacturer is its capacity to manage process changes. The CMO must show evidence of its willingness and ability to adapt equipment, personnel, quality system, quality control, validation, and operations as needed. 

For example, specific processes may require modifications in the standard operating procedures (SOPs) (e.g., modifications of environmental monitoring), so the quality system must be flexible to enable quick turnovers between product lines. 

The CMO must also demonstrate flexibility in the equipment available for production. The equipment required may differ significantly depending on the process, and the sponsor should always review the scales and options available. Even if an automatic vial filler is available, for example, the CMO might not have the required adaptor for the vials used. Such a modification may also result in changes in the associated software, which would then require validation. Similarly, a bioreactor may show an adequate height-to-diameter ratio, but not be equipped with the appropriate mixing device. All those aspects should be reviewed, and the contract should specify who is responsible for acquiring the missing equipment as well as for the associated qualifications. 

A skilled engineering department is required to ensure the flexibility of equipment, services, and utilities. Equipment adaptation and even custom design may be required for the production of clinical material. Good contacts with suppliers of small-scale cGMP equipment, services, and utilities help perform the required modifications quickly, ensuring adherence to the established timeline. The CMO's engineering team can prepare technical specifications for modifications to be performed and either implement those changes itself or have a supplier modify the equipment accordingly.  

Once the CMO's compliance has been assessed and the sponsor is certain that contamination and cross-contamination can be prevented, it is time to look at the CMO's management strategy. The way a CMO is managed can seriously affect product quality as well as the cost of clinical trials. 

All CMO personnel involved should be properly trained on cGMPs, technology transfer, regulatory requirements, and the management of clinical trials. A multidisciplinary team that includes representatives from all departments should be built. This team should be informed of the objectives, timelines, and budget for the project.  

It is also recommended the  sponsor be involved in all aspects of the technology transfer. Notwithstanding the application of the CMO's internal politics, a project manager should be on site as required and invited to observe all the steps of the manufacturing process. 

The sponsor should require the CMO to demonstrate both the seniority of the personnel involved in the project and a low turnover rate of key personnel. Personnel stability is an indication of good management and ensures project continuity for the duration of the entire clinical phase. 

Though challenging, choosing the right CMO is an important step in ensuring the quality of clinical materials. It is important to understand critical aspects of the CMO's capabilities. Only by auditing these key areas can the sponsor be assured of the quality of the materials produced.  

is the vice president of operations, and 

is the director of business development and project management, both at Validapro Biosciences, Laval, Quebec, Canada, 450.668.1144, 

It is important to understand critical aspects of the CMO's capabilities. Only by auditing certain key areas can the sponsor be assured of the quality of the materials produced.

Considerations When Outsourcing the Production of Clinical Material

As the 2008 election campaign moves toward its final stage, healthcare reform remains a high-profile issue. Both Democrat and Republican candidates have rolled out reform plans to expand access to healthcare through broader government programs plus tax breaks and incentives to help individuals obtain coverage. All these programs are costly and will require additional efforts to cut spending, prompting proposals for making the nation's healthcare system more efficient. 

The candidates envision large savings from adopting electronic medical records, encouraging prevention, and better coordinating care for patients with chronic conditions. These proposals, however, are likely to boost spending in the short run, prompting White House hopefuls to propose curbs on outlays for prescription drugs. A related strategy is to identify which therapies have sufficiently high value to warrant high price tags.  

Sen. Barack Obama promises to provide Americans with access to the "exact same drugs in Europe and Canada" that pharmaceutical companies sell at half the price as in the US. He plans to save up to $30 billion by repealing the ban on government price negotiations for drugs purchased by Medicare and wants to increase the use of generic drugs by government health programs.  

The comparative research approach may be preferable to price controls in the guise of government negotiations for the Medicare drug benefit, coverage denials, and limits on access to new technologies.

BioPharm International-06-01-2008

Avoiding the Pain of Out-of-Specification Results

Modeling of Biopharmaceutical Processes—Part 1: Microbial and Mammalian Unit Operations

The Power of Hypothesis

How to Evaluate the Cost Impact of Using Disposables in Biomanufacturing

Considerations When Outsourcing the Production of Clinical Material

Healthcare Reform Threatens Biotech Coverage and Access

Biotech Survives Difficult Opening to 2008

A Rational Approach for Setting and Maintaining Specifications for Biological and Biotechnology–Derived Products—Part 1

Funding the FDA, Finally

Validation Failure is Not an Option. Should it be?