The plant IT team chose an ultra-high availability architecture based on fault-tolerant servers. The servers' continuous availability
features and remote monitoring capabilities supported an off-the-shelf Windows server applications the company used with none
of the extensive customizations clustering would have required. The new system has met or exceeded the computer industry's
defining measure of 99.999 percent availability in real-world customer installations. The emphasis on device driver reliability,
in particular, counteracted an oft-overlooked root cause of operating system instability with third-party device drivers.
Therefore fault-tolerant computing and infrastructure design are important parts of the solution, but they are only one-half
of the picture. Infrastructure management is the backbone of continuous availability — and perhaps the highest barrier to
MAINTAINING A CONTINUOUSLY AVAILABLE INFRASTRUCTURE
Although most pharma companies can probably afford to build one, the ongoing cost of maintaining a highly available infrastructure
is beyond most plant-level IT staff and budgets, even at larger companies. The constant diligence needed to guard against
failures extends to even routine tasks performed hundreds of times per day at most companies.
For example, consider the plugging of a laptop into the network. That's not a risky act at most companies, but for a pharma
manufacturer trying to maintain 99.999 percent uptime, it could be a disaster. Just as a natural ecosystem is affected when
a new species is introduced, so is a continuously available ecosystem. Plugging an unauthorized laptop into the network may
introduce anomalies into an infrastructure — such as a virus. While losing a set of computers to a virus is disruptive in
any business, a virus in a pharma manufacturing facility can crash record collection or process control data collection and
force the company to scrap a batch that could be worth millions of dollars. Even seemingly innocuous tasks such as upgrading
applications, applying patches, and routine maintenance can result in slowdowns and crashes.
Corporate data centers in many industries, including major life science corporations, leverage outside service providers to
maintain computing infrastructures and systems compliance based on negotiated service level agreements. Yet, this practice
has not yet made it to the pharma manufacturing environment. Recent research has determined the root cause of this is trust.
Pharma manufacturing IT people know that such service providers do not understand the details of their environment and their
operations, as well as the regulatory mandates they face. These same IT people understand, however, that high availability
and QMS are synonymous. They know what they must do to meet their QMS needs, but are bandwidth and budget constrained.
So, compliance requires high availability, and high availability requires a robust QMS for IT infrastructure. But resources
are limited at the plant — time and money for robustness seem out of reach. And the core competency of a pharma manufacturer
is not IT. What is the solution?
A new breed of service providers is emerging that join the industry knowledge, manufacturing IT experience, and high availability
deployment and maintenance skills together. They supplement the manufacturer's IT staff and regulatory processes to provide
a robust QMS capability to pharma manufacturers. From design (or analysis of existing infrastructures, people, and processes)
to deployment, verification, and on-going management of the infrastructure, such services can be obtained within budgetary
constraints. Services such as patch evaluation and deployment, IQ of components, and operational and performance screening
(as a preliminary check prior to OQ and PQ) can be outsourced to relieve the burden of valuable internal compliance resources
and reduce time-to-delivery and cost of IT capabilities to the production managers. And, most importantly, high availability
can be achieved through services that can remotely monitor, detect, and help prevent slow-downs or outages while providing
capabilities to maintain proper configuration control and release management of upgrades and end-of-life scenarios.
WORRY ABOUT THE RIGHT THINGS
Vendors who define availability in terms of a single network element such as clustered servers, raid storage, or backup/recovery
software are only addressing part of the high availability solution. True continuous availability is not only beyond the capability
of clusters; it requires looking at the compute, network, and storage environment as a whole. Continuous availability does
not begin and end with any single component but treats the entire flow of information across these components as a potential
problem, including the software applications. And while it is vitally important to leverage fault-tolerant components in an
infrastructure based on appropriate risk evaluation, the monitoring of the entire environment to prevent failures and to facilitate
the lifecycle management of the components and stored information is also a crucial component of the infrastructure.