RQA Ireland Regional Forum Event: "GxP Allsorts Update"

 The Irish Regional Forum is taking place on the 19th March 2015 at ICON, South County Business Park, Leopardstown, Co. Dublin, Ireland. 10:00-15:45, Registration 10:00am.

The Forum's theme was titled 'GxP Allsorts' and is touching on some interesting areas.
The Impact of Clinical Trials Regulation - from the non commercial perspective (Stephen Liggett, Research Governance Manager QUB) to the Distributor perspective (Paul O'Connor, Global VP Quality, Almac) to the Sponsor perspective (Mark Rutter Associate Director of Regulatory affairs, Abbvie via Webex).
- GLP update - Marie O’Mahony Quality manager at INAB.
- Medical Devices updates - Colette McIntyre, GLP/GCP Specialist at Heartsine / Chair RQA Medical Devices Committee.
- GMP update - Ann McGee McGee Pharma International.

The Ireland Regional Forum is FREE for all RQA members and those people interested in RQA membership.

Evaluation and Selection

In our last post we looked at requirements definition. It was discussed how you need to identify and document requirements and quantify those requirements to create your essential criteria. It was also mentioned the importance of scaling down your POTENTIAL supplier list based upon your essential criteria. Today, we are doing to discuss Evaluation and Selection. (Please bear in mind that the following is geared towards an enterprise computerised system solution - but you can scale these steps as guidance for your needs). 

Evaluation:

At this stage, you probably have quite a long list of potential suppliers whose marketing will seek to convince you that they are what you need. Your role is now to remove the marketing and see exactly what the potential supplier can and cannot provide you. Do not be sold by the glitz and glamour of sales presentation. An internet based presentation will allow a much more focused look at the software and its potential fit for the organisation’s needs.

• Perform a GxP Assessment against the requirement. Identify, whether the solution provider will need be checked, if their quality approach is robust enough to provide a quality solution, (and perhaps supplement any potential validation needs).
• Research software solution against each requirement.
• Rank each solution against each requirement.
• If necessary, conduct interviews with a selected list of suppliers using a predefined questionnaire, based on the requirements. Keep the conversation focused on a list of prepared questions. This will provide enough information to begin comparing supplier against supplier and will probably allow for the immediate elimination of a few suppliers.
• Reduce the list of suppliers down as a response.
• Request a web based demonstration of the top ranking solutions and grade the outcome of the demonstration. Demonstrations can last from between, 1 to 3 hours, hence the need to reduce the list of potential suppliers.
• Develop a demonstration script and provide this to the selected suppliers.
• Reduce the list of suppliers down as a result of analysing the responses.
• Create and send a request for proposal to the top 3 suppliers.
• Reduce the list of supplier down as a response.
• Create and send a request for proposal of the top 3 suppliers.
• Request an onsite demonstration using defined use cases that YOU have prepared. This is, without a doubt, one of the most significant steps in any software selection process. Unfortunately, it is typically one of the most poorly managed steps in this process. Ensure to capture the teams reaction to each demonstration immediately after, when the information is still fresh in their minds.

It is very important to maintain control of the onsite demonstration, so that the users witness the actual value of the solution and not the value the supplier wants to portray. If the supplier representative spends the majority of the demonstration reviewing presentations and little time in the actual solution, then the supplier may be attempting to prompt the users to perceive a higher level of quality that is not necessarily in the solution. A supplier who invests little in the onsite demonstration may offer little investment in after-sales support. This is a useful marker for the selection process.

Selection:

• Evaluate the RFP responses. For example, is the response well thought out and written? Was all the known requirements quoted for? Is the quote within budget? Do they have a QMS?
• Evaluate the on-site demonstrations. For example, was the demonstration script followed? Did the system look user friendly? Were all questions answered confidently and satisfactorily?
• Reduce the short list to the top 2 suppliers.
• If necessary, perform and audit of the potential suppliers premises, QMS, SDLC, security and so on. Ensure that quality is built into the solution and whether the supplier’s documentation can be leveraged if the solution impacts GxP. Ensure that you are happy with their backup process, business continuity and particularly their support and maintenance approach.
• Select the supplier of choice and notify the suppliers of your decision.
• Negotiate the contact.

Contact us today to hear more about our robust COTS approach.


Next time we'll take a look at Implementation.

Requirements definition Supplier listing

A quick recap: 

We have looked at the pros and cons of COTS implementations. E.g.” Reduce development costs due to the mass production of reusable modules” or “What is the likelihood of customisation to meet all of the organisation needs?”  We discussed that COTS implementation is still a sizeable undertaking that many organisations take for granted.  We have a robust approach to ensure that the solution bought is the best solution to meet all the needs of the organisation: cost, value add, regulatory and use. 

What we want to look at today is;

• Problem definition: Research & Planning 
• Requirements definition 
• Solution & supplier evaluation 
• Solution and supplier selection  
• Solution implementation  
• Solution Review and sign-off 

Requirements definition and identify suppliers 

Identify the requirements 

This is a simple concept: Identify and document your requirements. But many business analysts will tell you that requirements are difficult to pin down. Requirements arise from;

• Previous analysis 
• Processes, records, training courses, manuals 
• Data storage and business continuity approaches 
• Consumer comments, other business areas comments 
• Regulatory needs. 
• Codes of practices, warning letters, industry trends 
• Technology trends 

• Existing defects 
• Interviews of functional areas 

Quantify the requirements based in risk 

As ever risk is central to every decision we make. No matter how you quantify risk (and no doubt you have a procedure for identifying and managing risks), establish clear definitions of what constitutes different levels of risk to your organisation (including ‘unacceptable risk’ as a benchmark). Prioritise the identified record keeping requirements according to this scale. An example can be: 

Step	Action
1	Assess conflicting requirements in terms of (e.g.):   the regulatory, safety or financial costs of meeting or not meeting the requirement  the business value of the information contained in the system  and so on…....
2	Identify and describe the consequences of not meeting each requirement, and determine the likelihood of each consequence occurring.
3	Weigh up the costs and benefits of meeting each requirement and the consequences of not meeting the requirements.
4	Discuss the issues with operational staff and management to reach an informed decision. Document this analysis and the decisions reached for future reference.

The results of this risk assessment, and risks linked to particular functions can help determine what requirements should be met. The various tables, matrices and other techniques used in risk analysis will help you to;

• identify specific areas of risk in your organisation 
• quantify and prioritise those risks in terms of the cost to, or impact on, your organisation (i.e. regulatory, safety, operational, financial and technical feasibility factors), and 
• make, justify and document recommendations for meeting requirements 

How to record a requirement 

There a plenty of requirements templates available, feel free to contact us for our version.  

Consider what data to compile in relation to each requirement;

• the name of the owner  
• the unique identifier 
• requirement description (objective language, UML) 
• any regulatory references 
• references to any linked requirements or dependent requirements 
• priority of the requirement  
• approval 
• requirement type (e.g. GxP, HR, Finance, Database, Business Continuity) 

A word of caution: Do not overstate the needs of the organisation. A supplier may need to respond to hundreds of requirements lists and it is best to have a list that details the needed requirements. A supplier will see the real needs of the organisation and better respond as a result. 

Upon completion, ensure to review the completed requirements with the functional areas and the project team. We find that a formal review process, where different reviewers assume different roles/perspectives provides a great static analysis of the requirements and helps reduce future faults. 

Contact us to find out more of formal inspections and reviews. 

Suppliers List 

The list of suppliers is ever changing. Research in this area is very important in order to be able to focus on a manageable list. Sources of information include: internet search engines and directories, consultants, industry associations, magazines, trade shows and vendors. Iteratively review and update this list to get the best list of candidates. 

Once a list has been established, seek to reduce this list based upon essential criteria. Once this is done, seek remote demonstrations from suppliers and reduce the list again to create a target list of suppliers for requesting proposals from. 

Draft RFP 

This step is often skipped, but in our experience it is the best way to communicate the full project requirements. There are plenty of RFP templates available. Feel free to contact us for our version.  

Elements of a RFP include;

• Organisation overview 
• Scope of project 
• Relationship with other systems 
• Schedule and deadline for response 
• Proposal instructions 
• Various services (e.g. training, QA/Test artefacts) 

Next time, we’ll examine the supplier evaluation portion of the COTS approach. 

COTS Series 2: Problem Definition & Risk Assessment Part

We talked previously about how Commercial Off The Shelf Solutions (COTS) use offers many benefits:

·         Reduce development costs due to the mass production of reusable modules

·         Reduced development time and effort as the coding has been done

·         Improved quality as the COTS has been used and tested by other users

But COTS come with a level of uncertainty:

·         What features exist and what features will be added/removed in the future?

·         Are there latent (hidden) defects within the COTS solution?

·         What is the capability of the COTS solution to integrate with other solutions?

·         What is the likelihood of customisation to meet all of the organisation needs?

·         What are the supplier’s stability, capability, support, quality and trustworthiness?

Furthermore, it is difficult to assess the strengths and weaknesses of COTS solutions with certainty as:

·         The black box nature of COTS solutions forces the consumer to rely on the vendors claims.

·         The description of the COTS solution can be incomplete.

·         The evaluation of COTS solutions is often rushed; poorly thought through and incomplete; with limited resources and under false assumptions that are not relevant to the organisation’s environment.

COTS is really is a risky business.

A poor implementation strategy and a poor understanding of the requirements leads to cost overruns, schedule delays, poor validated state and even system abandonment. Yet we get the question “if the coding is already done, surely buying a COTS system is quick and easy?” Capers Jones (Applied software measurement, 3^rd ed, 2008) has found that the percentage of software development life-cycle effort involved for coding ranges from: 16% for military systems; 20% for enterprise systems and 35% for software written by users who are not employed in the software industry. EmpowermentQE’s own metrics show that coding accounts for between 15% and 21% of the software development life-cycle effort for complex, engineered enterprise systems. Our point being that there is still a lot of effort required for a successful COTS deployment.

A 6 point COTS selection approach that although is geared towards enterprise based systems and can be tailored to any size of system – the scope of approach is based on risk.

TIP: Based upon the above start with the hypothesis that all COTS solutions and suppliers are not complete for the regulatory industry rather than the having mind-set that the COTS will be relatively quick and easy to deploy.  Starting with a pessimistic approach will increase thoroughness, capture and tie down more risks, and help increase success.

Today we’re going to introduce briefly look at the initial phase o four approach.

The approach is applicable to any solution and caters for the regulatory industry needs:

·         Provide a high level of confidence that the COTS solution will meet the business, technical and regulatory needs.

·         Leverage the knowledge, experience and documentation of the supplier.

·         Judge the quality and reliability of the COTS suppliers.

·         Gain documented evidence that COTS will always perform as intended – differentiate the need between supplier audit and supplier assessment.

·         Gain documented evidence of the data integrity of the COTS solution.

The above lists some of the steps to consider when researching and planning a COTS implementation.

This phase presents the opportunity to fully examine the business/manufacturing/operations/quality domain broadly and identify the goals, how the work is performed, who performs the work and why the work is performed. Examine the risks, stakeholders and so forth for setting the foundations for the requirements definition.  Review of key indicators or project drivers leading to the initiation of this project. Compare those project drivers to the overall business objectives held by the organisation to ensure they are aligned. Next develop a project plan and business case, which allow for executive support and approval for the selection process. All are necessary to ensure the project is headed down the right path at the project’s onset.

This stage will help identify:

·         How the business unit fits within the organisation.

·         The Business needs to consider (IT, regulatory, training, HR,…).

·         The business units strengths and weaknesses – can improvements be introduced?

·         Barriers to a COTS solution.

·         Regulatory and legal requirements.

This helps to ascertain the project context – how the project will be structured, success factors, staff, technology, constraints and initial risks.

Document the activities performed above to provide trace-ability of the initial research. For example, capture:

·         A list of all sources used: resource name and contact/access details

·         Copy of documentation sources

·         Details of any interviews, meetings or other consultative form

·         Details of all relevant stakeholders

·         Project Initiation Documentation

·         Project Plan   … and so forth

A top down and bottom up approach will help form a better understanding of the organisation needs from a micro and macro level, such as:  establish current problems that a COTS solution will solve; establish current assumptions and constraints; timelines; security needs; and  ability to meet regulatory compliance requirements

The breadth and depth of our activities is dependent of risk. A risk based approach to COTS is fundamental to ensure its success. Consideration is given to the existing systems performance.  After which a strategic systems and project plan can be designed and approved.  As per any software implementation approach, risk must be managed (identified, managed and evaluated) throughout the approach and tailored where required.

The above steps are applicable to major and complex solutions and the depth and breadth of their use is wholly dependent on the risk, complexity and regulatory needs of a proposed COTS solution.

Next time we will explore requirements definition.