Quality & Productivity: Can You Have Both?

Alison Messom MICR, Andrew Borissow

Session chaired by: Gareth Hayes HonFICR

Reporter: Wendy Tomlinson MICR

Keywords

Lean Sigma, Productivity, Quality, Six Sigma

---

 

This session was split into two sections: Alison considered the practical application of quality and productivity whilst working under increasing pressures to squeeze both out of clinical research activities; Andrew then provided a practical demonstration of how to evaluate, and subsequently improve, a simple process using Lean Sigma methodology. The overall question poised was “Is quality and increased productivity a mythical holy grail?”

Combining quality & productivity metrics

Alison’s part of the session began with the definitions of Quality and Productivity.

• Quality is the measure of excellence, a state of being free from defects without significant variations, whilst having a strict and consistent adherence to measurable and verifiable standards. Quality is intended to make sure we have a uniformity of outputs to satisfy the customer’s requirements.
• Productivity is a relative measure of the efficiency of a person, machine or system in converting the inputs into useful outputs. It is calculated by dividing the average output over a period of time by the total costs incurred or resources (such as capital energy or personnel utilised) consumed in that period. Productivity is a critical determinant of cost efficiency.

Increased quality often equates to increased costs (eg, through increased time spent on tasks and additional quality control measures). Therefore, by default productivity can drop for the same reasons. In turn, it is perceived that increased productivity often leads to decreased quality as compromises are made and quality checks drop in order to maintain output against tight deadlines and/or cost constraints.

Multi-dimensional metrics

In order to improve anything, you need to have a baseline from which to measure. Alison explained that in terms of business it is important to consider multi-dimensional metrics:

• Time, Units, Scope
• Turnaround, Tracking, Performance
• Forecast

Measuring one- or two-dimensional metrics doesn’t always give an accurate picture. Solely considering time, units or scope will yield a metric such as number of case report form (CRF) pages monitored per month or how many visits have been completed in a month. Combining two dimensions would allow the metrics to be narrowed down, so scope and unit together could yield the number of CRF pages collected per visit, ie, turnaround metrics or performance metrics against plan.

Far more useful is to combine all three to give you forecasting metrics, ie, how many visits have been completed to date combined with how much work is remaining or the query rate at a site. This information is truly useful in enabling a project manager to plan ahead.

CRA productivity

Alison chose to focus on CRA productivity. The key question is what should we measure to accurately assess a CRA’s productivity? For example, should it be the number of site visits per month, or should we be considering other metrics such as a reduction in the rates of Data Not Found (DNF) queries per page, ie, reducing the amount of rework and associated costs? Another interesting metric is to consider the rate of patient recruitment at a site: often CRAs disown this as being wholly outside their responsibility. However, they should remember that they do influence this by performing preliminary site visits (PSVs) thereby assessing the potential of the site and pool of potential patients, by providing training and motivation at the beginning and then throughout the recruitment period. Therefore, surely they should accept accountability.

Alison told two interesting anecdotes relating to the above examples and how individuals can strive to meet such metrics whilst losing sight of the aims of increased productivity and quality. Early on in her career, she took over from a CRA with the highest rate of monitoring visits in the team, but also the highest query rate. It turned out that this CRA was not actually monitoring; they thought that monitoring visits were simply a catch-up with the site staff and then a period of page-pulling, so no monitoring or quality checks were performed! At the other extreme, Alison managed a CRA who was judged on the rate of DNFs per CRF page collected. Under these circumstances, the CRA would defer page collection until they were absolutely sure no queries remained; the outcome was too many hours being spent on this activity!

Next, Alison demonstrated a tracking report looking across many metrics, such as the number of pages at site, the number of pages with data management and how many hours have been spent at site by the monitor to collect these pages. She demonstrated how this kind of report can help with tracking activities against the original plan and help to identify high/low levels of activity against average expected levels. For example, tracking the number of pages monitored per visit could demonstrate that the average team member is monitoring approximately 55 pages per visit. Any individual monitoring only 30 pages per visit would be highlighted on such a report and additional support could be given to identify whether it was a training issue or a site issue.

Using these types of metrics enables Intelligence-Led Monitoring. This allows flexibility in monitoring schedules in order to meet the varying monitoring loads, queries or training requirements across the investigational sites. This also allows for focus on sites where there is lots of data to collect or identified training or safety issues. The conventional 6-8 weekly monitoring schedule becomes irrelevant: the frequency of visits becomes more specific to the site’s needs and workload. The project manager can direct the monitors appropriately, thereby meeting the deliverables of the project whilst maintaining the quality.

Discussion

Following this summary, Gareth Hayes posed a question: ‘Who’s doing the monitoring of these metrics? Is it the project manager or the QA group?’ Alison’s response was to confirm that her project management team use the information, whilst emphasising the need for tracking of useful data, which drives productivity, rather than completing forms for the sake of completing them. The metrics that she uses are automatically pulled from information that the CRAs enter into their clinical trial management system (CTMS);collection of this information is not a duplication of effort. The metrics can highlight values that fall out of predetermined criteria.

Eric Boelema from CDR also contributed at this point, confirming the need for quality being brought in throughout activity rather than completing unbudgeted re-work at the end. Quality should be applied consistently rather than fixing it at the end. Alison completely agreed and stated that each functional group is responsible for the QC of the work within their own department: no-one should hand off work to someone else if it is deficient or defective, as it is not someone else’s responsibility to tidy up.

Process improvement with Lean Sigma

Andrew’ Borrisow’s session looked at a specific process improvement technique, Lean Sigma, and at the types of wastage and variation within processes, and discussed how to choose projects within an organisation where process improvement can work.
Practical demonstration

Andrew began with a simple practical demonstration involving three members of the audience (‘operators’) and a pack of playing cards. It simulated a process thatwas not running to the desired quality levels. The operators were instructed to drop playing cards as per a defined process (ie, the SOP) onto a piece of paper on the floor in front of them. The exercise was analogous to a process such as a CRF page being collected from site:

• If the playing card lands on the paper there are no DCFs
• If the playing card does not touch the paper there are DCFs for that page
• The desired outcome is to maximise the number of playing cards landing on the paper  (ie, CRF pages without DCFs)

The first round was intended to evaluate the current process. The volunteers dropped their cards, with a very small number successfully landing on the paper sheets, ie, a very low number of CRFs being collected from site without data queries on them. Alison used the number of successful card landings to calculate the Rolled Throughput Yield (RTY), the probability that a single unit can pass through a series of process steps free of defects. Therefore, this inadequate process has been measured: a requirement in order to evaluate the anticipated improvement that this process should yield.

A fourth volunteer joined the group, taking 2 cards each existing operator. This second round was demonstrated that adding more resource didn’t make it any more successful, just quicker!

The next step was to consider variables: what could be changed? The size of the paper? The height of the drop? The orientation of the cards? The air conditioning in the room?

Now to change one part of the process. A key point in Lean Sigma is to evaluate one or two changes at a time, ie NOT to change everything at once, which would make identification of successful improvement(s) very difficult.

The third round redefined the SOP by changing the orientation of the card when being dropped and this demonstrated a huge change in the RTY success rate, ie, from 1% to 46%. This provided a measurable outcome to evaluate the improvement. In a more complex process, the next step would be to look at the other variables and make subsequent changes evaluating the outcomes at each step.

To summarise:

• A process of concern was identified
• Measurements of the process were taken
• The measurements were analysed and variations identified
• The process was improved
• The process was re-measured to ensure that the improvements made yielded the expected results

Overview of Lean Sigma

Following this highly successful demonstration, Andrew provided more information about Lean Sigma. This is based on the Lean methodology, which focuses on eliminating waste so making a process faster and Six Sigma, which focuses on the reduction of variation thus producing consistent quality. By combining the impacts of Lean and Six Sigma, the outcome should be an efficient, fast process that produces continual and consistent quality.

Identification and elimination of waste in a process is dependent on considering both the process and the environment where it is in use. Examples of wastage common to many environments include waiting for document hand-off: therefore, the more hand-offs in a process, the greater the potential for waiting times. Intellect is another good example: are the right people doing the right tasks, or is a project manager completing a task that should be performed by a CTA (or vice versa)?

Andrew reiterated that Lean Sigma considers variation within a process. Lack of training can introduce variation, and long-standing habits can perpetuate it; people will work on a process in a certain way because they have always done it that way, and it will often be hard for them to implement changes even when their benefits have been demonstrated. Therefore, Lean Sigma also builds in control measures into any new process so it can be monitored following implementation.

Processes should always be developed to meet the expectations of the customer. Whilst it’s vital to provide a good, high quality process it’s also important to consider if the customer want a ‘Rolls Royce’ process or a ‘Mini’ one, as they won’t pay for a ‘Rolls Royce’ if they want a ‘Mini’!

Andrew explained that Lean Sigma projects are intended to be small, with evaluation and improvements completed within 6-8 months. Therefore project selection is vital. There are many different ways to select them, and feedback from customers and employees can be useful. All ideas are put into an ‘idea hopper’ to prioritise the appropriate projects, then ones which are expected to have a significant business benefit can be scoped out and continued.

Case studies

Andrew then provided some examples of organisations applying Lean Sigma. The US Army reported in 2007 that they hoped to hit the $2 billion savings milestone by the end of that year with implementation having started early/mid 2006. He also touched briefly on his own Green Belt Lean Sigma Case Study, to reduce the turnaround time of paper-based data collection forms (DCFs). With his Lean Sigma project team, he mapped out the original process, which included a huge number of steps. Analysis showed a huge variability in the turnaround of DCFs from site. 47% of DCFs were returned within 30 days of being sent to site; therefore 53% are taking more than 30 days. The aim of the project was to get 100% returned within 30 days: about a 200% increase! The team have mapped out a much shorter process, with 25 hand-offs of the DCFs being reduced to 10. This Case Study is still in its pilot stage, with ongoing evaluation of the process over the next 3 years, but is anticipated to have a huge beneficial outcome.

At the end of Andrew’s presentation Gareth reflected on the practical demonstration and the importance of getting people involved in the methodology and trained up on Lean Sigma; Andrew confirmed that everyone within the organisation needs to be aware of the methodology in order to ensure engagement of the right people in future project teams.

Overall, this was an excellent, informative session, which delivered exactly what was required.

---

 

Alison Messom MICR is Executive Vice President Clinical Operations with Averion International.
Andrew Borissow is Process and Systems Manager, Project Methodology Office with MDS Pharma Services.
Wendy Tomlinson MICR is Director, Clinical Operations with MDS Pharma Services.