Today we’re bringing endpoint quality into the 21st century.
Here’s how it works:
- Raters train interactively with AI patients, practising both interviewing and scoring, with immediate feedback.
- Expert quality review with AI, with immediate turnaround, making it simple and affordable to review all study assessments in real time.
- Raters retrain during the study if their quality drops.
- Sponsors can see quality in real time across raters and sites.
Everything is handled automatically.
CNS trials rely on each participant being reliably rated at baseline and across the trial, to detect a treatment effect.
Unwatched ratings nudged up to meet the inclusion criteria – due to the financial incentives of randomising participants – can inflate baseline scores, leading to an inflated placebo response.
Poor inter-rater reliability and rater drift over the course of the trial add measurement noise – especially when a participant’s rater changes during the trial. Noise costs power: the less reliable the ratings, the more participants a trial needs to detect the same effect.
Finding, training, and retaining rater staff is a pain point for many research sites.
Most rater training today relies on a presentation about the instrument, then watching a small number of pre-recorded interviews with actors pretending to be patients, which the raters score, and if their scores are within allowed error tolerances, they get approved to rate in the study. It often involves in-person meetings where the raters are trained together; sometimes as part of this, the raters discuss how they got to their scores; sometimes there’s an actor that the raters can do a practice interview with.
Despite this being the most common training approach, research on rater training suggests it’s not the most effective: practice administering and scoring the scale matters more than professional background, and interactive training is more effective than presentations and pre-recorded interviews. Today training also typically happens once at the start of the study, sometimes with refresher training, or a remediation call for raters flagged by central review.
Central review – where recordings of assessments are sent for review by a third party – is expensive, limiting the fraction of study assessments that are practical to review; it’s slow, taking days or weeks before the rater gets feedback on an assessment; it’s complicated to implement; and lastly – the reviewers themselves are not perfectly accurate or reliable. Although central review is routine in pivotal trials, almost nothing has been published on how accurate reviewers are – and reviewers can be wrong, reliably.
Coagen’s approach to continuous rater training and quality is based on the following:
- Rater training should be interactive. Raters should practise the interview itself, not only the scoring.
- Qualification based on performance. Raters should qualify when they meet a specified quality bar, and train until they meet that bar. The flip side of this is that people who’ve not traditionally been able to become raters can now become raters, where the protocol allows.
- Training should be personalised. Different raters will differ in what and how much training they need. They should not be limited to watching five training videos if they need more training; conversely, if they have past demonstrated performance on a given scale, it should be fast for them to qualify.
- Feedback should be instant. And it should be helpful. Feedback – both on training and in-study performance – should be available immediately, not delayed by days or weeks.
- Continually retrain raters during the study as needed on a per-rater basis.
- Make it extremely easy to record assessments compliantly, to facilitate quality review whether in person or remotely. No separate devices. No added friction. Privacy and security built in.
- Review every assessment. Make it economically and operationally feasible to review every assessment in the trial. Make expert review much cheaper. Make expert review much simpler to implement and integrate into raters’ workflows.
- Help sponsors make sense of quality review across raters and sites – even across trials in a portfolio.
How this ties into Coagen’s mission:
- Improving trial quality, so drugs that work can meet their primary endpoint.
- Bringing down trial costs by making expert review affordable.
- Enabling many more people, from more diverse backgrounds, to become clinical trial raters.
Coagen’s continuous rater training and expert quality review is available today. Request access.