Study protocol - you’re either with me or against me!
When it comes to assessing the impact of how well patients (and arguably centres) in a randomised clinical trial follow the protocol, the standard method for some decades has been via the identification of the ‘Per Protocol’ set. This ICH-driven subset of ITT patients essentially requires the review of blinded patient data at the end of the trial. Here, a decision is made by a multi-disciplinary data review team as to whether or not each patient has followed the protocol-defined procedures to an extent that does not impact their (primary) efficacy endpoint data. Efficacy analyses conducted on the ITT population are then repeated on this Per Protocol subset as part of the standard study reporting.
The ‘rules’ that this data review group use in deciding whether or not a patient should remain in the in the Per Protocol set are ideally (but not always) pre-specified, either in the protocol, but more usually in the analysis plan. However, there can be considerable vagueness in the specification, and hence flexibility in the application of these rules. The process can easily get a little wayward, to the extent that sometimes the tail can wag the dog. I’ve seen cases where the rules have been tweaked on the fly during these data review meetings such that not too many patients are removed from the Per Protocol set. There is always an understandable, albeit often underlying, sponsor fear of incurring low statistical power.
With these issues in mind, might we be better off avoiding this all-or-nothing, in-out, binary approach to how well each patient follows the protocol? Rather, might we instead grade each patient on a continuous 0 to 1 scale, where 0 = did not follow protocol at all (e.g. violated entry criteria, received not study treated, had no post-baseline visits, etc.) up to 1 = followed protocol to the letter (took all study meds, completed all visits, etc.) This score could then be summarised in of itself and used, for example, as a weight in re-weighted analyses of the efficacy outcomes. This type of approach would certainly require more thought about the relevant and relative impact of each protocol violation type, but that wouldn’t necessarily be a bad thing. It might go some way to avoid the temptation to tweak the rules.
Taking a wider view of the use of Per Protocol set, this analysis population has slightly fallen out of industry favour over recent years. Whilst it’s still of considerable importance in early phase studies and non-inferiority trials, even in those trials its relative importance versus ITT-based analyses has waned. The introduction of estimands, and the resulting industry emphasis on treatment policy-based analyses appears to further erode the Per Protocol set’s standing. Perhaps it’s time for a more nuance approach, where we should ask ‘how well did the patient follow the protocol?’, rather than ‘did they follow the protocol?’.
