Meeting Report – Reproducible Research and Registered Reports.
Laboratory Studies Section Meeting Report, Friends Meeting House, Euston Road 20th March 2017
Invited speaker
The meeting opened with a comprehensive presentation by guest speaker Professor Chris Chambers from Cardiff University on the registered reports project. This is a new publishing initiative supported by the journal Cortex and promoted by the Centre for Open Science. This class of publication gives researchers the opportunity to decide hypotheses, experimental procedures and main analyses before data collection. Part of the review process takes place before experiments are conducted and passing this stage of review virtually guarantees publication. This framework incentivises well-designed hypothesis-based research studies. It also helps to counter both publication bias and researcher bias and it directly supports reproducible research. This initiative is being adopted by many journals including Nature, Human Behaviour and a number of funders are piloting a registered reports funding model.
Working group updates
There were two presentations from the current working groups within the Laboratory Studies Section and RIPOSTE consortium. The RIPOSTE framework was developed to promote early and regular discussion of key design considerations so that statisticians could be more involved in laboratory based designs. The working groups aim to provide some simple worked examples that demonstrate the impact a more integrated approach might have on a study design. Both teams are drafting R based tutorials so that researchers with little or no prior experience in R can quickly learn how to analyse their data more appropriately.
Unit of Analysis
Dawn Teare presented an update on behalf of Alice Sitch and Nick Parsons on ‘Unit of Analysis and Repeated Measurements’. It is common practice to make replicate measurements in experiments but such data is frequently not analysed optimally. More importantly, knowledge of the components of variation in the various stages of measurement is not commonly factored in to the experimental design. The team have been working on an R-based tutorial with a simple example of analysing repeated measurements using a linear mixed model and demonstrate that estimating the components of variance could be used in the optimal design of a future similar experiment.
Left-censored biomarker data
Liz Hensor presented progress from work carried out jointly with Bethany Shinkins and Mike Messenger on how to analyse data where measurements are left-censored and present a ‘limit of detection’ issue. Their aim was to illustrate how the flexible NADA package in R could be used to analyse such data using a relevant biomedical example: C-reactive protein (CRP). They were able to compare methods to handle the left-censoring present in the routine measurement with paired measurements from a high-sensitivity CRP assay. Overall they have found that for summary data the robust regression on order statistics (rROS) method gives acceptable results for small skewed datasets whereas for group comparisons the generalised Wilcoxon score test performs best.
Laboratory Studies Section Members’ Presentations
Mike Messenger presented a new framework for the Quality Assessment of Measurement Procedures (QAMPs) using in vitro diagnostics. Measurement validity is a major concern for NIHR Diagnostic Evidence Co-operatives. He argued that a framework such as QAMPs is needed because the current standard QUADAS-2 does not consider sufficient measurement factors to fully assess validity. QAMPs has been developed and refined in consultation with laboratory professionals. He demonstrated its utility for one example but it needs further work before it can be used to support systematic review of measurement factors.
The final two presentations of the day were examples of the challenges of working on trials where laboratory-based measurements are required. Although a biomarker may be specified in a protocol the variation of techniques that can be used to make the measurements can be challenging.
Williams Sones gave an example of how helpful it can be for statisticians to have a good understanding of the biology underlying some laboratory-based techniques when it comes to handling these measurements in trials. His example involved the qPCR technique to quantify levels of mRNA. As this method relies on exponential amplification then a log transform of the data is entirely appropriate. He acknowledged it can be very difficult to understand the full pipeline of how the measurements are generated as lab scientists are used to manipulating data prior to forwarding the results to the statisticians.
Francesca Fiorentino spoke of her involvement in an early phase trial where for each specific outcome measure there could be multiple measurements due to the variety of techniques available. She recommended including the lab scientist in the evaluation of the best measurement for each specific outcome while all were still blind to allocation and before any comparative analysis.
__________________________________________________________________________________
We thank the participants and speakers for an interesting and varied day of presentations and discussions, and thank the Friends Meeting House for hosting us. Future meetings will be announced in due course. If anyone is interested in joining either of the working groups, or helping to run the Laboratory Studies Section, please contact Dawn Teare.