New Website to Help Scientists Design Better Experiments
Problems with the quality of animal experiments
More than 50 million laboratory animals are estimated to be used in experiments each year1. According to an editorial in Nature Genetics “Animal experimentation in scientific research is a good thing: important, increasing and often irreplaceable.”, but It goes on to say that “Careful experimental design and reporting are at least as important as attention to welfare in ensuring that the knowledge we gain justifies using live animals as experimental tools.” It suggests that many experiments are under powered, but rather than simply increasing sample size scientists should look closely at the various sources of variation. Controlling these and improving experimental designs to include variables such as gender will increase both power and the external validity of the results.
Twenty years have passed since the publication of the first paper to suggest that there is room for improvement in the design and statistical analysis of experiments using laboratory animals2. Many more papers have been published since then, but progress has been slow. A 2009 survey of 271 papers involving rats, mice or non-human primates3 found basic principles of experimental design were widely ignored. For example, 87% failed to mention randomisation, 86% did not report “blinding” in studies where it would normally have been appropriate, none of the studies justified the sample size that they used, 26% failed to state the sex of the animals, 24% reported neither age nor weight of the animals and 4% didn’t even state the number of animals which they had used. The omission of many experimental details was clearly a problem. The ARRIVE guidelines4, based on the CONSORT guidelines were subsequently developed to help eliminate such omissions. These have been accepted by many journals.
Poor experimental design may mean that the experiments can’t be repeated. In a commentary article in Nature the authors5 claimed to have attempted to repeat fifty-three “landmark” studies in cancer research, sometimes with the help of the authors. They “…acknowledged from the outset that some of the data might not hold up, because papers were deliberately selected that described something completely new, such as fresh approaches to targeting cancers or alterna¬tive clinical uses for existing therapeutics.”, but only six (11%) could be repeated. They considered this a “shocking” result. In a following commentary article the author6 claimed that “research is riddled with systematic errors” which could erode public trust if it is not put right. Interestingly, neither of these papers attributed these problems primarily to poor experimental design. They both suggest that the fault lies in a culture which strongly favours positive results, extreme competition so that investigators are more likely to make selective use of their data to obtain positive results (which amounts to scientific fraud) and excessive emphasis on publication in high impact journals by funding organisations.
Better training of research scientists
Whatever the cause of these problems, the majority of scientists want to design their experiments well, but in many cases their formal training in experimental design and statistics is rudimentary or non-existent. Those few scientists who are already proficient in statistics are usually too busy doing their own research to teach others, and in any case there is no incentive for them to do so. Nor are there many statisticians, with the required biological background, willing and able to take on this role. The scale of the problem is immense as there are many thousands of post-graduate students starting their research in the life sciences each year. It is logistically impossible to provide even short introductory courses for so many people scattered in so many universities world-wide. But given the right tools scientists should be able to teach themselves the basic principles, which are not too difficult to understand. The World Wide Web provides a flexible and economical way of helping them.
The new website www.3Rs-reduction.co.uk
This is a free, non- commercial site which can be used on desk-top, lap-top or tablet computers. It starts with the ethical background based on the “3Rs” (Replacement, Refinement and Reduction) introduced by Russell and Burch in 19597. Pages on research strategy, the” experimental unit”, the characteristics of a “good” experiment, avoiding bias, power and sample size follow. There are further pages on controlling variability and strains of mice and rats. The page on experimental designs explains how randomisation can be done using an EXCEL spread-sheet for both completely randomised and randomised block designs. There are also pages on factorial designs, regression and correlation, and statistical analysis, although the emphasis is on design not analysis. Scientists who have access to one of the large commercial packages such as SPSS or MINITAB are advised to use it, but alternatively they can down load R and use the R Commander front end to carry out most of their analyses. Raw data for example analyses is given. There are pages on presentation of results, literature and guidelines. A pdf of the ARRIVE and Gold Standard Publication Checklist can be down-loaded. Each section has an associated “Test yourself” page with true/false questions. Anyone who works through the site can down load a page where they can self-certify that they have done so. The hope is that Ph.D. supervisors and ethical review committees will get all new Ph.D. students and scientists new to animal research to work through the site before they start work with animals.