Making Sense of Medical Research

In the past 12 months, worldwide there have been 261 articles published in the peer-reviewed medical literature about chronic fatigue syndrome (CFS). Few of them get the level of media attention as did last week’s publication in the journal Science linking CFS and xenotropic murine retrovirus (XMRV). But each new discovery is subject to various interpretations and should be understood in the context of the scientific method.

Here is some information excerpted from a resource provided by the National Institute on Aging titled, “Understanding Risk: What Do Those Headlines Really Mean?” It is not specific to CFS or the XMRV study, but it may help those who have contacted us about how to make sense of this and other CFS studies (past, present and future).. It’s good information to consider each time a new study about CFS is released and as reports about progress in better understanding the XMRV link emerge.

“Understanding Risk: What Do Those Headlines Really Mean?”

Every day in the newspaper or on television we see stories about new medical findings. Perhaps we hear that a certain drug causes a 300% or three-fold increase in strokes. That’s a large increase—it sounds scary. But, if you know that in every 10,000 people not taking the drug, there are two strokes, then a three-fold increase really only means six more strokes. Maybe that’s not quite so frightening. It’s also confusing that sometimes stories seem to report opposite results—a new vaccine prevents a devastating infection, or it doesn’t. How are we to make sense of such stories? How do we know what to believe?

This fact sheet provides some background to help you understand these news reports. It might also help you judge which results are really important and which are simply interesting but not a reason to change how you take care of yourself.

When you learn about a new medical finding, ask yourself:

1. Was it a study in the laboratory, in animals, or in people? The results of research in people are more likely to be meaningful for you.
2. Does the study include enough people like you? You should check to see if the people in the study were the same age, sex, education level, income group, and ethnic background as yourself and had the same health concerns.
3. Was it a randomized controlled clinical trial involving thousands of people? They are the most expensive to do, but they also give scientists the most reliable results.
4. Where was the research done? Scientists at a medical school or large hospital, for example, might be better equipped to conduct complex experiments or have more experience with the topic. Many large clinical trials involve several institutions, but the results may be reported by one coordinating group.
5. Are the results presented in an easy-to-understand way? They should use absolute risk, relative risk, or some other easy-to-understand number.
6. If a new treatment was being tested, were there side effects? Sometimes the side effects are almost as serious as the disease. Or, they could mean that the drug could worsen a different health problem.
7. Who paid for the research? Do those providing support stand to gain financially from positive or negative results? Sometimes the federal government or a large foundation contributes funding towards research costs. This means they looked at the plans for the project and decided it was worthy of funding, but they will not make money as a result. If a drug is being tested, the study might be partly or fully paid for by the company that will make and sell the drug.
8. Who is reporting the results? Is the newspaper, magazine, or radio or television station a reliable source of medical news? Some large publications and broadcast stations have special science reporters on staff who are trained to interpret medical findings. You might want to talk to your health care provider to help you judge how correct the reports are.

Remember that progress in medical research takes many years. The results of one study need to be duplicated by other scientists at different locations before they are accepted as general medical practice. Every step along the research path provides a clue to the final answer—and probably sparks some new questions also.

Here is some additional information from wikipedia about the importance of independent confirmation of scientific results (http://en.wikipedia.org/wiki/Scientific_method and http://en.wikipedia.org/wiki/Reproducibility):

Confirmation

Science is a social enterprise, and scientific work tends to be accepted by the community when it has been confirmed. Crucially, experimental and theoretical results must be reproduced by others within the science community. Researchers have given their lives for this vision; Georg Wilhelm Richmann was killed by ball lightning (1753) when attempting to replicate the 1752 kite-flying experiment of Benjamin Franklin.[46]

To protect against bad science and fraudulent data, government research granting agencies like NSF and science journals like Nature and Science have a policy that researchers must archive their data and methods so other researchers can access it, test the data and methods and build on the research that has gone before. Scientific data archiving can be done at a number of national archives in the U.S. or in the World Data Center.

Reproducibility is one of the main principles of the scientific method, and refers to the ability of a test or experiment to be accurately reproduced, or replicated, by someone else working independently.

The results of an experiment performed by a particular researcher or group of researchers are generally evaluated by other independent researchers who repeat the same experiment themselves, based on the original experimental description. Then they see if their experiment gives similar results to those reported by the original group. The result values are said to be commensurate if they are obtained (in distinct experimental trials) according to the same reproducible experimental description and procedure.

Reproducibility is different from repeatability, which measures the success rate in successive experiments, possibly conducted by the same experimenters. Reproducibility relates to the agreement of test results with different operators, test apparatus, and laboratory locations.

Additional information on XMRV