What to consider when faced with new scientific evidence

Imagine that a bank has been robbed in your city, and you have been asked to come in and solve the crime.  As a first step, you go to the crime scene and start gathering as much evidence as you can find. 

After your visit to the crime scene, you have two primary pieces of evidence:

• fingerprints found on the main door to the bank and on the door to the vault
• blurry image of a person wearing a hood and dark sunglasses from the security cameras

After consulting some experts, you identify the person who left the fingerprints.  However, you discover that he is an employee of the same bank, explaining why his fingerprints would be on the vault. 

Video experts have determined that the person in the security footage is likely a male of average height, but the images are too blurry to determine much else.  Neither of these pieces of evidence is strong enough to tell you anything useful.

Upon returning to the crime scene, you find a note that the robber gave the teller to demand the money.  You immediately take the note to your forensic experts, and you discover that it has the same fingerprints found in your initial investigation. 

You still have some reasonable doubt—since those fingerprints belong to an employee, it’s not crazy to think that he was in the bank after the robbery and touched the note.  You provide the note to a handwriting expert, and you find that the handwriting is a match for the same bank employee whose fingerprints had been found.

The story is becoming much clearer now.  You talk to the management of the bank regarding your suspicions, and you learn that this employee was fired from his position the day before the robbery, and he had made a threatening remark when leaving for the last time. 

None of these pieces of evidence are strong enough to tell you who robbed the bank with certainty.  Once they are all considered together, though, it’s hard to conclude anything other than that the former bank employee was the robber. 

What makes scientific evidence strong?

Scientists are detectives in their own right; they gather individual bits and pieces of evidence in an attempt to uncover some larger truth.  Just as in our example with the bank robbery, not all pieces of scientific evidence are equally strong.  The way a scientific study is designed can have a big influence on how strong or weak the evidence is.

• Randomized control trials
  In clinical research, the gold standard is a randomized controlled trial (RCT).  In an RCT, the researchers will determine what type of person qualifies for the study and what outcomes are worth measuring.  They then recruit people into the study who fit their criteria, and those recruited into the study are randomly assigned to one of at least two groups. 
  
  Some participants will receive an investigational treatment, while others will receive a control.  By randomly assigning participants to their groups, researchers hope to reduce the chance of an outside, unknown factor influencing a participant’s outcomes.  This results in a stronger conclusion about the efficacy of the treatment and gives RCTs their status as the strongest level of evidence a single study can provide. 

• Systematic reviews
  As we saw with the robbery example, though, many pieces of evidence taken together can form an even stronger picture.  The same is true in science—the strongest form of evidence is a systematic review.  In a systematic review, researchers read dozens (or maybe even hundreds) of study results on a topic.  They also establish some rules to help them determine what types of studies to include or exclude. 
  
  After they have read the studies and decided which ones are relevant for inclusion, they then synthesize all the results into a set of common conclusions.  Often, this includes a statistical analysis of the results from all studies together, called a meta-analysis.  Systematic reviews are an important way to see how consistent a particular finding is, and to evaluate the strength of all the available evidence as a full package.

What to consider when faced with new findings

Hearing care professionals (HCPs) are faced with scientific results from hearing instrument manufacturers, the research community, mainstream media, and elsewhere.  When hearing about research on a brand-new topic area, it is important to consider the strength of those findings before making any strong conclusions. 

The same is true when new findings are at odds with what was previously seen on a given topic.  Taking the time to read such papers and to ask some basic questions can help you judge how seriously to take new findings. 

Questions like, “What other factors could explain these results, and how did the researchers take those into account?” or “Were the outcome measures the researchers used appropriate for the conclusion they try to draw?” are both a good start to thinking more critically about scientific reports.

HCPs may not be detectives in the traditional sense, but they do have to consume and form conclusions about a large amount of evidence. By keeping two key points in mind—1) not all evidence is equally strong, and 2) many weak pieces of evidence can form a strong story if they say the same thing—you can become a better consumer of research and be less prone to believe some of the more sensational headlines inspired by new findings.

You have likely heard about a new study just published in the field of hearing loss and cognition, called the ACHIEVE study.  You can learn more about this RCT by going to the study’s website (www.achievestudy.org).