The best patient outcomes come from thorough, evidence-based evaluation. This process, known as evidence evaluation in healthcare, ensures clinical and operational decisions are grounded in credible data. But how do we evaluate that evidence and how do we draw conclusions for defensible decisions? There are three main applications of evidence evaluation in healthcare.
For insurers, health technology assessments support evidence-based decision-making in the development of coverage policy, utilization management, and in defending appeals. In health systems, health technology assessments are used to better inform decisions regarding the acquisition or adoption of new and emerging technologies, as well as to reduce clinical variation in practice. For manufacturers, health technology assessments can help prove market viability by demonstrating safety and effectiveness. Together, these use cases demonstrate how evidence evaluation in healthcare supports transparent, defensible policy and purchasing decisions.
I recently had the opportunity to sit down with a group of Medical Directors from a variety of community health plans to discuss the approach to conducting rigorous health technology assessment for defensible decision-making. Key takeaways from this conversation included the critical importance of understanding HOW evidence evaluations and health technology assessments are conducted, the importance for quality and accreditation, and some targeted conversations about tricky situations. Health technology assessment remains central to value-based care.
As organizations look to accreditation by regulatory bodies, it is essential to understand the process of evidence evaluation in healthcare to guarantee that decisions are transparent and defensible.
Health technology assessment forms a crucial link in the research-to-policy continuum. This continuum starts with clinical research or bench science in attempts to gain regulatory approval. As part of this process, pivotal trials are published and disseminated. Organizations like Hayes use that peer-reviewed, published evidence to produce an evidence evaluation. That systematic approach to evaluating published data is central to the broader evidence evaluation process in healthcare.
The health technology assessment, in turn, aids in evidence-based decision-making. As access increases to the health technology, there is inevitably more clinical research and diffusion of that research into the clinical and scientific community. This triggers a reassessment of the health technology, which may lead to a refinement in policy decisions as a result. Ongoing evidence evaluation in healthcare ensures these reassessments remain data-driven and defendable. And thus the cycle continues…
So how does one begin to understand evidence evaluation in healthcare and health technology assessment?
In completing background research, the goal is to determine the epidemiology, clinical presentation, risk factors, clinical alternatives, and standard treatments for the indication of interest. In addition, it is important to understand the technical details or the mechanism of action of the health technology. Often the rationale for technology will help to define the framework parameters for your evidence assessment. Why did this technology come to market?
What gaps does it fill that current standard care does not fulfill? Sources that can provide valuable information for the background research include clinical and methodological subject matter expertise, existing health technology assessments, systematic or narrative review articles, professional or clinical practice guidelines, and regulatory documents. These foundational insights form the basis of reliable evidence evaluation in healthcare.
As you start to research the technology and indication of interest, you will begin to formulate clinically meaningful key questions. These questions form the foundation of your evidence assessment and guide a consistent evidence evaluation process in healthcare. In general, you will want to know:
To identify these questions, a framework must be developed and applied. This is commonly referred to as the PICOTS framework in which P = patients; I = intervention; C = comparators; T = timing (follow-up); S = setting. A well-defined PICOTS framework ensures consistency in evidence evaluation in healthcare.
For example, let’s say we are interested in evaluating electric nerve stimulation for the treatment of chronic low back pain. A key question that we might want to answer may include:
Does transcutaneous electric nerve stimulation (INTERVENTION) improve pain, quality of life, and function (OUTCOMES) in adults with chronic low back pain (PATIENTS)?
As the PICOTS are defined, additional parameters may need to be explicitly stated and defined. For instance, how are you defining chronic low back pain? Are you only interested in long-term results or are short term results sufficient (TIMING)?
The setting (S) as part of the framework will come into play when there are interventions that could be used in different facilities: outpatient versus inpatient, at-home use versus outpatient use, skilled nursing facility versus inpatient care.
This process of defining the framework and ultimately the eligibility criteria of the studies that will be used in the evaluation is often an iterative process as subject matter experts provide insight and as the search for literature is executed.
Outcomes are notoriously difficult to assess. We are faced with how to treat patient-reported (subjective) versus investigator-reported (objective) outcomes; the validity of the outcome measure; whether there is a defined and validated minimal clinically important difference for an outcome measure; and whether it is a surrogate measure of improvement. In choosing outcomes relevant for an evidence evaluation, you must consider all of these factors. This is when it is imperative to have both clinical and methodological subject matter expertise to weigh in on what is most relevant to not only evaluate the performance of a technology but the impact that the technology has on patients in everyday use. Such outcome selection is a crucial component of evidence evaluation in healthcare.
Once a framework and the key questions are formulated, you will want to conduct a systematic search using a sensitive and specific search strategy of multiple databases, such as Embase and PubMed’s MEDLINE. It is also best practice to perform a manual search of the reference lists of recent systematic reviews and health technology assessments to ensure that no evidence is overlooked. A thorough search process enhances the validity of evidence evaluation in healthcare.
Next, a systematic selection of the evidence occurs, aiming to include published peer-reviewed evidence that will aid in answering the key questions. This is where the PICOTS framework will come into play. For instance, you will not want to include a study evaluating transcutaneous electric nerve stimulation in patients with chronic neck pain, if the goal is to evaluate the technology in patients with chronic low back pain.
Once you have selected the studies that meet the criteria set out in your key questions, you will begin the process of critically appraising and interpreting the evidence. Critical appraisal relies on identifying the risk of bias, otherwise known as the internal validity of a study. When we critically appraise a study, we are really trying to decide:
In other words, based on the methods used to design the study and the methods used to assess the outcomes what inherent biases were introduced. In addition, did the investigators take liberties or not fully report their findings such that we question the validity of the outcomes presented in the publication.
Some of the factors that you may want to consider include whether the study was prospective, was the length of follow-up appropriate and were all patients evaluated at the same time points, was there unexplained attrition and how was this accounted for in the analysis, was there blinding of outcome assessors, patients, or administrators, and was the patient population representative of the patient population in the real world. Based on this appraisal, you will then assign a quality rating for each eligible study.
Among the important factors used to evaluate the outcomes of the study, you must determine whether statistical significance reflects clinical significance. Ultimately, we are trying to determine if treatment effects that are statistically significant provide meaningful value to the patient, and if so would we find this improvement if repeated in a similar population and under the same circumstances.
Once you have analyzed, synthesized, and appraised individual studies, you will then grade the evidence by outcome. You will want to consider the confidence you have in the effect for that specific outcome across studies.
In other words, for pain, did all of the studies identify an improvement in pain at the same endpoint and to the same degree? If not, what are the factors that impact our confidence that an improvement in pain will occur? Items to consider include the consistency of the results, the individual study quality of the studies, the directness of the results, and how precise the results are – meaning is there a wide range or do all the studies demonstrate the same amount of improvement in pain?
After you have quality ratings for the key outcomes, you will then evaluate the overall evidence. To do this, start by ranking your key outcomes by clinical importance, and then evaluate the evidence across those critical outcomes taking into consideration the same factors you did when evaluating the evidence for each outcome. This synthesis step is essential to comprehensive evidence evaluation in healthcare.
The balance of benefits and harms is key both with respect to assigning the quality rating but also in developing the evidentiary conclusion. Also, take into consideration the balance of benefits and harms compared with clinical alternatives. In our hypothetical example, we would want to know whether the benefits of applying transcutaneous electric stimulation were greater than an adverse effect that might have been experienced and then in comparison to the use of NSAIDs were the benefits greater and do those benefits outweigh the adverse effects experienced.
Ultimately, all evidence evaluations rely on your confidence that the evidence that exists is reproducible and without substantial bias introduced to it. A body of evidence that is given a quality rating of high reflects confidence that future research is very unlikely to change the result, specifically the direction or the estimate of effect. On the other hand, a low-quality rating suggests that confidence is lacking that the current evidence reflects the true effect. Recognizing these quality boundaries is part of responsible evidence evaluation in healthcare.
Additional evidence may or may not change both the estimate and the direction of effect. The evidence is too variable, inconsistent, or lacking in other domains for us to draw firm conclusions. Finally, a very low-quality evidence base is an evidence base in which there is either no evidence or not enough evidence to permit conclusions about the safety or effectiveness.
Health technology assessments generally incorporate additional information. Think about an all-too-common situation in which there is potential but unproven benefit. What do you do with it? Consider reviewing practice guidelines from professional organizations. Is there a national coverage determination from CMS? Are there local coverage determinations? Are representative payers in the industry covering the technology? What is the potential clinical or operational impact with adoption of this new technology? Where do you think the evidence might go in the future? These contextual considerations complete the cycle of evidence evaluation in healthcare.
Once a health technology assessment has been conducted, it is important to monitor the state of the evidence. To facilitate an annual review expeditiously, review newly published evidence, regulatory changes, changes in payer policies, and changes in professional guidelines to determine whether there will be an impact on your evidentiary conclusions. This continuous monitoring represents an ongoing commitment to evidence evaluation in healthcare.
symplr Evidence Analysis powered by Hayes, does all of this work for you. We have a team of scientists, clinicians, and experts dedicated to evaluating the evidence so that you can focus on making the right decisions for your organization and your members. symplr Evidence Analysis provides comprehensive health technology assessments that enable defensible, evidence-based decisions you can trust have been thoroughly researched and reviewed. Hayes’ proven methodology ensures every evaluation meets the highest standards of rigor and transparency.
