Comparative Effectiveness Research

Overview

This module provides an overview of comparative effectiveness research (CER), delineates the different types of stakeholders, who may be important to engage in CER, and includes tips, tools, and resources for engaging stakeholders in CER.

This overview module reviews the what, who, why, where, and how of stakeholder engagement in CER. These elements are critical to the design, implementation, and follow-up of effective stakeholder engagement.

WHAT is comparative effectiveness research?

Comparative effectiveness research (CER) was defined by the US Institute of Medicine in 2009 as "the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care. The purpose of CER is to assist consumers, clinicians, purchasers, and policymakers to make informed decisions that will improve health care at both the individual and population levels "(IOM, Initial National Priorities for Comparative Effectiveness Research, 2009).

Until the last quarter of the 20th century, the majority of research evidence available concerned a single test or treatment (drug, device, procedure, other intervention), usually only tested on one group of patients, and did not permit comparisons. Interest in combining and comparing evidence from multiple sources led to an interest in CER.

CER expanded and accelerated in the US during the 21st century. The timeline below outlines recent initiatives both internationally and in the US.

What comprises CER?

Several domains of CER were described in reports of the Federal Coordinating Council and IOM:

Evidence prioritization: The Council and IOM posit that, to work well, CER needs to address a range of questions that matter to stakeholders such as patients, communities, providers, purchasers, and other decision-makers in health care. The Council report prioritized CER addressing "areas of uncertainty within clinical and public health communities regarding decisions and variability in practice."
Evidence generation: New comparative research is part of CER. The Council report called for the application of CER to the needs of diverse populations and subgroups often not prioritized in funded research including women, children, elders, minorities, and people with disabilities and/or multiple chronic conditions. Several of these population groups have historically not participated in research, and their engagement in CER is one focus of stakeholder engagement.
Evidence integration: Some CER research methods also seek to synthesize prior research. Examples include meta-analysis and cost-effectiveness analysis.
Evidence dissemination and application: Disseminating CER findings to "patients, clinicians, and other stakeholders" is critical (Agency for Healthcare Research and Quality). CER should "assist consumers, clinicians, purchasers, and policymakers in making informed decisions that will improve health care at both the individual and population levels."

In addition, the Patient-Centered Outcomes Research Institute (PCORI) and the AHRQ are tasked with including the public in the research process and informing the public of their products.

What is stakeholder engagement?

The 2009 Institute of Medicine definition of CER highlights:

the need for CER research that is relevant to "real-world settings"
the critical role of disseminating CER findings to "patients, clinicians, and other stakeholders"
the importance of including decision makers in CER at both the individual and population levels.

Implicit in this definition is the engagement of a broad range of stakeholder groups. Stakeholders have been defined by AHRQ as "persons or groups who have a vested interest in the clinical decision and the evidence that supports that decision." Stakeholders are engaged in research, not as research participants, but in the research process itself.

The AHRQ identifies seven roles for stakeholders in the CER process.
Identify real-world questions for health and health care decisions Provide input to develop and prioritize research gaps Ensure key questions accurately describe decisional dilemmas Provide input on methods, clinical relevance, and appropriate outcomes Provide input to improve the quality and responsiveness of research findings Help craft meaningful messages of CER findings Share information with others; take the lead in using evidence to inform health care practices.

The AHRQ identifies seven roles for stakeholders in the CER process.

Identify real-world questions for health and health care decisions

Provide input to develop and prioritize research gaps

Ensure key questions accurately describe decisional dilemmas

Provide input on methods, clinical relevance, and appropriate outcomes

Provide input to improve the quality and responsiveness of research findings

Help craft meaningful messages of CER findings

Share information with others; take the lead in using evidence to inform health care practices.

WHO plays a role in CER?

Stakeholders are any individuals or groups who may have a critical role to play in the prioritization, generation, integration, interpretation, dissemination, or application of CER.

Stakeholders can be an individual, groups of people, organizations, a community, and companies. For example, stakeholders might include patients, parents, an activist group, administrators at an academic medical center, insurance companies, health plans, a member of Congress, or pharmaceutical representatives.

We can group potential stakeholders into seven broad categories, known as the 7Ps of stakeholders, similar to the framework proposed by AHRQ.

Stakeholder	Who they are
Patients and the Public	Current and potential consumers of patient-centered health care and population-focused public health, their caregivers, families and patient and consumer advocacy organizations.
Providers	Individuals (e.g., nurses, physicians, mental health counselors, pharmacists, and other providers of care and support services) and organizations (e.g. hospitals, clinics, community health centers, community-based organizations, pharmacies, EMS agencies, skilled nursing facilities, schools) that provide care to patients and populations
Purchasers	Employers, the self-insured, government and other entities responsible for underwriting the costs of health care
Payers	Insurers, Medicare and Medicaid, state insurance exchanges, individuals with deductibles, and others responsible for reimbursement for interventions and episodes of care
Policymakers	Including the White House, HHS, Congress, states, professional associations, intermediaries, and other policymaking entities
Product Makers	Drug and device manufacturers
Principal Investigators	Other researchers and their funders

This table provides examples of research questions relevant for different stakeholders.

Stakeholder	Example of Project
Patients and the Public	Research questions arising from patient concerns, aimed at improving quality of care
Providers	Research for new treatment modalities to be used by clinicians
Purchasers	Evaluating the cost effectiveness of different treatments for a disorder
Payers	Examining the cost effectiveness of different service bundling programs
Policymakers	Evaluating the efficacy of state- or federal-level programs
Product Makers	Testing off-label use of an existing intervention
Principal Investigators	Exploring initiatives that do not fit into existing funding models

WHY engage stakeholders in CER?

Engaging stakeholders in comparative effectiveness research (CER) has become significantly more important over the last decade, echoing efforts to include stakeholders in the research process in general as well as in clinical care decision-making. Click here to use the Tufts CTSI Tool: Involving Stakeholders in Your Research Project.

What are the goals for stakeholder engagement in CER?

Goals for increased stakeholder engagement in CER are overlapping but include:

Better science by:
Improving the efficiency, quality, and relevance of research questions, design, methods, measures, and analyses Getting the patient's and caregivers' voices heard Working with populations that are often under-represented in research.

Enhanced ability to apply findings in the real world by:
Improving the credibility of the research project Paying attention to issues raised by stakeholders throughout the process of conducting CER Building a cohort of stakeholders excited about and committed to applying the results of CER.

Prioritized funding for specific health care issues through:
Stakeholders voicing the need for research around identified issues. Increased transparency and public accountability in individual research projects and in local and national healthcare policymaking is required by funding agencies. The reality is that stakeholder engagement has not been widely embraced. Indeed, a number of barriers have been raised including: Hierarchical academic culture that reinforces the researcher as the authority Concerns about bias if stakeholders are included Increased costs and time associated with stakeholder inclusion, which may conflict with pressures to reduce costs and years of funding from federal and foundation funders Limited stakeholder interest and/or time to participate in research.

Prioritized funding for specific health care issues through:

Stakeholders voicing the need for research around identified issues. Increased transparency and public accountability in individual research projects and in local and national healthcare policymaking is required by funding agencies. The reality is that stakeholder engagement has not been widely embraced. Indeed, a number of barriers have been raised including:
- Hierarchical academic culture that reinforces the researcher as the authority
- Concerns about bias if stakeholders are included
- Increased costs and time associated with stakeholder inclusion, which may conflict with pressures to reduce costs and years of funding from federal and foundation funders
- Limited stakeholder interest and/or time to participate in research.

Why is there a focus on stakeholder engagement now?

Stakeholder engagement in research is not new. The public health and biomedical fields have been working to engage stakeholders in projects such as:

Public health needs assessments have used quantitative methods (e.g., surveys) or qualitative methods (e.g., focus groups) to identify community members' priorities for future research, policy, and programs.
Non-profit organizations focused on specific, often rare, diseases have collaborated with researchers to fund research and recruit participants.
The National Institutes of Health and the Centers for Disease Control have sponsored academic-community research partnerships in cities and their surrounding metropolitan areas as well as in rural areas to address the disproportionate burden of chronic diseases and public health challenges in underserved populations. Often these partnerships have prioritized not only conducting research, but community-capacity building, activism, and community-based participatory research or action research approaches.
A number of foundations including the Robert Wood Johnson Foundation recommend stakeholder engagement as a key part of program planning, implementation, and evaluation.
Implementation science actively engages community settings in research on the adoption of evidence-based practices into real-world settings.

Much of the focus of these efforts has been on consumers (patients and/or families, the public) or clinicians in community or practice settings. Stakeholder engagement in CER seeks to engage a wider spectrum of stakeholder groups, including administrators, policymakers, payers, and others.

WHERE does your research lie on the Tufts CTSI - CER Spectrum Model?

Are you unsure where your research question falls on the Tufts CTSI CER Spectrum? Click here to find out.

Evidence prioritization:

Both the IOM definitions posit that, to work well, CER needs to address a broad range of questions that matter to a range of stakeholders such as patients, communities, providers, purchasers, and other decision-makers in health care. The Council report particularly prioritized CER addressing "areas of uncertainty within clinical and public health communities regarding decisions and variability in practice."

Evidence generation:

CER includes the conduct of new comparative research. The Council report called for the application of CER to the needs of diverse populations and subgroups often not prioritized in funded research including women, children, elderly, minorities, disabled, multiple chronic conditions. Several of these population groups have historically not participated in the research process, and their engagement in CER is one focus of stakeholder engagement.

Evidence Synthesis:

Evidence synthesis refers to the systematic review of past research to provide an assessment of a specific research question, determining what is known, what is not known, what methods have been used and where new research is needed. It uses explicit scientific methods that are designed to include information from different sources and study designs, reduce bias, and inform clinical care and future research.

Evidence integration:

Some CER research methods also seek to synthesize previous research conducted. Examples include meta-analyses and cost-effectiveness analyses.

Evidence dissemination and application:

The critical role of dissemination of CER findings to "patients, clinicians, and other stakeholders" was also recognized. Both definitions highlight that CER should "assist consumers, clinicians, purchasers, and policymakers in making informed decisions that will improve health care at both the individual and population levels."

Feedback and Assessment:

For both researchers and stakeholders, review and feedback are essential for learning from past experiences and improving future projects.

HOW to successfully engage stakeholders in a project?

In this section, we walk through six broad steps in a CER research project, providing tips, tools, and resources for engaging stakeholders. The steps need not be followed in order; you may find that some steps inform previous steps. We define these steps as:

Determining why to include stakeholder engagement in my project

Planning stakeholder activities

Identifying and inviting stakeholders to participate

Preparing for the stakeholder engagement activity

Engaging stakeholders in the activity

Ending stakeholder engagement.

This step-by-step process is framed within the context of CER research, but these steps also reflect good practice with stakeholders engaged in any type of process. Institutions and/or agencies may choose to engage stakeholders more globally in planning processes with respect to stakeholder engagement in CER. We encourage interested researchers to also visit the resources section to learn more. You can also click here to download the Involving Stakeholders in Your Resarch Project tool.

Step #1: Determine why to include stakeholder engagement in your project.

There can be many reasons to engage stakeholders in your project, such as:

The purpose of stakeholder engagement in CER and PCOR are relevance, transparency and application. Stakeholder engagement can improve the relevance of research, increase its transparency, and accelerate its application in practice.
There are additional, practical reasons for including stakeholders in your research, including that it is part of the mission or goals of the funding agency or is called for in a Request for Applications (RFA).

Step #2: Plan the stakeholder activity for my project.

Before selecting a specific type of stakeholder activity for your project, consider your goals for including stakeholders. The UK's National Health Service Research Support Unit identifies several roles that consumers (and stakeholders) can play in research, ranging from having a stakeholder advisory board that meets periodically to providing consultation to a study, to shared decision-making regarding the entire research process as in community-based participatory research (CBPR) or action research.

Before you begin, ask yourself whether stakeholders should be involved for just one activity, or multiple times throughout the research project? The answer to this question will help determine which type of stakeholder activity you may want to consider.

In addition to determining whether stakeholders will have a consultative or shared decision-making role, it is also necessary to consider ethics and available resources (e.g.: human, financial, time, and technology). When considering possible activities, ask yourself:

Is the scope manageable and feasible?
What resources are available?

Types of stakeholder activities

Many mechanisms for engaging stakeholders exist. Depending on the aim, methods, and desired outcomes of the research you are conducting, different forms of engagement may be more appropriate than others. As part the Engaging Stakeholders in Effective Health Care Program, AHRQ listed strengths and limitations for the different mechanisms. Additionally, RWJ's Practical guide for engaging stakeholders in developing evaluation questions includes engagement mechanisms that work best based on different criteria (e.g., limited budget, dispersed geographies) and provides worksheets to help determine the best strategy.

Examples of stakeholder engagement mechanisms include:
Clinical recruitment Deliberation methods Delphi technique Expert consultant panels In-person meetings (focus groups or one-on-one)	Mixed methods (e.g., questionnaires followed by in-person meetings) Questionnaires (online or paper) Telephone interviews, conference calls Webinars, email Workshops

Step #3: Identify and invite stakeholders to participate

If, through these exercises and your own experience, you realize that recruiting, building, and maintaining partnerships with stakeholders will be too challenging, you may need to consider hiring someone to complete these activities.

Some questions to determine which stakeholders to include are:

What do I hope to get out of this stakeholder individual or group's participation?
How do I want them involved in decision-making?
How do I want them involved in my research question?

Note that stakeholders' desired outcomes (reduced costs, improved health care quality, etc.) may vary.

Once you have a created a list of potential stakeholders and determined how to engage them in your project, make sure you have the right stakeholder representatives at the table.

Stakeholder engagement takes time and effort, particularly if you are engaging stakeholders as collaborators or co-investigators.

If you will be engaging stakeholders in a group process, keep in mind the optimal size of functioning groups. If your goal is to simply inform stakeholders, larger groups can work; however, most organizational behavior literature recommends limiting groups to 6-12 people. Categories of stakeholders are not mutually exclusive and individuals who wear different "hats" may be able to function as representatives for more than one of the 7Ps.
To maximize efficient use of your time, identify people with good communication skills who can clearly articulate their perspectives and listen to other perspectives. In some fields, there are "professional stakeholders" who function as patient or community representatives across a number of research projects. The advantage of these individuals is that they understand the research process and often have the communication skills necessary to be successful as stakeholders. At times, however, these individuals may become so "professionalized" that they are less effective at representing their constituency.

Below are questions based on the CDC guide to engaging stakeholders in evaluation, to help determine which of the 7Ps should be included in the research project:
Who are the stakeholders we need to: Increase credibility of the research project? Implement the findings from the research project? Enable funding of future research projects? What to ask stakeholders: Who do you represent and why are you interested in this research project? What is important about this research project to you? What would you like this research project to accomplish? Considerations for selecting stakeholders: Is the research important to the stakeholder? Is the stakeholder knowledgeable about research context? Does the stakeholder have the capacity to engage?

Below are questions based on the CDC guide to engaging stakeholders in evaluation, to help determine which of the 7Ps should be included in the research project:

Who are the stakeholders we need to:
- Increase credibility of the research project?
- Implement the findings from the research project?
- Enable funding of future research projects?
What to ask stakeholders:
- Who do you represent and why are you interested in this research project?
- What is important about this research project to you?
- What would you like this research project to accomplish?
Considerations for selecting stakeholders:
- Is the research important to the stakeholder?
- Is the stakeholder knowledgeable about research context?
- Does the stakeholder have the capacity to engage?

Each stakeholder group has specific needs to consider:
Clinicians	Time constraints Flexible stakeholder activities (e.g., webinars) "Trying to get every major health care professional in the practice in the same room at the same time is quite challenging. So we've learned to try other methods where people can provide input by email, react to memos, and even have Webinars or conference calls, as ways to try to elicit that input." (O�Haire et al, 2011).
Consumers	"And the reality is that if a researcher will take the time, and that's extremely important, ask the questions, nurture a relationship, and patiently listen, that the community is very happy to engage and participate." (O�Haire et al, 2011). If engaging with patients with a specific disease, be sensitive about their current clinical state. There are inherent issues in balancing the power differential when consumers are included with other stakeholder groups. They may lack the scientific or political perspective that other stakeholders bring to the table; many consumers also "default" to a more passive role when with clinicians or other stakeholders. It's important to give the message that their voice is critical and to prepare them to participate with other stakeholders if you choose not to have separate meetings for different stakeholder groups. Critical to work around their schedules; parents of younger children may be negotiating jobs, after-school and weekend activities, and it may be difficult to include them. Others may have control over their time at work and be able to schedule times to participate during the day. Children, adolescents, the elderly, and the disenfranchised (e.g., prisoners, immigrants) all bring unique challenges to participation but can be essential voices to include.
Policymakers	Phone, one-on-one, small meetings Off-the-record discussions When contacting policymakers with busy schedules, adjust to their time and location for meetings and propose ways in which you may be helpful to them (e.g., networking through your institution, sharing information or knowledge).
Researchers	"One of the challenges is to merge the interest of academic researchers with those of the other stakeholder groups in a way that everybody gets something that they need out of it." (O�Haire et al, 2011).

Methods for identifying and selecting stakeholders

Recommendations by study team
Snowball sampling
Searching published literature
Searching databases/directories
Internet searches of individuals and organizations
Random sampling

See also the Robert Wood Johnson Foundation's Practical guide for engaging stakeholders in developing evaluation questions.

Recruiting stakeholders

After determining which stakeholders groups are relevant to the research project in Step #1, the stakeholders must be contacted to determine if they are willing and able to participate.

Considerations and strategies for recruiting stakeholders include:
Attend relevant stakeholder events to meet group leaders and start talking to people. Involve yourself in stakeholder causes and issues to show your interest and willingness to help (e.g., assist with another research project, share your connections and resources). Allow adequate time for recruitment. Use centralized databases or organizations to recruit stakeholders. For example, to recruit patients, you may consider: Consumers United for Evidence-based Healthcare (CUE) FDA Patient Representative Program AARP Decide methods of contact. Individual contact (email, phone, mail) Mass contact (advertisements, mass media) Decide methods for contacting non-responders (perhaps use a different method than initial contact). Monitor stakeholder representativeness and distribution. Consider conflicts of interest and biases. Determine financial support for participation (e.g., reimbursement, honoraria, staffing, none). Determine logistical support for participation (e.g., administrative, travel).

Considerations and strategies for recruiting stakeholders include:

Attend relevant stakeholder events to meet group leaders and start talking to people.
Involve yourself in stakeholder causes and issues to show your interest and willingness to help (e.g., assist with another research project, share your connections and resources).
Allow adequate time for recruitment.
Use centralized databases or organizations to recruit stakeholders. For example, to recruit patients, you may consider:
Decide methods of contact.
- Individual contact (email, phone, mail)
- Mass contact (advertisements, mass media)
Decide methods for contacting non-responders (perhaps use a different method than initial contact).
Monitor stakeholder representativeness and distribution.
Consider conflicts of interest and biases.
Determine financial support for participation (e.g., reimbursement, honoraria, staffing, none).
Determine logistical support for participation (e.g., administrative, travel).

Questions to discuss with stakeholders

Ethics	What ethical issues may this project bring up that may conflict with our respective values, colleagues, organizations, and affiliations? How will our own ethical standards be part of the research plan? How will the ethical standards of our respective organizations be part of the research plan? How will we deal with potential ethical conflicts?
Data Sharing	How will products be distributed, results (e.g., presentations, published papers) be shared, and messages be communicated to target audiences, including other researchers, funders, government agencies, and the community? How will our results be used to support new policy, programs, and research projects?
Collaboration	What will we gain through participating in a research project (e.g., connections to key stakeholders, improved patient outcomes, visibility, better funding opportunities, enhanced networking, etc.)? Are there any past issues that need to be discussed before engaging in research (e.g., problems with trust)? Are all research participants (i.e., research team members and stakeholders) clear on their roles and responsibilities? Are these roles clearly defined and easily accessible to all participants if necessary? What decision-making process will be used throughout this project? How will conflicts be resolved?

Evidence Generation

Evidence generation is the creation of information that can help patients, providers and other health decision-makers compare the benefits and harms of various interventions and strategies for preventing, diagnosing, treating, and monitoring health conditions in real-world settings.

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is evidence generation?

At the conclusion of this module, you will understand and be able to describe:
Key terms that are needed to guide your evidence prioritization activity, such as stakeholder, engagement, comparative effectiveness research (CER) and patient-centered outcomes research (PCOR) What evidence generation is.

WHO are the stakeholders in evidence generation activities?

You will be able to:
Identify relevant stakeholder groups for evidence generation, if necessary according to condition, intervention(s), or health care strategie(s). Explain the importance of achieving balance among competing viewpoints in your relevant stakeholder groups.

WHY engage stakeholders in evidence generation activities?

You will be able to describe the purposes of stakeholder engagement in evidence prioritization activities.

HOW to engage stakeholders in evidence generation activities?

You will be able to:
Describe and prioritize methods and mechanisms for engaging stakeholders for evidence generation. Recognize challenges and opportunities for engaging stakeholders in evidence generation.

WHAT is evidence generation?

There are three types of evidence generation:
Randomized Controlled Trials (RCTs), which can be clinical or practical trials Observational Studies, which can be prospective or retrospective Case Studies and Reports.

WHO plays a role in evidence generation?

Researchers may engage stakeholders before, during and after data collection and analysis when conducting a clinical trial, observational study or other evidence generation activity.

To improve the relevance, transparency and application of research questions, conduct and findings, stakeholder engagement is needed in every stage of a research project. Patient and consumer expertise is especially critical.

To learn more about involving stakeholders in your research project, use this tool.

WHY is stakeholder engagement important during evidence generation?

Stakeholders may be needed as advisors during pilot, implementation or evaluation stages of evidence generation. They may also be needed as subjects in preliminary or adjunctive research to guide a trial or study.

This may include an adjunctive qualitative focus group to interpret preliminary results, or it may involve conducting a survey to elicit preferences for the study of priority populations, interventions, comparators and outcomes.

WHERE in the Tufts CTSI CER Spectrum is evidence generation?

Early in the research process, stakeholders can assist with determining the research question you hope to answer. Guidance on engaging stakeholders to develop and prioritize research questions is offered in the previous module, Evidence Prioritization.

Following the development of an explicit research question, your research team and stakeholders will begin to explore how to generate new evidence that addresses the research question.

This is the primary purpose of this module on Evidence Generation - to guide you in the engagement of stakeholders as you conduct an evidence generation research activity.

Your research plan may also involve activities after the evidence has been generated. You may plan to synthesize the results of your research program with those of other programs, or to disseminate and implement your findings into practice settings. Stakeholder engagement in Evidence Synthesis and Dissemination and Implementation is described in Modules 3 and 5 of this series, respectively.

If you are unsure of where your research question falls on the Spectrum, click here.

HOW do you conduct stakeholder engagement for evidence generation?

Ten evidence generation steps are outlined below to use to engage stakeholders. Use the Evidence Generation Tool to follow detailed step-by-step instructions on how to conduct evidence generation.

Step #1: Agree on common definitions for your stakeholder engagement activity for terms such as:

Step #2: Agree on the purpose of stakeholder-engaged evidence generation for your research program or community.

Step #3: Agree on specific stakeholder engagement activities to support your study.

Step #4: Before you recruit stakeholders, identify appropriate stakeholder groups for your project by using the 7Ps

Step #5: Identify the objectives and related mode(s) of stakeholder engagement for your research activity. Stakeholders may be needed as advisors during pilot, implementation or evaluation stages of evidence generation, or as subjects in preliminary or adjunctive research to guide a trial or study. This may include an adjunctive qualitative focus group to interpret preliminary results, or it may involve conducting a survey to elicit preferences for the study of priority populations, interventions, comparators and outcomes.

Step #6: If your mode of engagement will involve the creation of a stakeholder panel or board, determine the appropriate size of the panel and determine appropriate balance across stakeholder categories on your priority list.

Step #7: Develop an evidence generation protocol.

Step #8: Identify and recruit individuals to fill the priority list of stakeholders for your evidence generation process.

Step #9: Evaluate and report on the process of engaging stakeholders in your evidence generation activity.

Step #10: Sustain relationships with your stakeholders.

Evidence Prioritization

Evidence prioritization is a systematic process that uses pre-specified criteria to prioritize or determine the most important evidence needed by patients, providers and other decision-makers in order to make informed choices about health interventions and strategies.

This module will help you to understand and embark upon evidence prioritization activities with engaged stakeholders.

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is evidence prioritization?

At the conclusion of this module, you will understand and be able to describe:
Key terms that are needed to guide your evidence prioritization activity, such as stakeholder, engagement, comparative effectiveness research (CER) and patient-centered outcomes research (PCOR). What an evidence prioritization activity is.

WHO are the stakeholders in evidence prioritization activities?

You will be able to:
Identify relevant stakeholder groups for your evidence prioritization activity according to condition, intervention(s), or health care strategie(s). Explain the importance of achieving balance among competing viewpoints in your relevant stakeholder groups

WHY engage stakeholders in evidence prioritization activities?

You will be able to describe the purposes of stakeholder engagement in evidence prioritization activities.

HOW to engage stakeholders in evidence prioritization activities?

You will be able to describe and prioritize methods and mechanisms for engaging stakeholders for evidence prioritization.

WHAT is evidence prioritization?

Evidence prioritization is a systematic process that uses pre-established criteria to determine the order for developing new evidence that patients, providers and other decision-makers need to make informed choices about health interventions and strategies.

There are three stages of evidence prioritization activities, which should be addressed sequentially:
Stage		Description
1.	Identify the mission, vision and objectives for research	For an example, review PCORI Draft National Priorities for Research and Research Agenda.
2.	Identify and prioritize research topics, including priority populations, conditions, interventions and strategies	For an example, review AHRQ's Priority Health Services, Conditions and Populations. To develop a topic, think about focusing on a specific area of a general topic. For example, within the general topic of childhood asthma, you may be interested in preventing asthma in young children. It is also helpful to scan the research literature for work that has already been completed to identify evidence gaps and key questions that have not been answered adequately.
3.	Identify and prioritize research questions	For an example, review the AHRQ Future Research Needs for Diagnosis of Obstructive Sleep Apnea. Several steps can be followed to move from a high-level key question to a specific researchable question: Develop high-level key questions, such as, "What research is needed to address evidence gaps about primary prevention of incident asthma in children?" Refine your key questions by assessing them against the Populations, Interventions, Comparators, and Outcomes (PICO) framework. This framework is helpful to gather missing information about your key question and to refine it into a researchable question. For example, to refine the key question about primary prevention of incident asthma in children, you might determine what is known/not known about the population(s) of interest, intervention(s) of interest, comparator(s) of interest and outcome(s) of interest: Populations: What populations are of interest? Infants? All children? Children between the ages of two and six? Are you interested in urban/suburban/rural populations? Certain racial and ethnic groups? Specific regions, such as national, state, county or metropolitan areas? Certain risk groups, such as previously-, currently- and never-diagnosed populations? Interventions: What interventions are relevant, including screening interventions, diagnostic interventions, school-based interventions, primary care-based interventions, and home-based interventions? Comparators: After the interventions of interest are identified, what interventions can serve as the comparator(s) of interest? Outcomes: Which outcomes are of interest - surrogate, clinical, or patient-reported? Identify specific future research needs. On the basis of your efforts to refine key questions using the PICO framework are there any specific future research needs that can be outlined for consideration in a prioritization exercise? For example, in urban K-8 school settings, what is the comparative cost effectiveness of annual spirometry screening for at-risk youth (previously or currently diagnosed) compared to annual or semi-annual spirometry screening for all youth (previously-, currently- and never-diagnosed)?

Click this link to use the Types of Evidence Prioritization Tool to define the purpose and scope of your evidence prioritization activity.

WHO plays a role in evidence prioritization?

Most researchers do not have the time or resources to carry their message into every health care practice setting. Having stakeholders partner with researchers can improve the application of evidence in practice. When conducting evidence prioritization, you will be able to identify relevant stakeholder groups for your evidence prioritization activity. If necessary, you will be able to identify relevant groups by condition, intervention(s), or health care strategie(s) and explain the importance of achieving balance among competing viewpoints.

To learn more about involving stakeholders in your research project, use this tool.

WHY is evidence prioritization important in your research?

Evidence prioritization is important because it helps patients, providers, and other decision-makers to make informed choices about the research topics that are most important to address. Furthermore, the evidence prioritization process can help to assure that important research questions are framed in ways that are most relevant to health and healthcare decision makers. This step in comparative effectiveness research is a systematic process that uses pre-specified criteria to determine the most important evidence necessary to make informed health care choices.

WHERE in the Tufts CTSI CER Spectrum is evidence prioritization?

This is the first step in the Tufts CTSI Translational Spectrum of CER. It is important for identifying key research questions and specific research needs. These steps are critical to the entire comparative effectiveness research process.

If you are unsure of where your research question falls on the Spectrum, click here.

HOW do you conduct evidence prioritization?

How is evidence prioritization conducted?

Use the Types of Evidence Prioritization Tool to help identify the stage of evidence prioritization you need. Click the Evidence Prioritization Tool to follow detailed instructions and a step-by-step guide on how to conduct Evidence Prioritization with your research team using the ten steps outlined below.

The ten evidence prioritization steps are outlined below.
Step #1: Agree on common definitions for your stakeholder engagement activity. You may decide to adopt each definition as described on this site, or modify definitions to meet your project's needs. Your research team's agreement about these terms will help to minimize confusion and misunderstandings during the prioritization activity. Step #2: Agree on the purpose and scope of your stakeholder engaged evidence prioritization activity. To establish agreement on the purpose and scope of your stakeholder engagement activity, talk to your research team about what type of evidence prioritization activity is needed. Step #3: Identify appropriate stakeholder groups for your project. To begin, use the 7Ps to develop a list of relevant stakeholder groups. Then, customize the list of relevant stakeholders to your research topic by asking your research team a series of questions. See Evidence Prioritization Tool. Step #4: Determine the appropriate size of your stakeholder evidence prioritization group and ensure balanced representation. Use the 7Ps to start building your stakeholder group and make sure every relevant viewpoint is represented. Within each of the 7Ps, consider every type of stakeholder. For example: providers may include individual clinicians representing different specialties and levels of care; institutional providers such as hospital and nursing home administrators, school nursing programs or community health centers. Consider every type of stakeholder who has an interest or stake in the outcome of your research. Once you have built your list, consider the relative balance of stakeholder groups with potentially competing viewpoints. For example: stakeholders with a commercial interest in the outcomes of research - including product makers, specialty providers and some payers - should represent a minority in evidence prioritization activities or may need to be excused from participating. Participation by stakeholders involved in providing or using health care services should be balanced by participation by stakeholders who pay for them. You should also ensure balanced representation by primary care physicians and specialists, since their needs may be different. Also consider whether your research is subject to regulation under the Paperwork Reduction Act (PRA) of 1995. If your project is funded through a Federal contract, it must comply with the PRA (note: projects funded privately, by states, and by Federal grants and cooperative agreements are not subject to this requirement). Evidence prioritization often concludes with a voting, balloting or survey exercise, and any of these exercises qualify as a federally-funded "survey" if they are funded through a Federal Contract and involve more than nine individuals who are not Federal employees. Qualifying surveys must be approved by the Federal government before they are used, and this process can take several months. With these facts in mind, you may wish to limit the size of your stakeholder board to nine individuals plus federal employees. Step #5: Clarify for your research team and stakeholders how stakeholder-engaged evidence prioritization can lead to informed decision-making for your community. Step #6: Develop an evidence prioritization protocol. An evidence prioritization protocol is a thorough description of the scope, objectives, and proposed methods by which you hope to arrive at a mission for research, a prioritized list of research topics, or a prioritized list of research questions. You can use this protocol to assure that every team member and stakeholder understands the process. The mechanisms you use to develop a priority list are also important. If your objective is to develop a consensus ranking, consider using a Delphi or modified Delphi approach. If you wish to elicit and record a variety of opinions, consider using a ballot exercise. Balloting can be broken down further into two types: ranking ballots, in which every individual ranks the full list of candidate topics or questions from their highest to their lowest priority, and voting ballots, in which every individual chooses a small number of priority topics (with or without rank) from the whole list and the research team combines votes to establish a group ranking. Step #7: Recruit individuals to fill the priority list of stakeholders for your evidence prioritization process. Once you have identified potential stakeholders, send a letter with a brief description of your evidence prioritization process, an outline of their proposed roles and responsibilities, and information about honoraria. Once a commitment is obtained, collect disclosure of interest (DOI) forms and other information from the stakeholder, as necessary. Step #8: Conduct the evidence prioritization exercise. Follow the protocol you developed in Step #6. Step #9: Evaluate and report on the evidence prioritization exercise. Report stakeholder activities in manuscripts and contract reports. As the evidence base grows, implement changes in future evidence prioritization activities. As changes are adopted, an iterative assessment process should follow. For evidence prioritization that is funded through a Federal contract, all public reports must be 508 compliant. Step #10: Sustain relationships with your stakeholders. Share the results of your evidence prioritization exercise with each stakeholder and invite stakeholders to share the results with others, including health decision makers and those who may be affected by health decisions. Invite stakeholders to participate in follow-up engagement activities. If your community or research program will use the results of this prioritization exercise to inform future research, invite stakeholders from the evidence prioritization process to continue working on the research.

The ten evidence prioritization steps are outlined below.

Step #1: Agree on common definitions for your stakeholder engagement activity. You may decide to adopt each definition as described on this site, or modify definitions to meet your project's needs. Your research team's agreement about these terms will help to minimize confusion and misunderstandings during the prioritization activity.

Step #2: Agree on the purpose and scope of your stakeholder engaged evidence prioritization activity. To establish agreement on the purpose and scope of your stakeholder engagement activity, talk to your research team about what type of evidence prioritization activity is needed.

Step #3: Identify appropriate stakeholder groups for your project. To begin, use the 7Ps to develop a list of relevant stakeholder groups. Then, customize the list of relevant stakeholders to your research topic by asking your research team a series of questions. See Evidence Prioritization Tool.

Step #4: Determine the appropriate size of your stakeholder evidence prioritization group and ensure balanced representation. Use the 7Ps to start building your stakeholder group and make sure every relevant viewpoint is represented. Within each of the 7Ps, consider every type of stakeholder. For example: providers may include individual clinicians representing different specialties and levels of care; institutional providers such as hospital and nursing home administrators, school nursing programs or community health centers. Consider every type of stakeholder who has an interest or stake in the outcome of your research.

Once you have built your list, consider the relative balance of stakeholder groups with potentially competing viewpoints. For example: stakeholders with a commercial interest in the outcomes of research - including product makers, specialty providers and some payers - should represent a minority in evidence prioritization activities or may need to be excused from participating.

Participation by stakeholders involved in providing or using health care services should be balanced by participation by stakeholders who pay for them. You should also ensure balanced representation by primary care physicians and specialists, since their needs may be different.

Also consider whether your research is subject to regulation under the Paperwork Reduction Act (PRA) of 1995.

If your project is funded through a Federal contract, it must comply with the PRA (note: projects funded privately, by states, and by Federal grants and cooperative agreements are not subject to this requirement). Evidence prioritization often concludes with a voting, balloting or survey exercise, and any of these exercises qualify as a federally-funded "survey" if they are funded through a Federal Contract and involve more than nine individuals who are not Federal employees. Qualifying surveys must be approved by the Federal government before they are used, and this process can take several months. With these facts in mind, you may wish to limit the size of your stakeholder board to nine individuals plus federal employees.

Step #5: Clarify for your research team and stakeholders how stakeholder-engaged evidence prioritization can lead to informed decision-making for your community.

Step #6: Develop an evidence prioritization protocol.

An evidence prioritization protocol is a thorough description of the scope, objectives, and proposed methods by which you hope to arrive at a mission for research, a prioritized list of research topics, or a prioritized list of research questions. You can use this protocol to assure that every team member and stakeholder understands the process.

The mechanisms you use to develop a priority list are also important. If your objective is to develop a consensus ranking, consider using a Delphi or modified Delphi approach. If you wish to elicit and record a variety of opinions, consider using a ballot exercise. Balloting can be broken down further into two types: ranking ballots, in which every individual ranks the full list of candidate topics or questions from their highest to their lowest priority, and voting ballots, in which every individual chooses a small number of priority topics (with or without rank) from the whole list and the research team combines votes to establish a group ranking.

Step #7: Recruit individuals to fill the priority list of stakeholders for your evidence prioritization process. Once you have identified potential stakeholders, send a letter with a brief description of your evidence prioritization process, an outline of their proposed roles and responsibilities, and information about honoraria. Once a commitment is obtained, collect disclosure of interest (DOI) forms and other information from the stakeholder, as necessary.

Step #8: Conduct the evidence prioritization exercise. Follow the protocol you developed in Step #6.

Step #9: Evaluate and report on the evidence prioritization exercise. Report stakeholder activities in manuscripts and contract reports. As the evidence base grows, implement changes in future evidence prioritization activities. As changes are adopted, an iterative assessment process should follow. For evidence prioritization that is funded through a Federal contract, all public reports must be 508 compliant.

Step #10: Sustain relationships with your stakeholders. Share the results of your evidence prioritization exercise with each stakeholder and invite stakeholders to share the results with others, including health decision makers and those who may be affected by health decisions. Invite stakeholders to participate in follow-up engagement activities. If your community or research program will use the results of this prioritization exercise to inform future research, invite stakeholders from the evidence prioritization process to continue working on the research.

Evidence Synthesis

Evidence synthesis is the process of systematically reviewing, evaluating and integrating research evidence (also known as systematic review, meta-analysis, research synthesis or pooling).

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is evidence synthesis?

Key terms that are needed to guide your evidence prioritization activity, such as stakeholder, engagement, comparative effectiveness research (CER), and patient-centered outcomes research (PCOR).
What evidence synthesis is.

WHO are the stakeholders in evidence synthesis activities?

You will be able to:
Identify relevant stakeholder groups for evidence synthesis, if necessary according to condition, intervention or health care strategy. Explain the importance of achieving balance among competing viewpoints in your relevant stakeholder groups.

WHY engage stakeholders in evidence synthesis activities?

You will understand the importance of community engagement as a necessary element of evidence synthesis and comparative effectiveness research (CER).

HOW to engage stakeholders in evidence prioritization activities?

You will know:
How to contact and engage stakeholders for evidence synthesis. The barriers and facilitators for engaging stakeholders in evidence synthesis.

WHAT is evidence synthesis?

Evidence synthesis is the process of systematic review, evaluating and integrating research evidence.

Evidence synthesis is also known as systematic review, meta-analysis, research synthesis or pooling.

There are several barriers to the successful engagement of stakeholders in evidence synthesis:
Lack of available stakeholder time Insufficient time or compensation to engage stakeholders Inadequate researcher time (the need for a quick response trumps the time required for the stakeholder to participate in the discussion) Reluctance of stakeholder to participate because their needs were not met in a previous engagement activity. With careful planning and communication, many of these barriers can be avoided or addressed.

There are several barriers to the successful engagement of stakeholders in evidence synthesis:

Lack of available stakeholder time
Insufficient time or compensation to engage stakeholders
Inadequate researcher time (the need for a quick response trumps the time required for the stakeholder to participate in the discussion)
Reluctance of stakeholder to participate because their needs were not met in a previous engagement activity.

With careful planning and communication, many of these barriers can be avoided or addressed.

WHO are the stakeholders in evidence synthesis activities?

As in other modules in this series, the best way to identify relevant stakeholder groups for evidence synthesis activities is to start with the 7Ps .

Once your stakeholder groups are identified, there are several methods for identifying individual stakeholders:
Ask your study team for recommendations. You are likely to receive good recommendations from people you trust; however, be aware the possibility for bias exists. Your team may recommend people who are similar to themselves. Conduct snowball sampling. Similar to asking your study team for recommendations, start by contacting people you know. These individuals may be suitable to serve as stakeholders on your evidence synthesis project, or they may be able to help you find other candidates. The group continues to grow, like a rolling snowball. Here, too, there is possibility for bias if people you know recommend candidates who are similar to themselves. Search the published literature to identify key informants. Published literature is an excellent place to find the experts in a given field; however, be aware of possible bias from recommending only people who publish frequently or prominently. Search databases and/or directories. In this method, you may encounter possible bias in the criteria used to establish the database/directory, and from the search criteria you use. Conduct Internet searches of individuals and organizations. Many, but not all, people and organizations have an online presence. Be aware that this method excludes stakeholders without an online presence, and further bias may stem from the search strategy you use. Conduct random sampling of a database or directory. Here, you may find bias in the criteria used to establish the database or directory.

Once your stakeholder groups are identified, there are several methods for identifying individual stakeholders:

Ask your study team for recommendations. You are likely to receive good recommendations from people you trust; however, be aware the possibility for bias exists. Your team may recommend people who are similar to themselves.
Conduct snowball sampling. Similar to asking your study team for recommendations, start by contacting people you know. These individuals may be suitable to serve as stakeholders on your evidence synthesis project, or they may be able to help you find other candidates. The group continues to grow, like a rolling snowball. Here, too, there is possibility for bias if people you know recommend candidates who are similar to themselves.
Search the published literature to identify key informants. Published literature is an excellent place to find the experts in a given field; however, be aware of possible bias from recommending only people who publish frequently or prominently.
Search databases and/or directories. In this method, you may encounter possible bias in the criteria used to establish the database/directory, and from the search criteria you use.
Conduct Internet searches of individuals and organizations. Many, but not all, people and organizations have an online presence. Be aware that this method excludes stakeholders without an online presence, and further bias may stem from the search strategy you use.
Conduct random sampling of a database or directory. Here, you may find bias in the criteria used to establish the database or directory.

How can you avoid bias in your selection of stakeholders, when each method described above has its own bias potential? By using multiple methods, you will be most likely to recruit stakeholders from a variety of backgrounds and specialties.

In addition, here are some further considerations to ensure appropriate representation and avoid bias:
Physicians	Choose physicians from primary care and specialty groups that patients may encounter in their diagnosis and treatment (e.g., primary care, radiology, pathology, non-invasive treatment, surgery, radiation therapy and others). When opinions differ regarding optimal approaches, attempt to balance the discussion by ensuring representation from opposing viewpoints.
Patients	To emphasize generalizability and to avoid bias from having a particular disorder, choose patients with and without the primary condition or conditions of interest.
Policymakers, Purchasers, Payers, Product Makers	Policymakers may have differing perspectives depending on their roles. Choose policymakers representative of several different functions, such as regulatory approval or reimbursement approval.
Principal Investigators (Researchers)	Basic science, clinical researchers, systems researchers and policy researchers may have alternative viewpoints, experiences and values that benefit the research.

WHY engage stakeholders in evidence synthesis?

Nationally, research is becoming more patient-centered through the engagement of stakeholders to improve health care. The research community needs to engage stakeholders to ensure patients, physicians and policymakers have the high-quality evidence necessary to make the best health care decisions.

Stakeholders are engaged to:
Nominate topics by identifing important and significant knowledge gaps (topic generation and identification) Further define the scope of priority topics (topic refinement) Facilitate study design, especially to help identify the most relevant patient population, interventions, comparators and outcomes Prioritize topics and outcomes (future research needs prioritization) Provide citations and characterize potential issues with implementation, such as barriers Translate research into health policy through discussions with policymakers.

Stakeholders are engaged to:

Nominate topics by identifing important and significant knowledge gaps (topic generation and identification)
Further define the scope of priority topics (topic refinement)
Facilitate study design, especially to help identify the most relevant patient population, interventions, comparators and outcomes
Prioritize topics and outcomes (future research needs prioritization)
Provide citations and characterize potential issues with implementation, such as barriers
Translate research into health policy through discussions with policymakers.

To specify an analytic framework and formulate an answerable research question, use the acronym PICO:
P	Patient population	Identify the relevant patient population of interest
I	Interventions	Identify the intervention being proposed (test, treatment)
C	Comparator	Identify the comparison or control intervention used to compare the intervention to other treatment options
O	Outcomes	Identify the outcome of interest (e.g., mortality, morbidity, cost, benefit, harm).

WHERE in the Tufts CTSI CER Spectrum is evidence synthesis?

Evidence synthesis is the third step in comparative effectiveness research (CER). Once you have developed your research question and conducted your trials or studies, your research team and stakeholders will conduct systematic reviews and meta-analysis of the evidence you generated. The primary purpose of this module on Evidence Synthesis is to guide you in the engagement of stakeholders as you conduct an evidence synthesis research activity.

If you are unsure of where your research question falls on the Spectrum, click here.

HOW do you engage stakeholders for evidence synthesis?

For a successful partnership with stakeholders:
Engage them at the very beginning of your research process. This lets them know you value their input, and will help to gain buy-in with your approach. Clearly outline your expectations, especially the anticipated time commitment for stakeholders, and the types of tasks or activities they will be asked to perform. Find common ground. Researchers and stakeholders enter the research process with different needs and goals: by understanding everyone's needs, you can develop the best possible set of answerable research questions. Be ready for the give-and-take. A partnership with a stakeholder is like any other relationship: there is discussion, compromise, and prioritization. Respect their knowledge and time. Contributing to research is a significant time investment for all stakeholders involved, and their knowledge will help you to obtain the best results. By respecting their contribution, stakeholders will be more likely to commit to participation in the duration of your project, and beyond. Trust each other. You and your stakeholders will be more forthright and honest if there is an environment of mutual trust.

For a successful partnership with stakeholders:

Engage them at the very beginning of your research process. This lets them know you value their input, and will help to gain buy-in with your approach.
Clearly outline your expectations, especially the anticipated time commitment for stakeholders, and the types of tasks or activities they will be asked to perform.
Find common ground. Researchers and stakeholders enter the research process with different needs and goals: by understanding everyone's needs, you can develop the best possible set of answerable research questions.
Be ready for the give-and-take. A partnership with a stakeholder is like any other relationship: there is discussion, compromise, and prioritization.
Respect their knowledge and time. Contributing to research is a significant time investment for all stakeholders involved, and their knowledge will help you to obtain the best results. By respecting their contribution, stakeholders will be more likely to commit to participation in the duration of your project, and beyond.
Trust each other. You and your stakeholders will be more forthright and honest if there is an environment of mutual trust.

The first step in engaging stakeholders for your evidence synthesis activity is by contacting them to determine their interest, appropriateness, and availability. This can be accomplished through:
Formal written communication, by sending letters through the postal service Informal written communication, by sending email Making phone calls to individuals or groups Offering presentations or informational sessions in-person or via webinars Placing advertisements in journals, newspapers, and online Hanging posters to recruit stakeholders.

Next, you will need to engage your stakeholders, obtaining their initial input and prioritization. Options for this step include:
Individual interviews, in-person or by phone: these can be formal or informal. Semi-structured interviews: for small groups of three to four people, these have an open framework to encourage focused, conversational, two-way communication. All interviewees are asked the same questions, which are developed ahead of time. Additional questions may be asked to probe for additional details or clarify particular issues. Focus groups: these meetings of groups of four to 12 stakeholders should be facilitated by a skilled moderator to obtain information about preferences and opinions in a relaxed, non-threatening environment. The moderator may used a series of predetermined questions to spark group discussion. Questionnaire: a series of questions designed to elicit opinions, beliefs, and attitudes. These can be administered online, by mail, over the phone, or in-person. Citizens' juries: Diverse members of the public act as jurors, and are given information relevant to the issue under debate by expert witnesses (innovators, patients, health care policymakers, and clinicians). A facilitator or moderator guides the ensuing discussion. Town meetings: Local media invite individuals who reside in a specific area to attend a public meeting to discuss issues relevant to their community. State and local officials, health care providers, researchers, manufacturers, and topic experts may be present, and anyone may speak. Symposia, conferences, workshops, or forums: designed for large groups, these events typically center around a particular topic or theme.

Next, you will need to engage your stakeholders, obtaining their initial input and prioritization. Options for this step include:

Individual interviews, in-person or by phone: these can be formal or informal.
Semi-structured interviews: for small groups of three to four people, these have an open framework to encourage focused, conversational, two-way communication. All interviewees are asked the same questions, which are developed ahead of time. Additional questions may be asked to probe for additional details or clarify particular issues.
Focus groups: these meetings of groups of four to 12 stakeholders should be facilitated by a skilled moderator to obtain information about preferences and opinions in a relaxed, non-threatening environment. The moderator may used a series of predetermined questions to spark group discussion.
Questionnaire: a series of questions designed to elicit opinions, beliefs, and attitudes. These can be administered online, by mail, over the phone, or in-person.
Citizens' juries: Diverse members of the public act as jurors, and are given information relevant to the issue under debate by expert witnesses (innovators, patients, health care policymakers, and clinicians). A facilitator or moderator guides the ensuing discussion.
Town meetings: Local media invite individuals who reside in a specific area to attend a public meeting to discuss issues relevant to their community. State and local officials, health care providers, researchers, manufacturers, and topic experts may be present, and anyone may speak.
Symposia, conferences, workshops, or forums: designed for large groups, these events typically center around a particular topic or theme.

There are several techniques you may wish to employ for your stakeholders to share their priorities.

Delphi technique: A series of consecutive anonymous questionnaires to determine the perceptions of a group of individuals.

Modified Delphi technique: The intent of engagement (to predict future events and arrive at consensus) and the procedure (a series of questionnaires) is similar to the Delphi technique. In the Delphi technique, the first questionnaire is used to elicit research topics. In the Modified Delphi technique, the first questionnaire includes preselected research topics.

Nominal Group Technique: A structured problem-solving or ideas-generating activity in which individuals' ideas are gathered and combined in a face-to-face, nonthreatening group environment. The process is intended to promote creative participation in group problem-solving. Each member of the group is invited to express their opinions, which are used to generate a list of priorities. Members may be asked to vote or rank priorities from the list. The voting process may occur multiple times.

When retaining a moderator to facilitate focus groups, he or she should:
Be neutral Have a cultural understanding of multiple stakeholder environments Ensure all stakeholders are heard, and that no comments are considered superior or inferior to others Set the tone with pre-meeting icebreakers and reading materials Encourage participation and focused discussion.

Click here to download the Tufts CTSI Evidence Synthesis Tool with examples of evidence synthesis.

Evidence Interpretation and Integration

Evidence interpretation and integration are the processes of conducting cost-effective analysis and decision analysis of research evidence to specify the options, identify important outcomes, determine the chances of harm and benefit, evaluate the quality of studies, integrate the evidence, and report on clinically meaningful results.

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is evidence interpretation and integration?

At the conclusion of this module, you will understand and be able to describe:
Key terms that are needed to guide your evidence interpretation and integration activity, such as stakeholder, engagement, comparative effectiveness research (CER), and patient-centered outcomes research (PCOR). What evidence interpretation and integration is.

WHO are the stakeholders in evidence interpretation and integration activities?

You will be able to:
Identify relevant stakeholder groups for evidence interpretation and integration, if necessary according to condition, intervention(s), or health care strategie(s). Explain the importance of achieving balance among competing viewpoints in your relevant stakeholder groups.

WHY engage stakeholders in evidence interpretation and integration activities?

You will understand the importance of community engagement as a necessary element of evidence interpretation and integration, and comparative effectiveness research.

HOW to engage stakeholders in evidence interpretation and integration activities?

You will know:
How to contact and engage stakeholders for evidence interpretation and integration. The barriers and facilitators for engaging stakeholders in evidence interpretation and integration.

WHAT is evidence interpretation and integration?

The processes of conducting cost-effective analysis and decision analysis of research evidence to specify the options, identify important outcomes, determine the chances of harm and benefit, evaluate the quality of studies, integrate the evidence, and report on clinically meaningful results.

There are several barriers to the successful engagement of stakeholders in evidence interpretation and integration:
Lack of available stakeholder time Insufficient time or compensation to engage members of the public Inadequate researcher time (the need for a quick response trumps the time required for the community to participate in the discussion) Reluctance of stakeholder to participate, because their needs were not met in a previous engagement activity.

With careful planning and communication, many of these barriers can be avoided or addressed.

WHO are the stakeholders in evidence interpretation and integration activities?

As in other modules in this series, the best way to identify relevant stakeholder groups for evidence interpretation and integration activities is to start with the 7Ps.

Once your stakeholder groups are identified, there are several methods for identifying individual stakeholders:
Ask your study team for recommendations. You are likely to receive good recommendations from people you trust; however, be aware the possibility for bias exists. You team may recommend people who are similar to themselves. Conduct snowball sampling. Similar to asking your study team for recommendations, start by contacting people you know. These individuals may be suitable to serve as stakeholders on your evidence synthesis project, or they may be able to help you find other candidates. The group continues to grow, like a rolling snowball. Here, too, there is possibility for bias if people you know recommend candidates who are similar to themselves. Search the published literature to identify key informants. Published literature is an excellent place to find the experts in a given field; however, be aware of possible bias from recommending only people who publish frequently or prominently. Search databases and/or directories. In this method, you may encounter possible bias in the criteria used to establish the database/directory, and from the search criteria you use. Conduct Internet searches of individuals and organizations. Many, but not all, people and organizations have an online presence. Be aware that this method excludes stakeholders without an online presence, and further bias may stem from the search strategy you use. Conduct random sampling of a database or directory. Here, you may find bias in the criteria used to establish the database or directory.

Once your stakeholder groups are identified, there are several methods for identifying individual stakeholders:

Ask your study team for recommendations. You are likely to receive good recommendations from people you trust; however, be aware the possibility for bias exists. You team may recommend people who are similar to themselves.
Conduct snowball sampling. Similar to asking your study team for recommendations, start by contacting people you know. These individuals may be suitable to serve as stakeholders on your evidence synthesis project, or they may be able to help you find other candidates. The group continues to grow, like a rolling snowball. Here, too, there is possibility for bias if people you know recommend candidates who are similar to themselves.
Search the published literature to identify key informants. Published literature is an excellent place to find the experts in a given field; however, be aware of possible bias from recommending only people who publish frequently or prominently.
Search databases and/or directories. In this method, you may encounter possible bias in the criteria used to establish the database/directory, and from the search criteria you use.
Conduct Internet searches of individuals and organizations. Many, but not all, people and organizations have an online presence. Be aware that this method excludes stakeholders without an online presence, and further bias may stem from the search strategy you use.
Conduct random sampling of a database or directory. Here, you may find bias in the criteria used to establish the database or directory.

In addition, here are some further considerations to ensure appropriate representation and avoid bias:

Providers	Choose health care providers and physicians from primary care and specialty groups that patients may encounter in their diagnosis and treatment (e.g., primary care, radiology, pathology, non-invasive treatment, surgery, radiation therapy and others). When opinions may differ regarding optimal approaches, attempt to balance the discussion by ensuring representation from opposing viewpoints.
Patients and the Public	To emphasize generalizability and to avoid bias from having a particular disorder, choose patients with and without the primary condition or conditions of interest.
Purchasers/Payers	Purchasers and payers may have a particular interest in the cost of care, particularly those for which they are responsible such as the cost of new technologies adn the cost of care in the absence of that new technology.
Policymakers	Policymakers may have differing perspectives depending on their roles. Choose policymakers representative of several different functions, such as regulatory approval or reimbursement approval.
Product Makers	Drug and device manufacturers often will have the primary data about the benefits and harms of their product and a well-developed business model to understand the placement and context of their product in the current treatment and testing environment.
Principal Investigators (Researchers)	Basic science (e.g., understanding of pathophysiological mechanisms that should be captured in a model), clinical researchers (e.g., clinical options and patient populations), systems researchers (e.g., implementations needs and barriers) and policy researchers (e.g., perspectives on requirements for changing policy) may have alternative viewpoints, experiences and values that may benefit the research.

WHY engage stakeholders in evidence interpretation and integration?

Stakeholders are engaged to:
Identify all relevant clinical options (patients, providers, principal investigators, product makers, payers, and policymakers) Characterize the most appropriate patient population, and subgroups (providers, principal investigators, product makers, payers, and policymakers) Specify the settings and locations (health system) and the perspective (for determining which costs will be considered (patients, providers, principal investigators, product makers, payers, and policymakers) Encapsulate in the decision model a coherent understanding of the disease or condition, i.e., a model structured to represent the natural history or progression of the disease, the effect of the intervention on that natural history, and the link between the clinical options and the outcomes (providers, principal investigators, product makers, payers and policymakers) Prioritize the outcomes that should be considered and captured with the decision model (patients, providers, principal investigators, product makers, payers, and policymakers) Provide citations and characterize potential issues with study implementation, such as barriers Translate research into clinical practice (providers), into shared decision-making (providers, patients and policy makers), and health policy, e.g., approval and reimbursement for new technologies or drugs (payers, policymakers) Prioritize topics and outcomes through value-of-information research using the decision model for research prioritization (patients, providers, principal investigators, product makers, payers, and policymakers)

Stakeholders are engaged to:

Identify all relevant clinical options (patients, providers, principal investigators, product makers, payers, and policymakers)
Characterize the most appropriate patient population, and subgroups (providers, principal investigators, product makers, payers, and policymakers)
Specify the settings and locations (health system) and the perspective (for determining which costs will be considered (patients, providers, principal investigators, product makers, payers, and policymakers)
Encapsulate in the decision model a coherent understanding of the disease or condition, i.e., a model structured to represent the natural history or progression of the disease, the effect of the intervention on that natural history, and the link between the clinical options and the outcomes (providers, principal investigators, product makers, payers and policymakers)
Prioritize the outcomes that should be considered and captured with the decision model (patients, providers, principal investigators, product makers, payers, and policymakers)
Provide citations and characterize potential issues with study implementation, such as barriers
Translate research into clinical practice (providers), into shared decision-making (providers, patients and policy makers), and health policy, e.g., approval and reimbursement for new technologies or drugs (payers, policymakers)
Prioritize topics and outcomes through value-of-information research using the decision model for research prioritization (patients, providers, principal investigators, product makers, payers, and policymakers)

Provide a Statement of Decision Problem and Objective
What decision problem prompted the analysis or what is the objective of the model evaluation? What disease or condition is being evaluated? What is the patient group? What are the diagnostic and/or treatment pathways? Whose perspective (primary decision-maker) does the analysis take?

WHERE in the Tufts CTSI CER Spectrum is Evidence Interpretation and Integration?

Evidence Interpretation and Integration is the fourth step in comparative effectiveness research (CER). Once you have developed your research question, conducted your trials or studies and systematic reviews to determine if an intervention works, your research team and stakeholders will conduct cost-effective analysis and decision analysis integrate best currently available evidence on choices (testing or treatment options), chances (the likelihood of various harms and benefits occurring with each option) and consequences (the valuaton of the outcome, e.g., life expectancy or quality-adjusted life expectancy) and assess the impact of uncertainty, tradeoffs, and values on whether that intervention should be adopted. The primary purpose of this module on evidence interpretation and integration is to guide you in the engagement of stakeholders as you conduct evidence interpretation and integration research activities. If you are unsure of where your research question falls on the Spectrum, click here.

HOW do you engage stakeholders for evidence interpretation and integration?

For a successful partnership with stakeholders:
Engage them at the very beginning of your research process. This lets them know you value their input, and will help to gain buy-in with your approach. Clearly outline your expectations, especially the anticipated time commitment for stakeholders, and the types of tasks or activities they will be asked to perform. Find common ground. Researchers and stakeholders enter the research process with different needs and goals: by understanding everyone's needs, you can develop the best possible set of answerable research questions. Be ready for the give-and-take. A partnership with a stakeholder is like any other relationship: there is discussion, compromise, and prioritization. Respect their knowledge and time. Contributing to research is a significant time investment for all stakeholders involved, and their knowledge will help you to obtain the best results. By respecting their contribution, stakeholders will be more likely to commit to participation in the duration of your project, and beyond. Trust each other. You and your stakeholders will be more forthright and honest if there is an environment of mutual trust.

For a successful partnership with stakeholders:

Engage them at the very beginning of your research process. This lets them know you value their input, and will help to gain buy-in with your approach.
Clearly outline your expectations, especially the anticipated time commitment for stakeholders, and the types of tasks or activities they will be asked to perform.
Find common ground. Researchers and stakeholders enter the research process with different needs and goals: by understanding everyone's needs, you can develop the best possible set of answerable research questions.
Be ready for the give-and-take. A partnership with a stakeholder is like any other relationship: there is discussion, compromise, and prioritization.
Respect their knowledge and time. Contributing to research is a significant time investment for all stakeholders involved, and their knowledge will help you to obtain the best results. By respecting their contribution, stakeholders will be more likely to commit to participation in the duration of your project, and beyond.
Trust each other. You and your stakeholders will be more forthright and honest if there is an environment of mutual trust.

The first step in engaging stakeholders for your evidence synthesis activity is by contacting them to determine their interest, appropriateness, and availability. This can be accomplished through:
Formal written communication, by sending letters through the postal service Informal written communication, by sending email Making phone calls to individuals or groups Offering presentations or informational sessions in-person or via webinars Placing advertisements in journals, newspapers, and online Hanging posters to recruit stakeholders.

Next, you will need to engage your stakeholders, obtaining their initial input and prioritization. Options for this step include:
Individual interviews, in-person or by phone: these can be formal or informal. Semi-structured interviews: for small groups of three to four people, these have an open framework to encourage focused, conversational, two-way communication. All interviewees are asked the same questions, which are developed ahead of time. Additional questions may be asked to probe for additional details or clarify particular issues. Focus groups: these meetings of groups of four to 12 stakeholders should be facilitated by a skilled moderator to obtain information about preferences and opinions in a relaxed, non-threatening environment. The moderator may used a series of predetermined questions to spark group discussion. Questionnaire: a series of questions designed to elicit opinions, beliefs, and attitudes. These can be administered online, by mail, over the phone, or in-person. Citizens' juries: Diverse members of the public act as jurors, and are given information relevant to the issue under debate by expert witnesses (innovators, patients, health care policymakers, and clinicians). A facilitator or moderator guides the ensuing discussion. Town meetings: Local media invite individuals who reside in a specific area to attend a public meeting to discuss issues relevant to their community. State and local officials, health care providers, researchers, manufacturers, and topic experts may be present, and anyone may speak. Symposia, conferences, workshops, or forums: designed for large groups, these events typically center around a particular topic or theme.

Next, you will need to engage your stakeholders, obtaining their initial input and prioritization. Options for this step include:

Individual interviews, in-person or by phone: these can be formal or informal.
Semi-structured interviews: for small groups of three to four people, these have an open framework to encourage focused, conversational, two-way communication. All interviewees are asked the same questions, which are developed ahead of time. Additional questions may be asked to probe for additional details or clarify particular issues.
Focus groups: these meetings of groups of four to 12 stakeholders should be facilitated by a skilled moderator to obtain information about preferences and opinions in a relaxed, non-threatening environment. The moderator may used a series of predetermined questions to spark group discussion.
Questionnaire: a series of questions designed to elicit opinions, beliefs, and attitudes. These can be administered online, by mail, over the phone, or in-person.
Citizens' juries: Diverse members of the public act as jurors, and are given information relevant to the issue under debate by expert witnesses (innovators, patients, health care policymakers, and clinicians). A facilitator or moderator guides the ensuing discussion.
Town meetings: Local media invite individuals who reside in a specific area to attend a public meeting to discuss issues relevant to their community. State and local officials, health care providers, researchers, manufacturers, and topic experts may be present, and anyone may speak.
Symposia, conferences, workshops, or forums: designed for large groups, these events typically center around a particular topic or theme.

There are several techniques you may wish to employ for your stakeholders to share their priorities.

Delphi Technique A series of consecutive anonymous questionnaires to determine the perceptions of a group of individuals.

Delphi Technique (modified) The intent of engagement (to predict future events and arrive at consensus) and the procedure (a series of questionnaires) is similar to the Delphi technique. In the Delphi technique, the first questionnaire is used to elicit research topics. In the Modified Delphi technique, the first questionnaire includes preselected research topics.

Nominal Group Technique A structured problem-solving or ideas-generating activity in which individuals' ideas are gathered and combined in a face-to-face, nonthreatening group environment. The process is intended to promote creative participation in group problem-solving. Each member of the group is invited to express their opinions, which are used to generate a list of priorities. Members may be asked to vote or rank priorities from the list. The voting process may occur multiple times.

When retaining a moderator to facilitate focus groups, he or she should:
Be neutral Have a cultural understanding of multiple stakeholder environments Ensure all stakeholders are heard, and that no comments are considered superior or inferior to others Set the tone with pre-meeting icebreakers and reading materials Encourage participation and focused discussion.

For examples of how to conduct evidence interpretation and integration through the steps in a decision analysis, please use the Evidence Interpretation and Integration tool.

Click here to download the Tufts CTSI Evidence Interpretation and Integration tool.

Dissemination & Application

Although closely related, dissemination and application differ on how they engage stakeholders and who is ultimately affected by the research results. Because of these differences, we will discuss dissemination and application separately when appropriate, but also illustrate their inherent association through examples and case studies.

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is dissemination and application?

At the conclusion of this module, you will understand and be able to describe:
Key terms that are needed to guide your dissemination and application activities, such as stakeholder, engagement, comparative effectiveness research (CER), and patient-centered outcomes research (PCOR). A broad definition of dissemination and application for research findings.

WHO are the stakeholders in dissemination and application activities?

You will learn with whom you need to communicate your research findings to ensure your study results are applied and health care is improved.

WHY engage stakeholders in dissemination and application activities?

You will be able to:
Explain why stakeholders are crucial to disseminating and applying evidence. Describe relevant examples of why certain dissemination and application efforts were successful and others were not.

HOW to engage stakeholders in dissemination and application activities?

You will know how to best present your project and its findings.

WHAT is dissemination? What is application?

Dissemination and application is the active distribution of research findings to real-world settings that may include adopting into clinical practice, informing practice guidelines, influencing health policy, and teaching research subjects and the general public.

Comparative effectiveness research (CER) that accounts for patient preferences is powerful information that can improve the outcomes of patient populations. This is evidenced by the formal adoption of patient preferences in guideline development processes. (IOM, Krahn) Further research is needed on how to engage stakeholders in developing guidelines, decision aids, comparing interventions, assessing findings, developing communication strategies, and dissemination within their formal and informal networks.

Dissemination is the distribution of information and intervention materials to a targeted public health or clinical audience. This occurs through various networks, settings, and media. Modes of disseminating research evidence include

peer-reviewed literature
social media
email
webinars or webcasts
reports
conferences
presentations
training programs
grand rounds
popular press
and word of mouth.

Similar to application, dissemination should be considered prior to and during the research process. This ensures your research results are relevant, timely, and tailored to your intended audience.

Application, or implementation, is the process by which research findings are used to improve health and health care. Terms in this definition are purposely vague because of the broad applications across all health care. For example, research findings can be any data, knowledge, results, or evidence that result from research. Further, the improvement of health and health care can be anything from approval of a new drug shown to alleviate disease symptoms, to updating a town-level advertisement campaign on influenza vaccine awareness. With such an encompassing definition, application can take many forms and play many roles. It is important to consider possible application strategies so you can design and target your research for intended uses and applications.

Please refer to the Dissemination and Application Tool for this section to learn how to most effectively convert your findings into the improvement of health.

WHO are the stakeholders in dissemination and application activities?

In dissemination and application, the stakeholders are anyone who may be impacted by the results of your research.

These include:
Policymakers The internal community, such as the Principal Investigator, research team, and other investigators The external community, such as the media and the public Research subjects, such as the patients involved in your study People likely to use your results, such as health care providers and students/academia.

You may wish to use the 7Ps to form a complete stakeholder list.

WHY engage stakeholders in dissemination and application?

Research published in the peer-reviewed literature draws a selective audience of predominantly practitioners and researchers who are likely already experts on the subject matter. The readership of these journals rarely includes patients and the general public. In addition, there can be a lag time of a decade or more between published research results and their meaningful impact on the health of patients and populations (Health Econ Group, 2008; Chen, 2010, #6). To address this gap, research that could potentially improve of the health of individuals and communities needs to be broadly disseminated to accelerate application to its target audience. Whether that audience is pediatric patients or governmental policymakers, getting the evidence out and implemented in an effective manner will better the health of all stakeholders.

When disseminating research results, two factors to consider are the directness and precision of the evidence being distributed. Limiting uncertainty during dissemination will assure your audience of the quality of the data, the results of your research, and implications for implementing your findings.

Directness, or how well interventions link to research outcomes, shows the impact of the intervention on the outcome of interest. A strong link will likely lead to widespread distribution, recognition for relevant stakeholders, and quicker implementation. Conversely, a weak or questionable link to outcomes of interest will cause more questions than answers, as the evidence contains more uncertainty.

Precision is the degree of certainty surrounding an effect estimate with respect to a given outcome. Precise evidence will garner similar results over multiple trials, and the precision of interventions can be reported when summarizing and disseminating data.

A frequent goal of comparative effectiveness research (CER) is the translation of results into clinical practice. From a CER standpoint, this is often in the form of a systematic review. Systematic reviews employ a scientific strategy to provide an overview of primary studies that summarize large bodies of evidence. The scientific approach limits biases in the search and allows clinicians, researchers, and other stakeholders to efficiently appraise and synthesize a body of literature on a given topic.

Two measures of the strength or quality of translation results (including systematic reviews) are risk of bias and consistency.

Risk of Bias is the degree to which the research outcome is protected against biases. The lower the risk, or higher the internal validity, the stronger the evidence (Owens, AHRQ, 2010).

Consistency is the degree to which the summarized studies have a common direction or effect (Owens, 2010). Inconsistent results among studies with similar measures and outcomes (outcome of interest) weaken the evidence base and increase uncertainty. When applying evidence, especially an intervention, in a real-world setting, it is important for safety, efficacy, and adoption to demonstrate consistent results. Please refer to the Dissemination and Application tool for this section to learn how to assess the risk of bias and consistency of the results you are attempting to apply.

WHERE in the Tufts CTSI CER Spectrum is dissemination and application?

Dissemination and application is the fifth step in comparative effectiveness research (CER). Once you have developed your research question, conducted your trials or studies and systematic reviews, and conducted cost-effectiveness analysis and decision analysis, it is time to translate your findings into guidelines, policy, social sciences, and implementation science.

The purpose of this module is to communicate your findings to as many stakeholders as possible, with the goal of improving health care.

If you are unsure of where your research questions falls on the Spectrum, click here.

HOW do you conduct dissemination and application?

Now that you know what dissemination and application are, and why to disseminate and apply your results, you need to know how to conduct these processes. Please refer to the Dissemination and Application Tool for a worksheet of the three exercises below which will help you determine how to disseminate and apply your study results.

DISSEMINATION AND APPLICATION TOOLS

EXERCISE #1 - DISSEMINATION

Make a list of what you are disseminating.
Remember, the products of your research are much more than a manuscript or poster.

My project aims to:
1.
2.
3.

My project will disseminate the following:
1.
2.
3.
4.
5.

EXERCISE #2 - STAKEHOLDERS

Who will your project impact or affect?

Brainstorm as complete a list of your project's stakeholders as you can.

Refer to the 7Ps. Draw a diagram with your project in the middle (rectangle) linking out to the stakeholders around you (ovals). Fill in the ovals with your stakeholders, as in the example below.

For more information, visit http://www.innovations.ac.uk/btg/resources/publications/dissemination.pdf.

EXERCISE #3 - PRESENTATION

How can you best present your project and its findings?
Complete the table below as a guide.

	Issue/Problem	Target Audience	Solution/Answer
1
2
3

Feedback and Assessment

This module will illustrate the process of assessing research so you can accurately appraise the impact of your work, recognize who it affects, learn from the experience, and build new areas of research. Assessing the positive and negative impact of research projects permits reflecting, learning, and improving on your team's efforts.

GOALS & LEARNING OBJECTIVES

In this module, you will learn:

WHAT is feedback and assessment?

At the conclusion of this module, you will understand and be able to describe:
Key terms that are needed to guide your feedback and assessment activities, such as stakeholder, engagement, comparative effectiveness research (CER), and patient-centered outcomes research (PCOR). The differences between process evaluation and product assessment. What should be assessed.

WHO should provide feedback and conduct the assessment?

You will learn how to identify the stakeholders who will provide the best information.

WHY engage stakeholders in feedback and assessment?

You will learn why:
Stakeholders are vital sources of information on the impact of research. Assessing the impact of research products is essential to understanding, learning, and improving benefits and areas of improvement.

HOW to conduct feedback and assessment?

You will learn:
How to best capture information that will determine the level of impact of your research. Based on these feedback, suggestions, and stakeholder engagement activities, you will be able to identify and improve future research opportunities. How to explore different methods to assess your research. Proper interpretation of, and actions from, your assessment.

WHAT is feedback and assessment?

In this module we differentiate between assessing the products of research and evaluating the process or methods used to develop that product. While research methodology and its resulting products are closely intertwined, the way in which they are assessed and their interpretation can vary drastically. We recognize that this distinction can be arbitrary and often blurred, but we draw a line between process evaluation and product assessment to discuss specific aspects unique to one or the other. For information on the evaluation of methods and processes, please review the other modules on this website.

Conducting feedback and assessment is important to determine:
Whether the goals of your study were met The impact of your research The quality of your research Best practices, areas of improvement, and future research opportunities.

Assessment strategies can take many different forms. Regardless of what is being assessed, there are four fundamentals every assessment must include:
Able to be measured Based on data Clear communication among those involved Continuous

If you apply the above principles when assessing your research, you will have a comprehensive, accurate assessment of the impact of your efforts.

WHO are the stakeholders in feedback and assessment activities?

It is essential to understand the effects of your research on different stakeholder groups. To achieve this, stakeholders must be involved in the planning, evaluation, and feedback of assessing research products. To decide which stakeholders to engage, use the 7Ps.

To learn more about involving stakeholders in your research project, use this tool.

WHY engage stakeholders in feedback and assessment?

Stakeholders can offer relevant, timely feedback regarding their own participation at each of the six stages of comparative effectiveness research (CER), including mechanisms for engagement, intensity of engagement, and support for patients and their advocates throughout the process. Researchers may also have feedback for stakeholders on the process of stakeholder engagement.

WHERE in the Tufts CTSI CER Spectrum is Feedback and Assessment?

Feedback and Assessment is the final step in comparative effectiveness research (CER). Once you have developed your research question, conducted your trials or studies and systematic reviews, conducted cost-effective analysis and decision analysis, and translated your findings into guidelines and policy, it is time to engage stakeholders for qualitative elicitation, data monitoring, and quality monitoring and measurement.

If you are unsure of where your research project falls on the Spectrum, click here.

HOW do you conduct feedback and assessment activities?

The following are tools and strategies for conducting feedback and assessment.

Click here to access the Feedback and Assessment Tools. To access the Feedback and Assessment Case Examples click here.

Tool or Strategy	Purpose	Drawbacks	Implementation
Report Cards	Visually present performance Measure against goals, organization averages, others being evaluated Written record of performance Identifies strengths and opportunities	Static, just "snapshot" analysis Does not account for special circumstances or unique situations	Disseminate on a regular basis Present results clearly and simply Offer accessible contact info for easy follow-up and further feedback
Multisource Assessment (360 degree feedback)	Captures perspectives from superiors, staff and peers Identifies positives and negatives in context of organization Keeps workers motivated to excel in all aspects of responsibilities	Resource-intensive Personalized review - does not account for benchmarks or organizational standards	Obtain feedback from right stakeholders Be aware of informal networks, relationships, or conflicts that could positively or negatively affect reviews
Concept and Process Mapping	Visual representation of workflow or processes Assesses potential conflicts and solutions Tells "what if" scenarios	Limited in scope Best for high-level solutions	Input data must be accurate to get realistic results Include data sources and assumptions
Surveys	Get feedback from many stakeholders Customize to meet needs Power in analyzing results	Generalizability limited by response rate Can be costly to administer and analyze Results may not be generalizable to intended population	Offer novel incentive to promote response rate Can be administered via email, hardcopy, or in-person
Interviews	Direct contact with stakeholders, promotes engagement Garner individual, detailed data and feedback Identify themes, vignettes, quotes Always follow IRB or institutional guidelines when interviewing stakeholders	Resource-intensive Scheduling Time-consuming to analyze results	Experienced interviewers are best when contacting stakeholders Be prepared for different responses and answers Have pertinent follow-up and probing questions