̽��app

Introduction

Participants and other stakeholders in today’s U.S. health care system are striving for the generation of new knowledge to guide care while at the same time they are also managing growing clinical and organizational complexity, directing considerable attention to curbing health care costs, and reducing inefficiencies. An important early milestone toward achieving these goals was the establishment of the Center for Medicare & Medicaid Innovation (CMMI) following the passage of the Patient Protection and Affordable Care Act (ACA) of 2010 to promote innovations in care delivery that fulfill the Triple Aim of improving health and health care at lower cost. In January 2015 the U.S. Department of Health and Human Services established the Health Care Payment Learning & Action Network (HCP-LAN, or LAN) to work in concert with partners in the private, public, and nonprofit sectors to transform the nation’s health system to emphasize value over volume by supporting and advancing the adoption of value-based payment and alternative payment models. Passage in 2015 of the Medicare Access and CHIP [Children’s Health Insurance Program] Reauthorization Act (MACRA) with strong bipartisan, bicameral support in Congress and from the White House set America’s health care system on a path to reform that emphasizes value over volume in federal payment programs. On November 2, 2016, the Centers for Medicare & Medicaid Services (CMS) issued the Final Policy, Payment, and Quality Provisions in the Medicare Physician Fee Schedule for Calendar Year (CY) 2017 (the Rule) (CMS, 2016). The Rule implements provisions in MACRA intended to promote quality, improvement activities, accessing care information, and attention to cost. These initiatives set the stage for broad engagement of patients and providers in the learning health system.

Beyond payment reform emphasizing health care value, opportunities to address the challenges facing modern health care can be found within a continuously learning health system, a bidirectional approach to learning in which the care delivery process creates new knowledge and care is adapted in response to knowledge generated. The Institute of Medicine (IOM, 2013) report Best Care at Lower Cost: The Path to Continuously Learning Health Care in America laid out a vision for a continuously learning health system, stating that: “The committee believes that achieving a learning health care system—one in which science and informatics, patient-clinician partnerships, incentives, and culture are aligned to promote and enable continuous and real-time improvement in both the effectiveness and efficiency of care—is both necessary and possible for the nation.” As shown by this vision (see Box 1) a continuously learning health system offers the promise of creating new knowledge, returning data to clinicians and patients to guide care, making progress in how person-centered care is delivered, reducing costs, and creating value for those served.

However, we assert that realization of the full potential of the continuously learning health system requires more active engagement of front-line clinicians (defined here as providers involved in day-to-day patient care interactions). Instead of the compartmentalized approach to knowledge generation, in which research that informs the delivery of care is separate from the care experience, we contend that future knowledge generation will be best accomplished with fully engaged clinicians, patients, and health care data.��Understanding that clinical situations are complex and require sophisticated judgment, we believe the goal of knowledge generation is to provide information to help inform decision making. Fostering the engagement and leveraging the insights of front-line clinicians in knowledge-generating activities will drive a continuously learning health system toward outcomes that are most relevant, easily translated, and valuable to clinical practice and patients.

Even more so, we believe that it is the ethical, professional, and intellectual responsibility of members of the clinical team to generate knowledge and improve care; and that the interest and commitment to do so by clinical teams exist. The time is right for the health care research and clinical ecosystems to remove the structural barriers separating them—including economic impediments to learning, misalignment of or lack of incentives, and countervailing clinical pressures—that currently inhibit clinician involvement and leadership within a continuously learning and transforming health system.

This paper explores how clinicians can advance and benefit from a continuously learning health system. We describe the potential and importance of engaging clinicians in knowledge generation; describe the challenges and strategies for aligning priorities between clinicians and researchers and creating active partnerships in the design and conduct of learning activities; and finally explore impediments to data collection at the point of care, potential facilitative approaches, and strategies for creating a knowledge-generating infrastructure attractive to and supportive of front-line clinicians. To begin a proactive dialogue, we identified priorities for action designed to demonstrate a potential path for creating a knowledge-generating infrastructure beneficial to clinicians, and ultimately, to their patients.

Why Now?

The delivery of care offers many opportunities, incentives, and challenges for front-line clinicians. Clinicians must balance increasing expectations for higher quality, performance measurement, and knowledge generation with the daily realities of their clinical practice.��While most clinicians still operate under a fee-for-service payment model, there is a growing emphasis on shifting payment and care from volume-based care to value-based care. CMS has already surpassed its goal of having 30% of care supported through value-based payment by the end of 2016 (with a goal of 50% by 2019–2020) (Burwell, 2015). Delivery systems increasingly participate in value-based programs such as accountable care organizations, assume upside and downside financial risk for the care they deliver, and are rewarded for effective population management.

There is a suite of novel research approaches that align with operations (Abraham et al., 2016; NIH Collaboratory, 2016; PCORnet, 2016) and emerging opportunities for researchers to be embedded within delivery systems (AcademyHealth, 2016). Meanwhile, clinical training programs are striving to address the complex and changing health care environment by moving away from traditional teaching models, which focus primarily on the basic sciences of biology and chemistry, toward ones that promote health services research, learning health systems, data sciences, and team-based care. This rapid change is requiring many medical schools to retrain their faculty and restructure curricula to meet the demands of an evolving health environment. Additionally, regulatory policies are encouraging greater measurement of performance and the use of electronic health record (EHR) systems.
For example, the CMS Quality Payment Program established to implement MACRA offers two payment incentive paths for providers to choose from: (1) the Merit-based Incentive Payment System (MIPS) and (2) the advanced Alternative Payment Model (APM) option, both of which measure performance and tie payments to performance on quality and other measures.

One of the defining elements of the learning health system is a culture of inquiry that encourages discovery and experimentation. While this culture may be nurtured and indeed expected and rewarded by the research and quality improvement communities, there is less opportunity, funding, time, or precedent for clinician participation to be similarly encouraged and rewarded. We contend that it is not for a lack of interest among front-line clinicians; indeed it is clinicians—those who interact daily with the patients they serve and experience diagnostic and therapeutic gaps—who may be first to recognize opportunities to improve the care that they deliver and to engage in and learn about new discoveries. However, the ecosystem as it currently exists has environmental impediments to clinician engagement in the learning health system.

Obstacles to Clinician Engagement in the Learning Health System

Productivity Pressure

Within the current health ecosystem, patient care and related documentation leave little breathing space for clinicians to engage in evidence-generation activities. Clinicians endeavor to meet the goals of the Triple Aim (IHI, 2016) while facing swelling pressures, regulations, reporting requirements, aging populations, higher pharmaceutical costs, misaligned incentives, and additional multiple stressors. Despite growing demands, they do not have adequate and timely access to the information required to respond to all that they are asked to do. It has been estimated that provision of all indicated preventive care services would require 7.4 hours per working day (Yarnall et al., 2003).

A 2014 survey by Mayo Clinic and American Medical Association investigators revealed that the prevalence of physician burnout is on the rise as compared with 2011 and continues to impact almost all specialties (Shanafelt et al., 2015). A recent time-motion study of clinicians in four specialties reported that for every minute spent in direct face-to-face contact with patients, two minutes was spent on average interacting with the EHR systems (Sinsky et al., 2016). Appointment times with patients are decreasing and time once available for reflection is increasingly consumed by administrative and clinical demands. Even when there are opportunities for clinicians to participate in knowledge generation, the infrastructure to support their involvement is often limited. Clinicians report grappling with uncoordinated measurement, documentation, payment, and other requirements (IOM, 2015).

Lack of Alignment to Prioritize Potential Learning Opportunities

The coordination that is essential to fully capturing and properly prioritizing the potential learning opportunities associated with problems as they are recognized by clinicians is currently lacking. Funding sources, incentives, data needs, and priorities of potential contributors to knowledge generation, including clinicians, researchers, and quality assessors, are misaligned.��Specific examples of these mismatches include:

• Clinicians have limited incentive to participate in research in the current paradigm.
• Researchers and clinicians are paid differently and rewarded differently for their professional activities.
• Activities, such as the translation of new knowledge to local standards of care that requires communication across teams of clinicians, are not generally compensated in the new value-based system.
• Clinicians are not routinely compensated for the additional time that is required to document clinical activities in an EHR system. This time is in addition to the clinician time that is still, for the most part, dedicated to billable activities (Sinsky et al., 2014).
• Clinical research focused on opportunities to reduce health care service utilization may negatively impact an organization’s bottom line and may therefore indirectly reduce the resources available for non-clinical care activities such as research.
• Requirements to document patient care and submit bills for payment are not always coordinated with documentation requirements for new quality metrics and/or research studies.
• Institutional recognition and advancement, within a non-academic medical center, may depend more on clinical productivity than on participation in research.
• Clinician access to the data relevant to their patients is lacking. In 2015, 78% of clinicians were using certified electronic health technology in their offices (Jamoom and Yang, 2016). However, most systems were primarily designed to meet administrative requirements associated with payment. The data within these systems are often not easy to access for the clinician, which limits optimal use of the rich data being entered for continuous learning. The relevance, timeliness, and value of feedback based on entered data are limited (Zulman et al., 2016). Excessively frequent feedback in the form of low-value electronic alerts has been shown to result in alert fatigue and clinician overrides (Isaac et al., 2009).
• Clinicians may view acceptance of an invitation to contribute to randomized clinical trials as potentially adding to their workload without rewards, sufficient reimbursement, or credit. Participation at one of many enrolling sites in a large multi-center clinical trial does not guarantee authorship and promotions for clinicians and is often not financially sustainable.
• The historical grant-funded research model has typically supported highly specialized research, but research questions have not focused on solutions to major clinical problems.

Lack of Technical Competencies

No one has all of the necessary training, competencies, opportunities to collaborate, or cross-discipline education to facilitate learning activities in this rapidly evolving ecosystem. Clinicians are essential, but high-quality learning requires a multi-disciplinary team. Depending on the problem, that might require participation of delivery science experts, statisticians, information technology staff, and research coordinators.

Priorities for Action

The priorities for action, which are articulated below (see Box 2), will require commitment by multiple stakeholders to transform how clinicians engage with, are rewarded for, benefit from, and are included in knowledge-generation initiatives. The charge to create new evidence that improves clinical care does not belong solely in the realm of research. We believe that clinician engagement in the learning health system is an issue of professionalism; it is the intellectual responsibility for members of the clinical team—those people involved in the day-to-day practice of health care delivery—to generate knowledge and improve care (AMA, 2016). The American Medical Association Code of Medical Ethics Principles V and VII speak directly to this responsibility:

V. A physician shall continue to study, apply, and advance scientific knowledge, maintain a commitment to medical education, make relevant information available to patients, colleagues, and the public, obtain consultation, and use the talents of other health professionals when indicated.
VII. A physician shall recognize a responsibility to participate in activities contributing to the improvement of the community and the betterment of public health.

The interest and commitment by clinicians exist. Clinicians are drawn to health care for the purposes of providing quality care and finding out what works. We contend that the conduct of knowledge-generation activities can and should align with and advance these goals and should occur within the delivery setting. It is more important than ever that the health care ecosystem identify and address existing structural barriers that limit clinician involvement and leadership in a continuously learning and transforming health system.��The following identified priorities for action are intended to describe a potential path for creating a knowledge-generating infrastructure that engages and benefits clinicians and their patients.

Create Incentives for Clinician Engagement

Align Priorities

The alignment of clinical research and quality improvement priorities for new knowledge generation is an essential early step toward the greater engagement of clinicians in learning activities. Just as the Patient-Centered Outcomes Research Institute (PCORI) requires investigators to engage patients as research proposals are first being developed, we envision similar early engagement of clinicians in learning health system research. As described above, clinicians operate in an environment of numerous competing priorities. Successfully making the case that clinicians have a stake in knowledge generation is more likely if the starting point is a jointly developed set of key questions and study aims. In all research and quality assessment activities (as demonstrated in Box 3) there should be the opportunity, and the expectation, that the topics to be studied will address pressing questions that concern clinicians and the patients they serve. Some priorities might cut across the clinical professions or engage clinicians practicing multiple specialties, while others may be specific to a particular clinical condition, site of care (e.g., inpatient; ambulatory), or an individual institution.

Engage Clinicians as Active Partners in the Design and Conduct of Learning Activities

In addition to clinician engagement in the priorities-setting stage, there is an important role for clinicians in designing data collection methodologies, especially for the aspects of knowledge generation in which they are essential, including determining which of their patients are appropriate candidates for learning activities, and, when indicated, introducing a study to their patients, collecting data at the point of care, and disseminating results.

Allow Engagement in Knowledge Generation to Satisfy Existing Professional Obligations

In order to incentivize clinician engagement in a continuously learning health system, clinical research data and quality improvement data collection should be embedded into the care delivery process and information should be gathered to serve multiple goals.��Therefore, knowledge-generating initiatives should take into account existing reporting requirements and programs and reward the time and effort spent on learning activities to fulfill other requirements (e.g., providing continuing medical education [CME] credit, contributing to specialty certification, satisfying MACRA, and similar obligations) (see Box 4). They should also consider other ways the learning activities could provide additional incentives such as co-authorship or acknowledgment (based on their roles) in publications and research proposals.

Generate Actionable, Timely, and Relevant Knowledge

In addition to financial resources, another important incentive to clinicians is the learning activity. Health care clinician teams are committed to providing better care and learning and knowing which interventions will work best for their patients. More productive partnerships among researcher, quality improvement experts, and clinicians are more likely to occur if the generation of actionable knowledge is the likely end result of their joint efforts. Partnerships will benefit from there being a direct line of sight between the proposed research and the potential benefit to patients.

Pragmatic clinical trials that adapt patient allocation based on interim results may have particular appeal to clinicians who are accustomed to modifying diagnostic and therapeutic approaches in response to interim results during an episode of care. Too often the analysis of study data is conducted outside of the clinical environment and is not made visible early to the front-line clinicians and institutions caring for study participants. Time lags from when the first patient is recruited to when data analysis is complete and the results disseminated via peer-reviewed literature lessen the direct benefits to the original data contributors or care system. Failure to publish the results of studies involving patients is a growing concern among funders of research and journal editors (Gordon et al., 2013).��To the growing list of those impacted by failure to publish, most importantly patients, we would also add the clinicians who recruit and care for the participants in those studies.

Clinicians—as members of research and quality improvement (QI) teams or as data contributors or when referring their patients for studies—need to receive something that improves how they care for their patients in return for their participation. This requires more than sharing new knowledge with them. Potential added benefits include more rapid implementation of positive results, comparative data to document their own outcomes and identify opportunities for improvement, and the great experience using, albeit slowly, EHR systems as tools for learning. To begin to address the misalignment of value-based incentives, efforts should be focused on reframing the concept of “findings” so that researchers and quality assessors work in collaboration with clinicians to learn what results from the data (e.g., risk and outcomes profiles, population-based data on their patients) would be useful and could be shared with them prior to the publication of a paper or a report. As an example, in an effort to make “evidence” more immediate and actionable for clinicians, the Function and Outcomes Research for Comparative Effectiveness in Total Joint Replacement and Quality Improvement (FORCE-TJR) research cohort assures that data submitted from consenting patients secondarily meet bundled payment program requirements (see Box 5) and return point of care evidence to inform care decisions (Franklin et al., 2012). After data capture, researchers provided clinicians and patients immediate data to guide individual care based on both aggregate local and national evidence as well as aggregate data to meet MIPS/APM criteria.

Address Productivity Pressure

Minimize the Competing Demands Placed on Clinicians and Embed Knowledge Generation into Work Flow

To realistically engage clinicians as active partners, knowledge-generation activities will need to exist, and be supported, within the clinical care environment. The priorities are to engage clinicians in setting priorities, engage their own patients when necessary, interpret results, and implement better practices that emerge from the collective data. These activities should not place anything other than a minimal additional burden on practicing clinicians. For example, data collection for learning activities should be integrated in the clinic in ways that complement existing work flow. Additionally, data collection could be completed by study and quality improvement personnel, thus limiting the expectation and strain to clinical teams. Finally, structured data collected for patient care and billing should be secondarily available in integrated data warehouses for analyses in research and quality improvement activities.��To effectively embed knowledge generation into the work flow will require that health delivery organizations provide tools and support for their clinicians and help develop a culture that encourages both evidence generation and consistent evidence implementation.

Address the Misalignment in Financial Compensation

Clinicians engaged in full-time or almost full-time patient care are currently compensated based primarily on the number of patients seen or procedures performed. Although MACRA implementation began on January 1, 2017, the value-based payment adjustments described in the Rule for work done in 2017 will not take effect until 2019. As a result, any clinically appropriate reduction in volume will result in lower practice revenue during the year in which services were or were not provided and any positive payment adjustment realized 2 years later. Clinician practices striving to reduce unnecessary or rarely appropriate services and implementing population health strategies may therefore not have the financial resources to become meaningfully engaged in uncompensated health system research. To begin to address the misalignment in funding we need to:

• Focus efforts on reframing research proposals so that full-time researchers or health systems budget to at least partially compensate front-line clinicians for their time away from patient care and research
  staff to support clinicians in research activities; and
• Explore a MACRA equivalent for clinical trials enabling and reversing the outflow of clinical trials.

Given the increasing pressure on physician time and effort, new reimbursement structures should be put in place to reward those who contribute time to a learning health care system through the volume of care delivery as opposed to the integration of care and research through clinical trials.

Address technical competencies

Incorporate the competencies of continuous learning within medical and graduate school curricula

As we seek to create an ecosystem in which clinicians contribute to quality improvement, engage in research, work in care teams, improve population health, and accept greater accountability for outcomes of care, we have an opportunity to ask how medical schools and clinical training programs can promote the competencies clinicians will need if they are to become full participants in a learning health system (AHRQ, 2014).

To this point, we believe that additional exploration is critical to identify:

• The specific competencies needed for pre-medical and current medical students and practicing clinicians.��For example, a recent Association of American Medical Colleges (AAMC) survey recommended inclusion of biostatistics and epidemiology courses in pre-medical curricula to prepare clinicians to interpret formal research analyses and clinic-based aggregate data (Marantz et al., 2003). Today’s population health curricula are too often limited to analyses of formal research trials and statistical output and do not address the analytic tools of learning health systems such as propensity analyses of existing data, statistical process control and trend
  analyses of practice patterns, or individual prediction models for precision medicine.
• Effective strategies for articulating the value of new competencies to medical school leaders.
• The potential value of new tools and/or regulations to help incentivize training in the research enterprise, such as required toolkits that young investigators would need for professional success or to meet certifying board requirements for comparative effectiveness research (CER) and/or QI training during residency and/or fellowship training.
• Exemplar clinical programs and their rates of return. Schools and delivery systems at the forefront of clinician-researcher collaborations in a learning health system should be identified and channels established to share widely their lessons learned and strategies used such as:
  • Opportunities such as 1 year of dedicated time during which clinicians engage in research, post-clinical fellowships, and/or physician assistant and medical student shared learning.
  • Curricula focused on embedded training in topics such as data science, trial design, QI, and CER. For example, within the current health environment, students would benefit from learning how to generate new effectiveness data in the course of practice that can be tailored to individual patient risks and inform care and understanding the measurement principles of process and outcomes of care.
  • Change and implementation principles for clinicians to foster the adoption of CER/best practices in the local delivery system.
  • Training in team-based care and research activities.

Conclusion

A fully realized learning health system needs clinicians to assume an active leadership role. Much of our societal effort to advance a learning health system has addressed other sectors, including health system leaders, payers, purchasers, patients, and the public, as well as technical capabilities to allow the secondary use of electronic health data. Consideration of clinicians’ roles has concentrated largely on identifying ways to avoid or minimize their active participation. This approach has constrained progress because many of the most important challenges and opportunities depend on clinicians’ active participation in a wide array of activities, including practice improvement, quality initiatives, and hypothesis-driven research, including clinical trials. There is abundant evidence that many clinicians would choose to participate in many research activities. As noted above, the reasons this occurs so rarely are not subtle. They are also addressable, even in the context of the demands of contemporary medical practice.

Our most important emphasis is to engage clinicians directly in priority setting for the research and other learning in which they will participate. This approach has obvious relevance for activities that inform immediate clinical practice. We believe many clinicians will also choose to participate in research that may have a longer lead time to affect their practice, if they see its relevance to their patients’ unmet needs. Additional ways to provide positive incentives to participate are to ensure that information derived from trials, registries, and other structured learning flows back to clinicians regularly.

However, it is also important to offload some current obligations of clinicians. We believe there are substantial opportunities for structured participation in learning health system activities to satisfy existing requirements. Awarding CME credit is a simple way to do this. More importantly there is currently an opportunity for these valuable and professionally rewarding activities to be structured to satisfy meaningful use requirements and the evolving requirements of MACRA.��In addition to creating the ability of high-quality learning activities to satisfy existing requirements, we note the importance of creating roles for clinicians that take advantage of their unique functions in delivery systems. In addition to priority setting, they are often the only ones who can evaluate their patients’ needs and they are the ones who have the rapport and credibility to introduce novel care options to their patients.��We should design activities to take advantage of these abilities, but not require clinicians to perform other tasks, including consenting and data collection, when other professionals can complete them. We also fully
support providing direct financial compensation for clinician engagement whenever that is possible.

Finally, we should create opportunities for clinicians at all levels of professional development to acquire expertise about the ways in which knowledge development can be incorporated directly into clinical practice.��The emergence of pragmatic clinical trials, cluster randomization, step wedge introduction of policies and practices, and quasi-experimental designs as widely applicable methods to understand and improve clinical care is relatively recent. To the extent that clinicians understand these methods, even at a conceptual level, they will be more effective partners in advancing the health of their patients. Teaching these methods in medical schools and post-graduate programs will be valuable; they can also be introduced in an array of CME formats.

We recognize that there are formidable barriers to clinicians’ active engagement in learning health system activities. We also see a clear path to increasing such engagement in a manner that clinicians will see as improving both their day-to-day professional lives as well as advancing their ability to provide better care.

Join the conversation!

• Active engagement of front-line clinicians in the learning health system means better outcomes for patients: http://bit.ly/2ld5Opp
• @the̽��appedicine presents 3 action priorities to transform how clinicians engage with the learning health system: http://bit.ly/2ld5Opp
• A learning health system needs clinicians in leadership roles. Aligning priorities to make this possible: http://bit.ly/2ld5Opp

Download the graphic below and share it on social media!

References

1. Abraham, E., C. Blanco, C. Castillo Lee, J. B. Christian, N. Kass, E. B. Larson, M. Mazumdar, S. Morain, K. M. Newton, A. Ommaya, B. Patrick-Lake, R. Platt, J. Steiner, M. Zirkle, and M. Hamilton Lopez. 2016. Generating Knowledge from Best Care: Advancing the Continuously Learning Health System.��̽��app Perspectives. Discussion Paper, National Academy of Medicine, Washington, DC.��https://doi.org/10.31478/201609b
2. AcademyHealth. 2016. Delivery system science fellowship. Available at: www.academyhealth.org/dssf (accessed January 13, 2017).
3. AHRQ (Agency for Healthcare Research and Quality). 2014. Improving cultural competence to reduce health disparities for priority populations. Rockville, MD: AHRQ. Available at: https://effectivehealthcare.ahrq.gov/products/cultural-competence/research-protocol (accessed August 14, 2020).
4. AMA (American Medical Association). 2016. Code of medical ethics. Available at: https://www.ama-assn.org/about-us/code-medical-ethics (accessed November 10, 2016).
5. Burwell, S. M. 2015. Setting value-based payment goals—HHS efforts to improve U.S. health care. New England Journal of Medicine 372(10):897-899. https://doi.org/10.1056/NEJMp1500445
6. CMS (Centers for Medicare & Medicaid Services). 2016. Final policy, payment, and quality provisions in the Medicare Physician Fee Schedule for Calendar Year (CY) 2017. Available at: https://www.cms.gov/Newsroom/MediaReleaseDatabase/Fact-sheets/2016-Fact-sheetsitems/2016-11-02.html (accessed November 2, 2016).
7. Franklin, P. D., J. J. Allison, and D. C. Ayers. 2012. Beyond joint implant registries: A patient centered research consortium for comparative effectiveness in total joint replacement. JAMA 308(12):1217-1218. https://doi.org/10.1001/jama.2012.12568
8. Gordon, D., W. Taddei-Peters, A. Mascette, M. Antman, P. G. Kaufmann, and M. S. Lauer. 2013. Publication of trials funded by the National Heart, Lung, and Blood Institute. New England Journal of Medicine 369(20):1926-1934. https://doi.org/10.1056/NEJMsa1300237
9. IHI (Institute for Healthcare Improvement). 2016. The IHI triple aim initiative: Better care for individuals, better health for populations, and lower per capita costs. Available at: http://www.ihi.org/engage/initiatives/tripleaim/pages/default.aspx (accessed November 11, 2016).
10. Institute of Medicine. 2013.��Best Care at Lower Cost: The Path to Continuously Learning Health Care in America. Washington, DC: The National Academies Press. https://doi.org/10.17226/13444
11. Institute of Medicine. 2015.��Vital Signs: Core Metrics for Health and Health Care Progress. Washington, DC: The National Academies Press. https://doi.org/10.17226/19402
12. Isaac, T., J. S. Weissman, R. B. Davis, and et al. 2009. Overrides of medication alerts in ambulatory care. Archives of Internal Medicine 169(3):305-311. https://doi.org/10.1001/archinternmed.2008.551
13. Jamoom, E., and Yang, N. 2016. Table of electronic health record adoption and use among office-based physicians in the U.S., by state: 2015 National Electronic Health Records Survey. Available at: http://www.cdc.gov/nchs/data/ahcd/nehrs/2015_nehrs_web_table.pdf. (accessed November 10, 2016).
14. Marantz, P. R., W. Burton, and P. Steiner-Grossman. 2003. Using the case-discussion method to teach epidemiology and biostatistics. Academic Medicine 78(4):365-371. https://doi.org/10.1097/00001888-200304000-00008
15. NIH (National Institutes of Health) Collaboratory. 2016. NIH Collaboratory. Available at: https://www.nihcollaboratory.org/Pages/default.aspx (accessed January 13, 2017).
16. PCORnet (National Patient-Centered Clinical Research Network). 2016. Patient-Centered Network of Learning Health Systems (LHSNet). Available at: http://pcornet.org/clinical-data-research-networks/patient-centerednetwork-of-learning-health-systems-lhsnet-phase-ii (accessed January 13, 2017).
17. Shanafelt, T. D., O. Hasan, L. N. Dyrbye, C. Sinsky, D. Satele, J. Sloan, and C. P. West. 2015. Changes in burnout and satisfaction with work-life balance in physicians and the general-US working population-between 2011 and 2014. Mayo Clinic Proceedings 90(12):1600-1613. https://doi.org/10.1016/j.mayocp.2015.08.023
18. Sinsky, C. A., J. W. Beasley, G. E. Simmons, and R. J. Baron. 2014. Electronic health records: Design, implementation, and policy for higher-value primary care. Annals of Internal Medicine 160(10):727-728. https://doi.org/10.7326/M13-2589
19. Sinsky, C., L. Colligan, L. Li, M. Prgomet, S. Reynolds, L. Goeders, J. Westbrook, M. Tutty, and G. Blike. 2016. Allocation of physician time in ambulatory practice: A time and motion study in 4 specialties. Annals of Internal Medicine 165(11):753-760. https://doi.org/10.7326/M16-0961
20. Yarnall, K. S. H., K. I. Pollak, T. Østbye, M. A. Krause, and J. L. Michener. 2003. Primary care: Is there enough time for prevention. American Journal of Public Health 93(4):635-641. https://doi.org/10.2105/AJPH.93.4.635
21. Zulman, D. M., N. H. Shah, and A. Verghese. 2016. Evolutionary pressures on the electronic health record: Caring for complexity. JAMA 316(9):923-924. https://doi.org/10.1001/jama.2016.9538