Chapter 87: Outcomes Research and Reporting

Since the Institute of Medicine (IOM) published its first report in the quality series, To Err is Human, in 1999, there has been increased awareness of the problems that face health care in the United States and an increased emphasis on improving the quality of care our patients receive.¹ Outcomes research, which “studies the end results of medical care—the effect of the health care process on the health and well-being of patients and populations,”² is one type of research used to study, understand, and improve health care quality. Outcomes research differs from more traditional medical research in its focus and endpoints; it seeks to understand the effectiveness of an intervention rather than its efficacy. Ultimately, the results of outcomes research can be used to benchmark performance, reduce adverse events, formulate clinical practice guidelines, and inform health policy decisions.^3,4

In the critical care setting, outcomes research can be particularly impactful for several reasons. First, ICU patients are at a high risk of death and preventable harms due to the acuity of their illnesses and the frequency and complexity of interventions.⁵ Additionally, the costs of ICU care are substantial. ICU patients inhabit only 10% of inpatient beds, but account for almost a quarter of acute care hospital costs.^6,7 In 2005, critical care costs in the United States totaled $81.7 billion per year—4.1% of national health expenditures and 0.66% of the gross national product.⁸ Furthermore, resource utilization varies significantly, but an association between increased expenditure and quality of care has not been clearly elucidated.^9,10,11,12 Therefore, outcomes research in critical care has the potential to make ICU care better, safer, and more cost-effective.

This chapter describes the historical and theoretical basis for outcomes research, methods used by outcomes researchers, endpoints of outcomes research, and common challenges and limitations.

Florence Nightingale and Ernst Codman are often cited as the earliest outcomes researchers—Nightingale for her work studying combat deaths during the Crimean War, and Codman, an early 20th century surgeon at the Massachusetts General Hospital, for advocating hospital reporting of patient outcomes after noting that hospitals routinely reported the number of patients treated, but not the effects of the treatment.^13,14,15 However, no framework for outcomes research existed until 1966, when Avedis Donabedian put forth a model for evaluating health services and the quality of medical care.¹⁶ The Donabedian model suggests three dimensions for assessing health care quality: structure, process, and outcome (Figure 87–1). Structure refers to the characteristics of the setting or environment in which care is delivered, while process focuses on whether appropriate medical practices are utilized. Outcomes range from mortality and length of stay to functional status and quality of life. Although Donabedian recognized the inherent limitations of outcome measures (see the “Limitations” section), he believed that outcomes “remain the ultimate validators of the effectiveness and quality of medical care.”¹⁶

In 1989, the United States Congress acknowledged the importance of outcomes research when it created the Agency for Health Care Policy and Research [later the Agency for Healthcare Research and Quality (AHRQ)] “for the purpose of enhancing the quality, appropriateness, and effectiveness of health care services and access to care.”¹⁷ The Patient-Centered Outcomes Research Institute (PCORI) was established in 2010 as a part of the Patient Protection and Affordable Care Act, further solidifying outcomes research in the American lexicon.

Outcomes Research Versus Traditional Clinical Research

Outcomes research “focuses on the effects of medical care on individuals and society.”⁴ It is this research focus, not necessarily the methodology, that distinguishes outcomes research from traditional clinical research.^4,13,18 Table 87–1 describes the differences between outcomes research and traditional clinical research.⁴ Traditional clinical research is hypothesis-driven, and evaluates efficacy, asking whether an intervention works in an idealized setting. Outcomes research is concerned with effectiveness: Does the intervention help an individual patient in a real-world setting?¹⁹ Additionally, outcomes research is patient-centered, rather than disease-centered, and focuses on “what one ultimately wants health care to achieve,”²⁰ for example, improvements in functional status or quality of life. Furthermore, outcomes research tends to be more inclusive in what it considers an intervention. Traditional clinical research often involves the evaluation of new drugs or devices, while the interventions in outcomes research can range from a new drug to a new structure for health care delivery. In order to achieve its goals, outcomes research tends to utilize observational study designs and draw methods from the social sciences more frequently than traditional clinical research (see the “Methods” section).

Commonly Used Outcome Measures

Mortality

Mortality is perhaps the most tangible and meaningful endpoint. It is easy to define and almost uniformly recorded. It sounds relatively simple: Did the patient survive the acute illness or insult? However, using mortality as an endpoint is actually not so straightforward. One must first determine the appropriate time at which to assess mortality: ICU or hospital discharge? 30, 60, or 90 days? Years? Selecting a timeframe that is too short may provide an inaccurate assessment of the impact of an intervention if the natural history of the condition being studied is longer than the study period. A specific intervention might reduce 30-day mortality, but increase 90-day mortality. Conversely, long-term outcomes may reflect the patient’s disease prognosis, age, premorbid conditions,²¹ and preexisting functional status^22,23 more than the ICU care they received. Therefore, the appropriate mortality endpoint depends on the specific research question, the study design, and the mechanisms of the disease or treatment being studied.⁴

Even when an appropriate mortality endpoint is chosen, outcomes can be artificially affected by patient choice, as well as ICU and hospital practice patterns. For instance, a patient’s decision to transition to comfort care may hasten death; the availability of palliative care suites for transfer of ICU patients receiving comfort measures only alters ICU mortality; and the availability of long-term care facilities for transfer of ventilator-dependent patients could alter hospital mortality. Additionally, comparing mortality rates across different hospitals and ICUs is fraught with problems, largely due to difficulty in adjusting for differences in case-mix.²⁴ Using mortality as an endpoint may not be feasible if the mortality of the condition being studied is relatively low, as the sample size required to generate adequate power to detect a clinically relevant difference will be quite large. And, of course, the use of mortality as an endpoint does not consider morbidities or the quality of life of the survivor.

Health Status

Health status encompasses several measures of patient-assessed outcomes, including functional status and quality of life. Health status is an important endpoint for critical care outcomes research, as recent data has shown that the long-term consequences of critical illness on physical functional status, cognitive function, and quality of life can be profound.^25,26,27,28

Measurement of health status is not standardized, and relies on patient interviews and questionnaires. Physical functional status can be assessed using the 6-minute-walk test, Medical Outcomes Study Short-Form 36 (SF-36),^25,26 the Karnofsky Performance Status Scale score, the Barthel Index, and the Lawton-Instrumental Activities of Daily Living score.^29,30 Tools to assess mental health status, focusing on symptoms of depression, anxiety, and post-traumatic stress disorder, include the Center for Epidemiologic Studies–Depression Scale,³¹ the Hospital Anxiety and Depression Scale,³² and the Impact of Events Scale-Revised.³³ The Informant Questionnaire on Cognitive Decline in the Elderly, the Trail Making Test, and the Repeatable Battery for the Assessment of Neuropsychological Status have been used to determine cognitive function.^27,28,34 Quality of life can be determined from the results of these various tests, or can be assessed specifically via tools such as the EuroQol-5D³⁰ or SF-36.³⁵ Quality of life data can then be used to calculate quality-adjusted life years (QALYs), which take into account both mortality and quality of life.³⁵

Of note, completion of these instruments by ICU survivors themselves is not always possible, and the use of surrogates to complete them may not be as useful. Additionally, interpretation of these tests requires knowledge of the patient’s baseline prior to ICU admission, which may also require a surrogate for determination, and is subject to recall bias. Nonetheless, data is mounting that ICU patients continue to suffer long after their discharge from the hospital: Depression and anxiety may be present in nearly half of ICU survivors,³⁶ and post-traumatic stress disorder in one-third.^32,33 Decline in cognitive function is common and can persist for up to eight years.²⁷ And physical disability is nearly ubiquitous; in one study, 100% of patients reported experiencing subjective weakness and decreased exercise capacity and almost a quarter of patients were unable to return to work at 5 years after discharge from the ICU.²⁵ Thus, continued work on targeting improvement in health status for ICU survivors is paramount.

Cost

Economic evaluation is “the comparative analysis of alternative health care interventions in their relative costs (resource use) and effectiveness (health effects).”³⁷ With the aging of the US population, and the availability of new and more expensive treatments, the cost of critical care is increasing.^38,39 As mentioned earlier, ICU care is disproportionately expensive,^6,7 costing over $80 billion per year in the United States alone.⁸ Therefore, identifying cost-effective interventions in the ICU is incredibly important. Cost-effectiveness analyses (CEAs) produce a ratio in which the numerator is the cost of the intervention, and the denominator is the benefit in terms of the clinical outcome.⁴⁰ The clinical outcome is often reported in terms of QALYs, such that the ratio produced is cost per QALY.

Given the increasing importance of CEAs, and the increasing frequency with which they are performed, both the US Public Health Service and the American Thoracic Society convened panels to address methodological issues and provide recommendations for the reporting of CEAs.^40,41 Their recommendations include describing the model used; identifying model assumptions; describing how estimates of effectiveness, costs, and health states were obtained; and defining the type of costs, year of costs, inflation adjustment methods, and discount rates used. Even when following these recommendations, critical care outcomes researchers face additional challenges that make CEAs more challenging in the ICU setting. These include the complexity of ICU patients, the lack of data on effectiveness of interventions in the ICU, the unavailability of cost data, and the infrequent collection of ideal outcomes measures for CEAs (eg, long-term quality-adjusted survival rates).⁴⁰ The challenge of obtaining accurate cost data can make CEAs difficult to interpret and compare, and limits the generalizability of the results. Charge data (ie, what the patient is billed) are often substituted for cost; however, even when adjusted with cost-per-charge ratios, these data are department and institution specific, and may not reflect actual costs.^37,42 Therefore, interpretation of CEAs must be undertaken with great caution.

Quality Measures

A common focus of outcomes research is quality improvement. Outcome measures for quality improvement research must be “granular enough to be meaningful to clinicians [and to] adequately drive quality improvement interventions.”³ Since preventability is one of the central tenets of quality improvement, outcome measures must also be viewed as preventable.

A recent study by Martinez et al. utilized a consensus process to identify meaningful outcomes measures for quality improvement in the ICU.³ In the study, 164 ICU providers identified five preventable outcomes: pressure ulcers, central line-associated bloodstream infection, pulmonary embolism, methicillin-resistant Staphylococcus aureus infection, and gastrointestinal bleed. Indeed, data support the preventability of these outcomes. For instance, in a landmark study by Pronovost et al., central-line associated bloodstream infections were eradicated in the state of Michigan.⁴³ However, prior to this study, central-line associated bloodstream infections were not uniformly considered preventable. Therefore, although the development and use of outcomes measures for quality improvement must rely on existing data suggesting preventability, clinicians and researchers must also be creative and innovative, continuing to question our current knowledge base of what is preventable.

Methods

Although the focus of outcomes research differs from that of traditional scientific research, the methods used need not. Outcomes research can take the form of case-control studies, cohort studies, and even randomized controlled trials (RCTs). However, outcomes researchers tend to utilize observational study designs and draw methods from the social sciences more frequently than traditional clinical researchers. The use of large administrative datasets is a growing trend in outcomes research. Additionally, outcomes researchers use qualitative methods to generate hypotheses and describe complex phenomena that do not lend themselves to quantitative methods or traditional hypothesis testing.⁴

Observational Studies

Observational studies can be prospective or retrospective, and include cohort studies, case-control studies, and cross-sectional studies.⁴⁴ Such studies seek to identify associations between an exposure (eg, a medication, intervention, or organization of health care delivery) and outcomes, and can utilize primary or secondary data. Primary data are collected to answer a specific research question, while secondary data are data that already exist but are reemployed to answer a novel research question.⁴⁵

Administrative data are a type of secondary data that were originally collected for reasons other than research. Administrative data include health care encounter data, enrollment data, clinical data, data registries, performance data, survey data, and national data.⁴ Examples of data sources utilized in critical care outcomes research include Medicare, the University HealthSystem Consortium, the National Inpatient Sample, and the National Hospital Discharge Database.⁴⁶ There are several benefits to using administrative data for outcomes research. First, administrative data may provide answers to research questions that ethically, legally, or practically cannot be answered by RCTs. Second, large registries and administrative data may be broader in scope and thus more generalizable than primary data. Furthermore, large datasets can more efficiently answer questions associated with rare diseases or outcomes. Finally, policymakers may be more interested in outcomes assessed via administrative data. In fact, policy concerns regarding Medicare spending, racial disparities, and unexplained geographic variation in health care are fueled by the results of analysis of administrative data.⁴⁶

However, the use of administrative data is not without its drawbacks. As with all observational studies, studies using administrative data are subject to bias and confounding. Confounding is of particular concern since patients are not randomly assigned to the exposure of interest; thus, any association between the exposure and outcome could be due to a third, unmeasured variable (see the “Limitations” section). However, the primary concern specific to administrative data is data quality. Unlike traditional clinical research, in which the study design is completed before data collection begins, with administrative data, the quality of the data must be assessed before designing the study (but, of course, after the research question is defined).⁴⁵ The Directory of Clinical Databases in the United Kingdom recently developed a framework for assessing the quality of administrative data.⁴⁷ The framework focuses on data coverage and data accuracy (Table 87–2). Coverage is determined by the representativeness of the data, the completeness of recruitment, the variables included, and the extent of missing variables, while accuracy is determined by the collection of raw data, the definitions and rules utilized, the reliability of coding, the independence of observations, and the method of data validation.⁴⁶

Once the quality of the data has been assessed, the process of research can continue (Figure 87–2). Several aspects of this process are unique to secondary data analyses, and deserve mentioning. First, the analysis plan must be developed a priori in order to maintain the validity of the study. Since secondary data, by definition, already exists, it can be tempting to perform preliminary analyses before finalizing an analysis plan; this temptation should be avoided as it can bias the results. Similarly, “data-dredging” via post-hoc analyses may result in the identification of erroneous, or at least meaningless, associations, since exploring any 20 associations will, on average, produce one result that is statistically significant to P < 0.05.⁴⁵ Thus, the number of statistical tests performed should be minimized. In addition, adjustment for bias and confounding must be performed. However, it is important to remember that even the most sophisticated statistics cannot compensate for poor data quality.

Qualitative Research

Qualitative methods are increasingly being used in outcomes research. Qualitative research uses methods such as interviews, focus groups, field observations, and document review (eg, diaries) in order to “understand complex social processes, organizational change, individual health behaviors, and nuanced aspects of environmental context that influence quality of care, health care delivery, and health outcomes for individuals and populations.”⁴⁸ Qualitative research differs from quantitative research in that it describes the breadth and complexity of a phenomenon rather than measuring occurrences to determine frequency, incidence, prevalence, or magnitude.⁴⁹ Given the complexity and nuanced aspects of critical care, qualitative research can be particularly useful in this setting.

Qualitative methods should be considered when: (1) the phenomena of interest are difficult to measure quantitatively, (2) a comprehensive understanding of a problem is desired, (3) the goal is to generate insight as to why an intervention has a specific impact, and (4) special populations are being studied.⁴⁹ Once collected, qualitative data can be coded and analyzed. Analysis focuses on identifying taxonomies and themes that can explain and predict outcomes.⁵⁰ Although qualitative methods alone can provide interesting and rich data, mixed methods—which combine quantitative and qualitative methods—are even more impactful as they benefit from the strengths of each approach.⁴⁹

Randomized Controlled Trials

RCTs are traditionally designed as efficacy studies—assessing the impact of an intervention in an ideal setting—and are thus most commonly used in conventional clinical research. RCTs can also be utilized in outcomes research if the study design focuses on effectiveness—the implications of an intervention or exposure in a “real-world” setting. Unlike observational studies, RCTs do not have the problem of confounding since, by definition, the exposure is allocated randomly. The lack of confounding makes it easier to conclude causality. However, the use of RCTs in outcomes research is relatively rare for several reasons. First, RCTs are expensive and time-consuming. Second, it may not be legal or ethical to randomize patients to certain exposures or interventions. And, finally, some exposures (eg, socioeconomic status or insurance status) may be impossible to randomize.⁴ Nonetheless, RCTs remain the “gold standard,” and should be considered by outcomes researchers when feasible. Recently, two RCTs of early mobilization in the ICU, designed as effectiveness studies, showed improved functional status for patients who received physical therapy during their critical illness.^51,52

As with all scientific research, critical care outcomes research faces its own set of challenges and limitations. Some of these limitations are methodological—for instance, due to issues with bias and confounding—and are common to outcomes research across all fields of medicine, while some limitations are unique to critical care medicine.

Methodological

The methodological challenges inherent to outcomes research are, obviously, specific to the study design being employed. Herein, we will consider the limitations of observational studies, since a large portion of outcomes researchers rely on observational methods.

The primary methodological limitations of observational studies are bias and confounding. Bias is a systematic error in the design or conduct of a study. There are several different types of bias that can obscure the results of observational studies. Selection bias exists when the patients in one group are somehow fundamentally different from the patients in the other group. Observational bias exists when there are systematic differences in the way in which the data was obtained. Recall and recording bias are types of observational bias. Recall bias frequently comes into play in studies evaluating ICU patient and family experiences, since the patient’s ultimate outcome may affect how they remember their course of care. Recording bias is especially important when using administrative data, since data collected for coding and billing may be affected by efforts to increase reimbursement.⁴⁵

A confounding variable is an extraneous variable that is correlated with the independent variable being studied; confounding exists when the exposed and unexposed groups being studied differ in some way that is also related to the outcome. Randomized controlled trials obviate the concern for confounding since randomizing attempts to ensure the exposed and unexposed groups are similar. However, observational studies are subject to confounding, and when confounding occurs, the effect of the exposure on the outcome will be distorted.⁴⁵ Severity of illness is a common confounder in critical care outcomes research. For instance, consider a comparison of two groups of ICU patients with respiratory failure, where one group received a new mode of mechanical ventilation and the primary outcome measure was ventilator-free days. If the group receiving the intervention was younger and had a significantly higher PaO₂-to-FiO₂ ratio at admission, they may have more ventilator-free days simply because they were less sick.

Unlike bias, which is a systematic error and cannot be fixed, there are five commonly used ways to control for confounding (Table 87–3).^4,53 Matching attempts to identify confounding variables and matches the exposed and unexposed groups based on these variables. Although this technique is relatively simple, the number of confounding variables upon which groups can be matched is limited since an exact match may not be present, and the analysis is limited since matching variables cannot be evaluated as exposure variables. Additionally, overmatching, which occurs when matching for a variable that is not actually a confounder, can decrease the power of the study. Similar to matching, stratification involves the identification of confounding variables and creation of subclasses based on them. Age is a variable that is commonly stratified, for instance 18 to 65 years old and greater than 65 years old. However, stratification becomes unwieldy when attempting to account for multiple confounders as the number of subgroups increases exponentially.

Multivariable adjustment is perhaps the most commonly used method to adjust for confounding. In multivariable adjustment, the researcher creates a model that includes the exposure and outcome of interest, as well as the confounding variables. The model used depends on the outcome being studied—logistic regression is used when the outcome is binary, linear regression when the outcome is linear, and proportional hazards when the outcome is time to an event. These techniques hold the confounding variables constant while assessing the relationship between the exposure and the outcome. Severity of illness scores, such as the Acute Physiology and Chronic Health Evaluation Score (APACHE),⁵⁴ Simplified Acute Physiology Score (SAPS),⁵⁵ or Mortality Prediction Model Score (MPM),⁵⁶ is commonly included in multivariate analyses in an attempt to account for how sick patients are on admission to the ICU. These scoring systems take into account a large number of variables and provide a single number. However, one must remember that such scores may not capture all pre-existing comorbidities nor pre-existing frailty, which may have a large impact on the patient’s clinical trajectory.⁵⁷ Furthermore, multivariate models are not without their pitfalls. Careful attention to which variables are included in the final model, and rigorous testing of model fit are required to avoid erroneous conclusions.

Propensity scores and instrumental variables are two more sophisticated methods of adjusting for confounding. Like severity of illness scores, propensity scores combine multiple variables to create a single value. This value represents a patient’s probability of being exposed to the variable of interest.⁵⁸ Patients can then be matched or stratified based on their propensity score, or the propensity score can be included in the multivariate model. The use of propensity scores is increasingly common as they are a powerful tool to decrease the effects of confounding. Instrumental variables, like propensity scores, are associated with the exposure, but are selected because they are independent of outcomes. If an outcome is then analyzed by the instrumental variable, a random distribution of patients is created, which simulates randomization to the exposure. Although enticing, the use of instrumental variables is relatively uncommon, largely because it is difficult to identify variables that do not correlate with the outcome being studied.

Unique to Critical Care

In addition to the methodological limitations associated with observational outcomes research discussed above, ICU outcomes researchers face a unique set of challenges, largely due to the breadth of patients, diseases, therapies, providers, and health care delivery models associated with critical care medicine. Performing research requires identifying a specific question and defining the disease, treatment, patient population, or provider to study. Defining such variables in the ICU can be complex.⁴ Disease states, such as sepsis or respiratory failure, may be exceedingly broad and lack exact diagnostic criteria or accepted treatment. The patient population is highly variable, including medical, surgical, cardiac, and neurological patients. And the physicians caring for patients range from pulmonologists and cardiologists to anesthesiologists, surgeons, and surgical subspecialists. Therefore, “critical care” does not have an explicit definition in the literature. Although critical care is often defined geographically as patients cared for in an ICU, this is increasingly problematic as the chronically critically ill may be cared for outside the ICU, and some patients in the ICU may not actually be in critical condition.⁴

Furthermore, as mentioned above, critically ill patients may not be able to consent to or participate fully in research due to the severity of their illness, administration of sedative medications, or use of interventions such as mechanical ventilation that decrease their ability to communicate.

Outcomes research studies the end result of medical care on patients and society. Outcomes research in critical care is particularly important due to the severity of illness of ICU patients and complexity of the care they receive, as well as the overwhelming cost of ICU care. Critical care outcomes researchers often utilize observational study designs influenced by the social sciences, and focus on patient-centered outcomes such as mortality, functional status, and quality of life. The limitations of observational study designs can be addressed by several statistical methods. Continued emphasis on critical care outcomes research is of paramount importance as we attempt to understand and improve the long-term outcomes of survivors of critical illness.