Chapter 63: Controversies: Scoring Systems in Critical Care

Management of the critically ill patient is complex. The acuity of illness, multiple organ system derangements, patient heterogeneities such as age, comorbidities, uncharted genomic and epigenetic variability, and the availability of evolving treatments and systems of health care delivery make the prediction of prognosis an elusive, moving target.

Originally created in response to limited ICU resources, an aging population and rising healthcare costs, scoring systems^1,2,3 (also called scoring models) are intended to provide objective predictions or probabilities of mortality and long-term outcome to aid clinical decision making and planning. Their role has been expanded to benchmark or compare the performance of ICUs and assess the quality of care provided (eg, observed mortality vs predicted mortality), to predict the resources needed or the appropriateness of ICU admission depending on the severity of illness and finally to assess patients for inclusion in research studies or to compare severity of illness and assess case-mix dissimilarities between patients in various study groups.

Although intensivists will increasingly encounter assessments of their ICU patient population, ICU structure and process and physician performance via scoring systems, the application of scoring systems is not without significant controversy. This chapter discusses the main ICU scoring systems in use today,^4,5,6,7 how their performance is assessed, how they are utilized, the limitations and the controversial issues surrounding their use.

The type, timing, and quantity of data collected vary significantly among the numerous scoring systems. Some scores, such as the sequential organ failure assessment, measure the organ dysfunction and thus the severity of the disease at any point of time to monitor the clinical evolution. Others, known as general risk prediction scores focus primarily on survival. These systems were developed on the assumption that the severity of illness and mortality are related to acute physiologic derangements that appear early in the course of the disease.⁸ The most prevalent are acute physiology score chronic health evaluation (APACHE), mortality probability model (MPM), and simplified acute physiology score (SAPS). They were developed approximately 30 years ago and have since undergone 3 to 4 revisions (Table 63–1). APACHE IV combines a total of 27 variables collected within the first 24 hours of ICU admission, MPM₀III and SAPS III combines, respectively, 16 and 20 variables collected within 1 hour of ICU admission. They all depend on acute physiologic variables collected on admission such as vital signs, electrolytes, cell count, blood gas, etc, in addition to chronic health variables and admission diagnosis.

Scoring systems provide a numerical estimation (score) to quantify the severity of illness, which is correlated to mortality, length of stay (LOS), morbidity, and long-term functional outcome (ie, quality of life). Prognostic models are constructed using prospective cohorts of patients with predefined candidate variables, outcomes and time points. Candidate variables are preselected variables, such as any patient characteristics, lab values or risk factors that are thought to play a role in the prediction of the model. A statistical analysis method, called logistic regression, will then identify and weight the variables that can predict the outcome. Although the original versions of the APACHE and SAPS scoring systems used expert opinions to determine the predictive variables, more recent versions use multiple logistic regression techniques (logistic and Cox regression) to also select and weight the variables.

The performance or ability to predict the outcome, is assessed by comparing the model predictions of mortality risk to the actual mortality in the studied patient populations across different ICUs. The measures used to evaluate outcome prediction include discrimination and calibration. Discrimination is the capability to distinguish between patients who will develop the event of interest (eg, death) versus those who will not (eg, survival).⁹ Discrimination is represented by the area under the receiver operating characteristic curve (AUROC) (Figure 63–1), where the true positive rate is plotted against the false positive rate. The greater the area under the curve, the more accurate the system is able to discriminate between a given outcome, for example, alive or dead. Calibration depicts how the model performs over a wide range of predicted mortality; it estimates the scoring system’s accuracy in mortality prediction over a wide spectrum of severity of illness, usually in deciles from low to high (Figure 63–2).⁹ For example, scoring systems tend to underestimate mortality in patients with a high severity of illness and underestimate mortality in patients with a low severity of illness.

The predictive performance of models must be validated.¹⁰ Internal validation refers to the analysis of the performance of the model in the ICU dataset population from which the model is derived; this is the population in which it will perform the best because the variables were adjusted to optimize the model. This is followed by a process called external validation in which the performance of the model is assessed in patients from ICUs that were not part of the original dataset used to develop the model. The ICUs used for external validation should be of the same type (eg, general ICU vs general ICU not general ICU vs specialty ICU [eg, cardiac]) as those used in the internal validation.

Although they were introduced more than 30 years ago, scoring systems have been used in only 10% to 15% of the total yearly ICU admissions in the United States.¹¹ This lack of clinical utility at the bedside can be explained by the lack of predictive applicability in the individual patient, the limited practical utility of the models (too complex or cumbersome to implement) and the costs. To be used in clinical practice, the model should be simple, readily available and not expensive. The least expensive and simplest scoring systems to implement have been the most prevalent. For example, the APACHE II system—no cost, in the public domain—is today the most used system to assess patients for inclusion in research studies based on severity of illness or to compare severity of illness between control and treatment groups and assess case mix dissimilarities between patients in various study groups. APACHE II is also used in many ICUs to assess predicted mortality compared to observed mortality, thus as a quality measure of ICU performance to national norms. However, the APACHE II system was created and validated 30 years ago. Ultimately, prognostic scoring systems become obsolete over time if they are not updated to follow the advancement of medical sciences. Due to improvements in medical care and in population health, the mortality predictions of older systems tend to overestimate the current mortality rate. This trend is obvious in inexpensive systems which are almost guaranteed to provide a good result—most ICUs, using APACHE II, will have a lower than predicted mortality and thus favorable benchmarking.

It is important to understand the different components of APACHE IV, which is the paradigm of a more modern system that allows predictions of both hospital mortality and expected ICU LOS. APACHE IV is characterized by an internal validation on a very large database with selected variables determined by multiple logistic regression—a more rigorous statistical method than arbitrary expert opinions. There are 2 versions, 1 in the public domain that is essentially a web-based calculator; 1 enters the patient variables and quickly generates a score that correlates to a predicted mortality. The other version is managed by a company, Cerner (Vienna, VA), which maintains the database and analyzes the APACHE IV data across different hospitals, providing updated and repeated analysis over time with better resolution and potential value. Because the accuracy of the prediction is highly dependent upon the quality of data collected, this elaborate system is associated with significant costs to avoid any deviation from the precise data-collection methodology that may result in errors in predictions. The cumbersome computer-based data entry requires considerable training, and often, a full time specialist dedicated to collect data on site. Because of the cost, the online “calculator” version of APACHE IV is more commonly used.

Once a scoring system is found to be accurate, one must assess if the prediction obtained has any influence on the management; it becomes beneficial to use in clinical practice if an impact study demonstrates decisional power.^12,13 To date, the impact of APACHE IV (or any other general ICU scoring system for that matter) on patient outcome, on clinical decision-making or cost has not been addressed in any prospective study or a randomized controlled trial.

Despite the existence of many prognostic models and their attractive potential utility, scoring systems have a limited clinical value at the bedside and are seldom used in clinical practice. There are a number of reasons for this.

Prognostic models should closely follow the advancement of medical sciences and be updated in a timely fashion to avoid becoming obsolete. There may be limitations of the predictive ability after modification of a variable. As diagnosis and treatment evolve scoring systems require repeated recalibration and validation, although this is difficult and time consuming to achieve. Also the lack of collaboration between researchers that might improve upon a single model leads to the multiplication of redundant, often competing and nonvalidated models.^10,13,14,15

Most of the available information on scoring systems is based on internal validation, less studies report external validation and almost none report the impact on clinical decision-making and outcome.¹⁶ After the external validation process is completed, the impact of the scoring system on clinical decision-making, patient outcome, or costs should be assessed in a management study or randomized controlled trial^12,13 before being used in clinical practice.

Scoring systems designed for general medical or surgical ICUs would be inappropriate if used in specialized units such as coronary, burn or pediatric where patients differ from the populations of the original and external validation samples from which the system was derived. Clinical judgment remains fundamental because not all relevant variables are included in the models.

The outcome and the time points might be different than those for which the scoring system was designed. For example, changes in hospital practices resulting in earlier discharge of patients to long-term acute care facilities thus reducing the hospital mortality or calculating scores outside of the intended and validated ranges, for example, daily versus within the first hour or first 24 hours.

To be used in clinical practice, the user should understand the existing models, be convinced of their utility and trained in their application. The barriers for the effective use of scoring systems may be related to the limited practical utility of the models. Some models such as APACHE IV are complex and require dedicated personnel and a specific training; they can be expensive to use, therefore, limiting their spread across the healthcare community.

The more data required by the scoring system, both in amount and complexity, the greater the opportunity for errors and missing data. In theory, different users with similar backgrounds in critical care should obtain similar scores for the same patient population, however, low interobserver reliability resulting in dissimilar scoring values by different users can occur.

Scoring systems provide only probabilities, not certainties. It would be inappropriate to deny or withdraw care based solely on a probability derived from any existing scoring system. The course of illness over time is important. Clinical variables may not be included in these models and clinical judgment and ethical principles need to be employed when making decisions regarding prognosis in an individual patient. For example, the physiologic response of sepsis which is linked to unknown genomic factors is not taken into account in any scoring system. Human physiology is complex and clinical assessment prevails over scoring system predictions.

Benchmarking Intensive Care Units Based on Scoring Systems

The ranking of intensive care units and institutions based on severity of illness versus mortality rate is controversial. Different populations of patients in different ICUs are not always comparable even when their severity of illness scores is identical. For example, a hospital with the same mean prognostic score but a higher mortality does not necessarily underperform when compared to another hospital as there may be other elements that are not represented in the dataset captured by these systems. For instance, as the lack of health insurance is associated with increased 30-day mortality and decreased use of common procedures for the critically ill, the insurance status may be an indicator of mortality not picked up by patient and clinical characteristic variables.¹⁷ In elective surgical admissions, there is an association between the socioeconomic status and hospital mortality that is not explained by case mix or the withdrawal of active treatment.¹⁸ Finally, among trauma patients, the uninsured receive less trauma-related care and have a higher mortality rate.¹⁹ These examples illustrate the fact that 2 different populations of patients can have similar severity of illness scores with very different mortality rates.

Although scoring systems are numerous and potentially useful, they are not commonly used in clinical practice in part because many flaws exist in their development, validation, and maintenance. The obstacles to the implementation of scoring systems are significant. However, the strategies to overcome many of these barriers are becoming available as clinical information systems become electronic and more sophisticated. In the future, systems may be able to parse notes in the medical record to form alerts in real time and the costs and errors involved with specialized and redundant data entry may be avoided.