| HOME | HELP | CONTACT US | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
Sign In to gain access to subscriptions and/or personal tools.
|
|
|
|||||||
Sign In to gain access to subscriptions and/or personal tools. |
|||||||||
Letters to the Editor |
Lund, Sweden
Dear Editor:
The article entitled "Rigorous Statistical Reliability, Validity, and Responsiveness Testing of the Cincinnati Knee Rating System in 350 Subjects with Uninjured, Injured, or Anterior Cruciate Ligament-Reconstructed Knee" (by Barber-Westin et al., July/August 1999, pp 402–416) concerns an interesting topic: how to evaluate the outcome of ACL injury. The authors were interested in finding if the Cincinnati Knee Ligament Rating System could be used for such purposes and state that "the rating system has acceptable reliability, validity, and responsiveness for use in outcome studies after knee ligament reconstruction." However, the paper fails to study or discuss several important aspects that are critical to outcomes research and have been the focus of recent methodologic studies.
Outcomes research has been defined as "the study of the end result of health services that take patients’ experiences, preferences, and values into account."3 To ascertain what is important to patients—that is, to ascertain content validity—evaluation instruments should be developed in a process involving patients (the experts on patient-relevant outcomes). The instrument in question was developed by orthopaedic surgeons (experts in ACL surgery) and evaluates what is important to orthopaedic surgeons. Some patient-related aspects are taken into account, but they are weighted differently, according to the constructor’s belief of their importance, and then mixed with objective outcomes such as radiographs and instability testing. The study in question (in Table 5 of the Barber-Westin article), as well as other recent studies, have pointed out that patient-relevant outcomes like pain, swelling, or function are not related to objective outcomes such as instability testing or radiographs. Aggregating these unrelated parameters makes interpretation of the result difficult.
Construct validity has been defined as "the degree to which an instrument measures the theoretical construct it was designed to measure." In the current study, it is not stated what theoretical construct the Cincinnati Knee Rating Scale is supposed to measure. Instead, to prove construct validity, the total score is compared between subgroups that, in three of nine comparisons, are based on subscores included in the total score, indicating that the authors are arguing in a circle.
The Cincinnati Knee Rating System was initially developed to be observer-administered and has a format that requires significant skills to be completed by patients. The administration format is important in outcomes research because it has been shown that observers (biased or not) score significantly better than patients do when scoring the questions themselves. It is strongly recommended that instruments used in outcomes research should be patient self-administered. In the current study it is stated that in the treatment group, "the patients completed the questionnaires before surgery and at the most recent follow-up evaluation." However, in the next paragraph we are told that because these patients were in a prospective outcome study "they were interviewed by an unbiased person other than the surgeon to ensure the accuracy of the answers." These two statements are confusing and might confound the results.
It is also confusing why group 1, the 250 patients undergoing ACL reconstruction who were entered in a prospective outcome study, were not used for test-retest purposes. Instead, 50 patients with knee diagnoses including meniscus tears, knee ligament tears, patellofemoral disorders, and degenerative joint disease were used to assess test-retest reliability. Despite not having subgroup analyses, it was concluded that the rating system is reliable for all these diagnoses.
It is of utmost importance that the results of knee treatment can be compared between studies and between centers. However, it is just as important to consider from whose perspective we should evaluate the treatment. Patient-relevant outcome measures are now promoted in general health care, orthopaedics, and sports medicine, and they should be considered the primary outcome measure in clinical trials.123 The instrument in question was developed by orthopaedic surgeons and reflects the orthopaedic surgeons’ perspectives of ACL-injured patients. For the purpose of outcomes research, patient-friendly and self-administered questionnaires proven valid to assess the patient’s perspective should be used.
REFERENCES
Cincinnati, Ohio
The issues addressed by Dr. Roos are important to the topic of outcomes research and we appreciate the opportunity to provide our opinion regarding this matter and other questions raised in the letter.
First, there appear to be differences in the literature regarding the definition of outcomes research from that provided by Dr. Roos, who stated that outcomes research relates to the "patients’ experience, preference, and values" and "should be considered the primary outcome measure used in clinical trials." Simmons et al.3 stated that "It is a wide spectrum of research activities that include assessment of treatment, measurement of treatment, and when that data is gathered, management of the treatment." These authors further delineate the differences between outcomes research and traditional research, "Outcomes research covers what the patient thinks of the results of the medical care he or she has been given; traditional research covers the standard evaluation of range of motion, strength, radiographs, etc. Outcomes research is in no way meant to replace the usual methods of research in the evaluation of treatment for musculoskeletal disease; it is meant to add another dimension for evaluation." Keller et al.1 described outcomes research as encompassing "...analysis of large databases ...meta-analysis, small-area analysis of health-care utilization; prospective clinical studies emphasizing patient-oriented outcomes of care; and development of decision-making analytical models, cost-effectiveness studies, and practice guidelines." It would appear from these citations that outcomes research covers a broad area and that the patient-perceived segment, while important, is one portion of the overall evaluation and cannot be the only tool used to assess results of treatment.
We incorporated three separate questions in the Cincinnati Knee Rating System to measure the patients’ perception of outcome. These questions are analyzed separately from the overall rating analysis, and are therefore not included in the aggregate overall score or category. These questions include the Patient Perception Scale (Appendix I, p 413 of the original article), a question regarding whether the patient would have the operation again (original article, p 411), and a question that asks the patient to compare the overall knee condition with the preoperative state (original article, p 411). Additionally, in our article we recommended the incorporation of the SF-36 general health outcome instrument and stated that "...this tool provides important general physical and mental health data not currently assessed with any other sports medicine knee-evaluation rating system." We do agree that a disease-specific patient perception instrument may provide even more meaningful data than a general health outcome instrument. A comparative study found that the Musculoskeletal Function Assessment questionnaire had better content validity and responsiveness than two other commonly used health-status measures.2 Other authors have recently published patient-perceived disease-specific instruments in regard to knee surgery; a comparative analysis similar to the one performed by Martin et al.2 needs to be accomplished before the recommendation of one of these instruments over another.
Second, we and other authors have stated that a comprehensive knee rating system must include both patient-perceived measures and objective measures (such as knee stability) to derive the final outcome that is presented to the orthopaedic community. There are problems that may arise by assessing only patient-perceived outcomes. A common problem that we previously identified and discussed in detail is that biases may occur in the results if questionnaires are not designed to detect "knee-abusers," changes in activity level, and symptoms according to specific activity levels. For instance, the problem of the knee-abuser is a common one in which the patient indicates that he or she has returned to sports and is satisfied with the knee, but on further questioning one discovers that symptoms of pain or swelling occur with activity, which could be deleterious to the joint over the long term. In this instance, the patient-perceived response would indicate a favorable result, but our questionnaire (along with an interview) would detect a problem that should be reported. Additionally, the results of these patients should be sorted from the rest of the population who returned to activity without symptoms to provide an accurate account of the results in regard to this variable. To simply provide the number of patients who were satisfied with their present knee condition would not detect this very important problem, and the result would be a positively biased result.
Another example of a potential bias is a reduction in activity level from before surgery to follow-up. Patients may state that they are satisfied with their knee condition, but did they have to decrease their activity level to reach or maintain their satisfaction? The goal in the majority of ACL reconstructions is to return patients to their former activity levels (or to increase their activity levels). Therefore, patients who decreased their level of activity may not be considered as successful a result as those that returned to the same or increased level. The change in activity levels is usually not scientifically assessed in general or patient-perceived outcome instruments. Other institutions have found problems with knee rating systems that assess only subjective patient-perceived symptoms. One study that compared several rating systems found that the Lysholm system (which assesses subjective patient symptoms and functions only) was "not sensitive to detect changes over time, and is not recommended as an outcome instrument after ACL reconstruction."
Third, most knee rating systems provide a system of grouping (or aggregating) results, which requires combining patient-perceived and objective outcome measures. We always provide in our clinical studies an assessment of many of the individual components of the rating system as well as the final grade or score. Examples of some of these individual components include arthrometer test data; symptoms of pain, swelling, and giving way; sports activity levels; and the six functions assessed by the patient questionnaire. These data are provided so that detailed comparisons can be made with other studies that use the Cincinnati Knee Rating System for both individual components as well as the overall rating or score.
Fourth, questions of the methods we used to ascertain construct validity were raised. We used the Engelberg model as described in the study of the Musculoskeletal Function Assessment instrument. Clinical hypotheses were generated and tested against the overall rating score. While Roos maintained that "three of nine comparisons [were] based on subscores included in the total score," this was, in fact, not true. None of the factors (or subgroups) involved in this analysis were a function of the overall rating score. For instance, in hypothesis #7 (symptoms with or no sports), subgroups were developed from a "yes" or "no" subgrouping question, and not the symptom point score derived from the questionnaire.
Fifth, Roos also stated that the Cincinnati Knee Rating System "has a format that requires significant skills to be completed by patients." The results of the reliability analy-ses show that the intraclass correlation coefficients ranged from 0.71 to 1.0, indicating that all variables had adequate reliability. We would suspect that if the rating system were truly difficult to be completed by patients, these correlation coefficients would have been lower. Additionally, our reliability study did not include a patient interview. The patients in the prospective study on ACL reconstruction were interviewed by a researcher (not the surgeon) to ensure accuracy of their responses to detect potential knee-abusers and other biases previously discussed. These data were used for the validity and responsiveness analyses, and not for the reliability analyses. This was because these patients were in a prospective study and the preoperative and 2-year follow-up data had already been collected. We thought that, since these patients had already completed all of the questionnaires at least twice (and more often, in most cases), their prior experience and knowledge would not lend itself to a valid reliability study. Additionally, we stated in the "Discussion" that "This study did not provide a general validation of this instrument for all age groups (pediatric or geriatric) or for all types of knee disorders" and wish to continue to emphasize this fact.
We acknowledge the importance of including patient-perceived outcome questions in the assessment of the results of knee ligament reconstruction. However, we continue to believe that other important objective data must be included, and that certain specific questions must be asked to avoid the potential biases in results that we have discussed. Other authors agree: one study comment was, "A desiderata of future rating scales would be a balance among subjective symptoms, subjective function, and objective findings." Another proposed that, "The system should include measures of symptoms in relation to specific activities, measures of limb impairment, and measures of knee instability." Yet another concluded that, "symptoms, functional testing, examination results, stability and instrumented testing are all essential and should be included." We trust that the evolution of precise and meaningful outcome instruments will continue, and that this evolution will benefit both patients and clinicians in determining the most appropriate treatment for knee ligament injuries.
REFERENCES
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | CONTACT US | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |
