Abstract

Background:Despite the importance of case logs in evaluating residents, no studies assess their accuracy in neurological surgery. Studies from other specialties reveal variations in reporting. This study assesses the accuracy of neurological surgery resident case logs at a single institution.

Methods:Data was collected from three databases: billing data and two separate resident-managed case logs [department log and Accreditation Council for Graduate Medical Education (ACGME) case logs], containing records of procedures performed by 14 neurological surgery residents at a single institution over a 1-year period. The billing data was used as a proxy for a census of procedures performed during the study period. The difference between the number of procedures logged by residents and the number of procedures billed was calculated to determine the accuracy of the resident case logs.

Results:Over the study period, 2150 procedures were billed at the institution, whereas 1749 procedures were logged in the ACGME case log and 1873 in the department log, representing an error rate of −18.65% and −12.88%, respectively. The error rate varied significantly (−1150% to +50.23%) between ACGME procedure categories. In 13 of the 22 ACGME procedure categories, the procedures were under-logged by residents in both resident-managed case logs. No category demonstrated over-logging in both case log systems.

Conclusion:Resident managed case logs are an incomplete representation of clinical work. The cause for inaccuracy is multifactorial. The authors suggested that further research is necessary to validate their results and to identify means by which the accuracy of case logs can be increased.

Keywords: Case log, evaluation, residency

INTRODUCTION

Neurological surgery resident case logs are an important factor in evaluating residents and residency programs. As such, resident case logs are an Accreditation Council for Graduate Medical Education (ACGME) accreditation requirement.[ 1 ] Despite this importance, there are no established “gold-standard” guidelines for how residents and program are supposed to maintain case logs or studies assessing the accuracy of resident-managed case logs in neurological surgery.

Studies in other specialties demonstrate resident case logs to be a poor marker of actual procedural history; for example, only 14% of neurology residents report logging 100% of their patient encounters.[ 2 ] Previous studies also demonstrate variability in case log accuracy between clinical settings and institutions. In orthopedic surgery, 96% of the residents reported routinely logging their procedures performed in the operation room (OR), whereas only 29% of the residents routinely logged procedures performed in the clinic.[ 5 ] In otolaryngology, the range between the minimum and maximum number of total recorded procedures by graduating otolaryngology residents was 4127 procedures (range: 730–4857 procedures) between the academic years 2007–2008 and 2009–2010.[ 3 ] While some of this variability can be explained by the difference in case volume between programs, it is likely that discrepancies in accuracy of procedure logging were present in both extremes.

Despite evidence that resident case volumes do not necessarily correlate with procedural competency,[ 4 ] the ACGME uses the data to evaluate residency programs in their compliance with case category minimums.[ 1 ] Therefore, maintaining accurate resident case logs is important for all neurological surgery residency programs. The purpose of this study is to use objective historical data to assess the accuracy of case logs performed by neurological surgery residents at a single institution.

MATERIALS AND METHODS

The study setting included a census of procedures performed by all 14 neurological surgery residents at the University of Minnesota between 7/1/2012 and 6/30/2013. Data collection required neither interaction with study participants or identifiable private information and was exempt from IRB approval by the University of Minnesota.

The accuracy of neurological surgery resident-managed case logs was determined by comparing the data of two separate resident-managed case logs against the census data. The census of procedures performed during the study period was collected using hospital billing data. Billing data was chosen because of its dedicated data collection and entry personnel (i.e., a coder) and strong financial incentives to fully and accurately collect the data. It is noted that the billing data included procedures performed without the assistance of a neurological surgery resident; while this discrepancy was not directly accounted for in data collection, by report, the number of cases performed in the OR at the University of Minnesota institution without a resident present is <1%. In addition, minor procedures for which the attending may not have been present would not appear in the billing data. The case log data were obtained by querying two separate resident-managed case log systems in which neurological surgery residents are required to record all procedures performed at the institution: the University of Minnesota Department of Neurosurgery Surgeon and Resident Database (department log) and the ACGME Resident Case Log System (ACGME log). The department log is maintained for quality review purposes, whereas the ACGME log is maintained as required for resident evaluation and program accreditation. For the ACGME log, only procedures logged as “Lead Resident Surgeon” or “Senior Resident Surgeon” were included to limit the possibility of including multiple residents logging the same procedure. The data collected in these databases relies on resident diligence to complete data entry, which is enhanced by the use of logs in the semi-annual review of resident performance, as well as a monthly review of data entry and overall case log numbers. However, these reviews are mainly focused on individual resident educational experience (i.e., meeting individual case requirements from the ACGME case logs) and not on accuracy of the case logs.

Once the data were obtained from all three databases, each procedure was categorized into 1 of 22 categories based on the defined current procedural terminology (CPT) codes determined by the ACGME. Where multiple CPT codes were billed, the primary intent of the surgery was reviewed in the medical record to choose the primary CPT code for categorization. Resident-specific data was not obtainable through the hospital billing data, therefore, the differences between the census data and the case logs were calculated at a cohort level for each of the 22 ACGME procedure categories for neurological surgery.

The accuracy of the resident-managed case logs was calculated by taking the difference between the census data and case log data. Because the number of residents included varied between the 22 ACGME categories, both the absolute and relative differences were calculated. The absolute difference is reported as the difference between the number of cases performed in the census and logged in each of the resident-managed case logs. The relative difference is reported as the percent error of the resident managed case log, which was calculated by dividing the absolute difference by the number of procedures performed in the census. A descriptive analysis of the 22 categories was performed for the relative difference and percent error of each of the resident managed case logs. All calculations were performed using Excel 2013 (Microsoft; Redmond, WA). A percent error of ±15% was considered significant. No inferential statistics were calculated as the data represented the entire population of residents and procedures performed at the institution.

RESULTS

The descriptive analysis of the accuracy of the resident managed case logs is summarized in Table 1 . Over the study period, 2150 procedures were performed by the 14 residents, with 1749 procedures logged in the ACGME case log and 1873 in the department log, representing a total percent error of −18.65% and −12.88%, respectively.

Table 1

Descriptive analysis of the accuracy of ACGME procedure categories within resident managed case logs over a 1-year period

A breakdown of the accuracy of the resident managed case logs by ACGME procedure category is shown in Table 2 . A varying number of residents is listed for the individual ACGME categories; this is because the residents were divided between different clinical services and clinical rotations. The accuracy of the resident-managed case logs varied significantly between the ACGME procedure categories, with a percent error ranging from −96.00% to +50.38% for the ACGME log and −1150% to +59.23% for the department log. Of the 22 procedure categories, 13 had a significant percent error of greater than ±15% in both resident-managed case logs, and there were 6 categories with greater than ±30% error in both case log systems. The three most inaccurate categories were “craniotomy for epilepsy,” “endovascular therapy for tumor or vascular lesion,” and “radiosurgery.” As demonstrated in Table 3 , in 13 of the 22 categories, procedures were under-logged by residents in both case log systems, and no category demonstrated over-logging in both case log systems. The categories with the highest rates of over-logging were “additional procedures” and “minor procedures/critical care.”

Table 2

Accuracy of resident managed case logs by ACGME category over a 1-year period

Table 3

Under-recording and over-recording of >15% by ACGME procedure category

DISCUSSION

The results of this study demonstrate that at our institution the accuracy of neurological surgery resident-managed case logs is highly variable with a consistent bias towards under-recording of procedures. The underlying cause of this inaccuracy is likely multifactorial and was not specifically addressed in the study methods. Per discussion of these results with residents, several potential causes for reporting inaccuracy emerged. These include the time and energy necessary to complete administrative tasks, such as logging cases, as well as difficulty identifying the correct procedure codes within the ACGME resident case log system.

In addition, an analysis of the patterns in the data reveals additional understanding of why resident-managed case logs have difficulty in accurately representing the clinical work performed by residents at our institution. In Table 3 , the 22 ACGME case categories are divided into two categories depending on their tendency to be under- or over-recorded by residents. This assessment demonstrates only six ACGME categories that are more likely to be over-recorded. The study methods did not assess the underlying cause of this pattern; however, we infer two possible reasons. First, certain categories, such as “extracranial vascular procedures” and “C-spine fracture operative stabilization,” include less common procedures that may be difficult for residents to reach the minimum required case numbers. The importance of logging these less common procedures may lead to a tendency to be over-recorded. In addition, other categories that tend to be over-recorded, such as “minor procedures/critical care” and “additional procedures,” may be secondary to artifact inherent in the study methods. The use of billing data is an imperfect proxy for all procedures performed at our institution. These categories of procedures (i.e., lumbar punctures, ventriculostomies, etc.) are often performed without the presence of an attending, which prevents their inclusion in the billing data. Even if accurately logged by residents, any procedure performed and logged by a resident that is not billed for will inadvertently appear to be over-recorded by our methods. This is a significant inherent limitation of the present study, as “minor procedures/critical care” procedures account for 22.5% of the cases logged by residents in the ACGME log. Despite this strong bias, the overall results demonstrate a strong trend towards under-recording most procedures in the case logs.

In regards to the 10 ACGME categories in which residents consistently under-recorded procedures in both case logs, no clear pattern was identified. One possible explanation for the overall trend of under-recording procedures is a consequence of the great deal of clinical and administrative responsibilities placed on neurological surgery residents. It is feasible that when pressed to their limits the administrative responsibilities, such as recording case logs are the first to decline. This process, which we call case log fatigue, is a real problem per discussion with residents at our institution. This problem may be increased at our institution due to the use of two case logging systems.

The methods of the present study have additional limitations that affect the validity and generalizability of the results. As mentioned above, there are no established “gold-standard” guidelines for how residents are supposed to log cases. For example, if a craniotomy is performed on a tumor in a patient presenting with seizures, then the procedure may be logged with the “Craniotomy – Tumor” or Craniotomy – Epilepsy” category. This ambiguity inherent to the ACGME case log system unavoidably decreases the precision of the present study.

In addition, during the data collection process, multiple factors inherent to the case log systems were identified that may contribute to the inaccuracy of resident managed case logs. Often a single surgery may include multiple procedures. When logging these procedures, the practice of unbundling a surgery into multiple component procedures for the ACGME case log has resulted in confusion and prevented accurate categorization. In addition, while reviewing the billing data, there also appeared to be discrepancies in how CPT codes were used in real world billing versus the assigned category by the ACGME.

Many hurdles prevent the accurate recording of clinical work of neurological surgery residents. These include the time and energy needed to complete this administrative task, the lack of clarity in the division of responsibility between multiple residents in a procedure, as well as in the allocation of responsibility between the ACGME, residency programs, and the residents in taking ownership of the process.

The present study was performed at a single institution and may not be generalizable to other residency programs, therefore, further research is necessary to validate these results. However, these results and the previously cited literature suggests that this problem is not restricted to a single residency program or specialty. To improve the accuracy of this data, we suggest that efforts be directed at the following goals: reduce the administrative effort needed to collect the data, improve the tools and interface to enter the data, and simplify the procedure categorization system.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.

References

1. . Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education in Neurological Surgery. 2014. p.

2. Gill DJ, Freeman WD, Thoresen P, Corboy JR. Residency training: The neurology resident case log: A national survey of neurology residents. Neurology. 2007. 68: E32-3

3. Rosenberg TL, Franzese CB. Extremes in otolaryngology resident surgical case numbers. Otolaryngol Head Neck Surg. 2012. 147: 261-70

4. Safavi A, Lai S, Butterworth S, Hameed M, Schiller D, Skarsgard E. Does operative experience during residency correlate with reported competency of recent general surgery graduates?. Can J Surg. 2012. 55: S171-7

5. Salazar D, Schiff A, Mitchell E, Hopkinson , W . Variability in Accreditation Council for Graduate Medical Education Resident Case Log System Practices Among Orthopaedic Surgery Residents. J Bone Joint Surg. 2014. e22: 1-9