Abstract Details

Title Inter-Rater Reliability of EEG-Based Encephalopathy Grading

Topic Epilepsy/Clinical Neurophysiology (EEG)

Presentation(s) S30 - Epilepsy/Clinical Neurophysiology (EEG): Technology (4:54 PM-5:06 PM)

Poster/Presentation
Number 008

Background

The VE-CAM-S (Visual EEG Confusion Assessment Method – Severity) scale quantifies encephalopathy severity based on EEG features. However, inter-rater reliability of experts using the scale has yet to be assessed.

Objective

To measure inter-rater reliability of experts in assessing encephalopathy severity using the VE-CAM-S grading system.

Design/Methods

We created an online test with thirty-two 15-second EEG samples. Each question asked users to indicate the presence/absence of each of 29 EEG features, 11 of which were used in the VE-CAM-S. Gold-standard was based on consensus of 3 authors (IS, FN, MBW). Ten experts from 6 institutions participated. We quantified performance by average spearman correlation of VE-CAM-S scores with the gold standard, and average sensitivity/specificity. We performed a qualitative analysis to identify errors in recognizing EEG features that most affected VE-CAM-S scores.

Results

The average [95%CI] correlation between VE-CAM-S scores with the gold standard was 0.73 [0.59-0.86]. Specificity was very high (>90%) for all but generalized delta (77%). Sensitivity was high (>70%) for all but brief generalized attenuations (69%), generalized periodic discharges (67%), generalized theta (63%), BIRDS (57%), generalized alpha (57%), extreme delta brushes/EDB (50%), and generalized beta (50%). Probable reasons for errors were subtlety of some findings; confusing some findings (e.g., generalized beta vs. myogenic artifact, burst suppression vs. brief generalized attenuations); failure to correctly recognize BIRDs (mislabeled as focal IEDs) and EDB (mislabeled as GRDA). The largest errors occurred when experts missed or falsely identified features that carry higher weight in the VE-CAM-S scoring rubric.

Conclusions

Expert agreement in VE-CAM-S scoring is high. Error analysis identified several ways to improve future versions, including breaking high-stakes features into smaller parts; creating a “cheat sheet” with scored examples to allow scorers to choose the closest match; and designing teaching materials to help scorers recognize subtle variations of high-stakes patterns.

Authors/Disclosures
Fabio Nascimento, MD (Washington University Medical School) PRESENTER	Dr. Nascimento has received publishing royalties from a publication relating to health care. Dr. Nascimento has a non-compensated relationship as a Editorial Team Member with Neurology RFS that is relevant to AAN interests or activities. Dr. Nascimento has a non-compensated relationship as a Production Team with Neurology Podcast that is relevant to AAN interests or activities.
Anika Zahoor, MD, MBBS	Ms. Zahoor has nothing to disclose.
	No disclosure on file
	No disclosure on file
Christine A. Eckhardt	Dr. Eckhardt has stock in Moderna.
Haoqi Sun, PhD (Massachusetts General Hospital)	Dr. Sun has nothing to disclose.
Adam S. Greenblatt, MD (Washington University in St. Louis)	Dr. Greenblatt has nothing to disclose.
Aline Herlopian, MD (Yale University)	Dr. Herlopian has nothing to disclose.
Ioannis Karakis, MD, FAAN (Emory University)	Dr. Karakis has received personal compensation for serving as an employee of Emory University School of Medicine. Dr. Karakis has received personal compensation in the range of $500-$4,999 for serving as a Consultant for UCB. Dr. Karakis has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for GSK. Dr. Karakis has received personal compensation in the range of $10,000-$49,999 for serving as a Consultant for Epitel. Dr. Karakis has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Ceribell. Dr. Karakis has received personal compensation in the range of $500-$4,999 for serving as an Expert Witness for Hill Ward Henderson.
Roohi Katyal, MD (Louisiana State University Health Sciences Center)	Dr. Katyal has nothing to disclose.
Chetan Nayak, MD (University of Texas Health Science Center at Houston)	Dr. Nayak has nothing to disclose.
Marcus Ng, MD (University of Manitoba)	The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Eisai Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving as a Consultant for Paladin Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving on a Speakers Bureau for Eisai Canada. The institution of Dr. Ng has received personal compensation in the range of $500-$4,999 for serving on a Speakers Bureau for UCB Canada. Dr. Ng has received publishing royalties from a publication relating to health care.
Jonathan Williams, MD (Washington University)	Dr. Williams has nothing to disclose.
Irfan S. Sheikh, MD (UT Southwestern)	Dr. Sheikh has nothing to disclose.
M. B. Westover, MD, PhD (MGH)	Dr. Westover has received personal compensation in the range of $5,000-$9,999 for serving as a Consultant for Beacon Biosignals. Dr. Westover has stock in Beacon Biosignals. The institution of Dr. Westover has received research support from NIH. Dr. Westover has received publishing royalties from a publication relating to health care. Dr. Westover has a non-compensated relationship as a cofounder with Beacon Biosignals that is relevant to AAN interests or activities.