Sleep Staging
How do you detect sleep stages?
In this version of SomnoBot, sleep stages are automatically detected using well validated neural networks called RobustSleepNet (RSN) [1] and AnySleep [2]. Both models have been trained and validated on a large number of polysomnographic recordings from subjects with different medical conditions, from different studies, and from different clinics. Public checkpoints of the models are available on GitHub here (RSN) and here (AnySleep).
Your polysomnographic recordings (PSGs) are scored by SomnoBot on your computer in your browser. This means that no PSGs are transmitted to us or any third party. When you visit our website, our implementations of the models are downloaded to your web browser, which runs the neural networks locally.
SomnoBot will detect sleep stages regardless whether your recordings are strongly contaminated with artifacts or not. This is often very helpful, particularly for contaminated recordings. However, SomnoBot will yield sleep stages even for segments such as flatlines (e.g., when electrodes were not plugged in). This is expected behavior. We plan to detect artifacts in a future version of SomnoBot.
What is "High-frequency Sleep Staging"?
High-frequency sleep staging provides sleep stages at higher temporal resolution than traditional 30-second epochs, which originated from the practical constraints of manual annotation on paper strips rather than from a physiological rationale [3]. At this point, we consider such sleep stages experimental, and we note that standards for high-resolution sleep staging are lacking. However, it is well established that sleep unfolds at much finer temporal resolutions than can be captured by 30-s epochs, as evidenced by gradual sleep stage transitions and micro-arousals. Researchers have demonstrated that algorithms detecting sleep stages at higher temporal resolutions can be more discriminative for distinguishing healthy controls from patients with obstructive sleep apnea ([4], [2]), narcolepsy type 1, or insomnia [2]. Moreover, models such as AnySleep [2] can capture short-scale sleep dynamics, such as arousals, in their high-frequency sleep stage predictions. This is why we believe that high-frequency sleep staging may serve as a foundation for novel sleep disorder biomarkers.
Which channels should I select to detect sleep stages?
We recommend selecting at least two EEG channels and one EOG channel. Indeed, the neural networks were designed to handle arbitrary EEG montages and have been demonstrated to achieve stable performance for different EEG montages (see Table 4 in Ref. [1] and Figure 1 in Ref. [2]). The results also suggest that selecting EEG and EOG channels will yield more accurate scores than selecting only a single channel such as EOG.
Are the sleep stages correctly detected?
We have taken every possible measure to ensure that SomnoBot's sleep scores are accurate. However, as a good scientist, you should not blindly trust the sleep scores. It is always a good idea to check the scores on a sample of epochs from your recordings.
Below we explain
- what it means to be "accurate" in sleep stage scoring,
- what we suggest as a simple protocol that you can use to evaluate whether SomnoBot is indeed accurately detecting sleep stages in your data, and
- how we ensured that the implementation of the underlying neural networks are correct.
What does it mean to be "accurate" in sleep stage scoring?
Even expert scorers make errors. When different human experts manually score the same PSG recording, they will usually disagree on some epochs (inter-rater variability), despite their best efforts and training [5]. When asked to score a recording twice, even a single human expert will often not identify exactly the same sleep stages for all epochs (intra-rater variability) [6]. One approach to increasing the accuracy of sleep scoring is to have different experts score the same recordings and derive an expert consensus score from these scores. Agreement between experts depends on the sleep stage, with substantial agreement reported for W and R stages, moderate agreement for N2 and N3 stages, and fair agreement for N1 stage [5]. In fact, a comprehensive study suggests that the percentage of epochs where all expert scorers are in agreement decreases with the number of expert scorers involved, reaching only 20-30% of epochs in the limit of an infinite number of expert scorers [7].
In light of these results, we propose that the scoring performance of automated scoring systems should be compared to that of expert scorers. Indeed, the underlying neural networks used in SomnoBot have been shown to achieve agreement with expert scorers that is comparable to, or higher than, the agreement observed among expert scorers themselves (Figure 2 in Ref. [8], and Supplementary Figure S4 in Ref. [2]). This is encouraging and suggests that the networks can reproduce expert scoring behavior within the range of inter-scorer variability.
A simple protocol to evaluate detection accuracy in your data
We always recommend visualizing the hypnogram to check for anomalies. Keep in mind that N1 is particularly difficult to score for both expert scorers and automated systems.
If you want to assess whether SomnoBot scores sleep as you do, we recommend the following.
-
Select one of your PSG recordings and score sleep manually.
-
Let SomnoBot detect sleep stages for the selected recording.
-
Compare your scores with SomnoBot's scores by calculating Cohen's Kappa.
Below, we link to resources to help you do this. The higher the Cohen's Kappa, the better the agreement between your scores and SomnoBot's scores. Keep in mind that you cannot expect perfect agreement between your and SomnoBot's scores. Even human experts do not achieve perfect agreement (see previous section).
-
Compare your Cohen's Kappa value with those obtained between pairs of expert scorers of the DODO/DODH datasets [9].
If your Kappa value is within the distribution of Kappa values, this is a good indication that SomnoBot is detecting sleep stages similar to you, with an agreement comparable to that between two expert scorers. If your Kappa value is below the distribution of Kappa values, then SomnoBot's scores are in less agreement with your scores than two expert scorers would normally be in agreement with each other. In such a case, you may want to reconsider whether or not you want to use SomnoBot for your data.
Figure: Probability distributions of the agreement (measured by Cohen's Kappa) between pairs of expert scorers for scoring sleep in recordings of healthy subjects (DODH) and subjects with obstructive sleep apnea (DODO). The distributions indicate a ‘substantial’ agreement (Kappa values between 0.61-0.80) for the majority of recordings. Expert scorers tend to agree less on sleep scores for subjects with sleep apnea compared to healthy subjects, where we can observe recordings with only ‘fair’ agreement (Kappa values between 0.21-0.40). Datasets DODO and DODH are publicly available [9]; this figure was created by the authors of SomnoBot.
Kappa values between 0.21-0.40 indicate a ‘fair’, between 0.41-0.60 a ‘moderate’, between 0.61-0.80 a ‘substantial’ and above 0.81 a ‘near-perfect’ agreement between scorers [5]. Compare the Cohen's Kappa value you determined between your scores and SomnoBot's scores for a sample recording with the distributions shown in the figure to determine whether you want to trust SomnoBot's scores or not.
There are several scripts available online that can help you calculate Cohen's Kappa for your script language of choice, for instance for Matlab, Python, or R. Likewise, you can use an Excel sheet we created to help you calculate Cohen's Kappa and is available online here.
How we ensured that the neural network was correctly implemented
SomnoBot uses well validated neural networks called RobustSleepNet (RSN) [1] and AnySleep [2]. The models have been developed, trained and validated using data from various clinics and sleep studies and are publicly available here (RSN) and here (AnySleep). To run the models in your browser, we have ported the neural networks to SomnoBot using the same model weights as in the original publications [1], [2]. However, computers operate with finite numerical precision, which varies between programming languages, libraries and computing hardware. This will inevitably lead, in rare cases, to SomnoBot predicting a different sleep stage for a given epoch as compared to the original implementations. We tested how often such a discrepancy can be observed on 70 PSG recordings from the IS-RC dataset [10]. Out of 84,347 epochs only 2 epochs were scored differently compared to the original implementation, which corresponds to a misclassification rate of 0.002%. We consider the observed misclassification rate to be negligible, indicating that our implementation closely follows the original one.