yasa.SleepStatsAgreement#
- class yasa.SleepStatsAgreement(ref_data, obs_data=None, *, ref_scorer='Reference', obs_scorer='Observed', subjects=None, confidence=0.95, alpha=0.05, effect_size_gates=None, bootstrap_kwargs=None, log_transform=False)[source]#
Evaluate agreement between sleep statistics reported by two different scorers. Evaluation includes bias and limits of agreement (as well as both their confidence intervals), various plotting options, and a calibration function for correcting a constant bias of the observed scorer.
Features include:
Get summary calculations of bias, limits of agreement, and their confidence intervals.
Test statistical assumptions of bias, limits of agreement, and their confidence intervals, and apply corrective procedures when the assumptions are not met.
Get bias and limits of agreement in a string-formatted table.
Calibrate new data to correct for a constant bias in observed data.
Visualize Bland-Altman plots.
For a complete walkthrough of the evaluation pipeline of Menghini et al. (2021), see the Evaluating a wearable or staging algorithm against a reference tutorial.
See also
Added in version 0.7.0.
- Parameters:
- ref_data
pandas.DataFrame A
pandas.DataFramewith sleep statistics from the reference scorer. Rows are unique observations and columns are unique sleep statistics.Alternatively, the output of
yasa.EpochByEpochAgreement.get_sleep_stats, i.e. a singlepandas.DataFramewhose first index level is the scorer. In that caseobs_datamust beNoneand the two scorers are taken from the index in order of appearance (reference first).- obs_data
pandas.DataFrameor None A
pandas.DataFramewith sleep statistics from the observed scorer. Rows are unique observations and columns are unique sleep statistics. Shape, index, and columns must be identical toref_data.- ref_scorerstr
Name of the reference scorer. Ignored when
obs_dataisNone.- obs_scorerstr
Name of the observed scorer. Ignored when
obs_dataisNone.- subjectsNone, dict, or array-like
The subject of each session, for datasets with several nights per subject (repeated measures). Either a
dict(orpandas.Series) that maps each session ID (the index ofref_data) to a subject ID, or an array-like with one subject ID per row ofref_data, in the same order. IfNone(default), each session is treated as coming from a different subject. Not needed whenref_datais the output ofEpochByEpochAgreement.get_sleep_statswith asubject_idlevel, since the subjects are then read from the index. See the Notes and Several nights per subject.Added in version 0.8.1.
- confidencefloat
Confidence level (between 0 and 1) for the confidence intervals applied to bias and limits of agreement. Default is 0.95 (i.e., 95%). The same level is used for both parametric and bootstrapped confidence intervals.
- alphafloat
Alpha cutoff used for all assumption tests. Default is 0.05.
- effect_size_gatesdict or None
Effect-size thresholds that must be exceeded, in addition to
pvalue < alpha, for the normality, constant-bias and homoscedasticity assumptions to be considered not met. Because the power of these tests grows with the number of sessions, the p-value alone would flag small and practically irrelevant deviations in large samples: the p-value assesses the statistical evidence for an association, the effect size its practical magnitude. Pass a dict to override one or more of the defaults, orNone(the default) to use all of them. Setting a threshold toNonedisables that criterion. Keys:'skew'(default 1.0) and'kurtosis'(default 2.0) — normality is only considered not met if the differences are markedly asymmetric (|skew| >threshold) or heavy-tailed (excesskurtosis >threshold).'r2'(default 0.1) — proportional bias is only present if at least 10% of the variability of the differences is explained by the reference value.'sd_ratio'(default 1.5) — heteroscedasticity is only present if the fitted absolute residual changes by more than 50% between the lower and the upper end of the observed reference range, i.e. if the typical error varies materially across the range. The criterion is applied symmetrically (a ratio above 1.5 or below 1/1.5), because the variability can either grow or shrink with the reference value.
Added in version 0.8.0.
- bootstrap_kwargsdict
Optional keyword arguments passed to
scipy.stats.bootstrap. Defaults usen_resamples=1000andmethod='BCa'. The keys'confidence_level','vectorized', and'paired'cannot be overridden.- log_transformbool
If
True, apply the Euser et al. (2008) log-transformation method to all sleep statistics. Limits of agreement are then expressed asbias ± slope × ref, whereslopeis derived from the standard deviation of log-ratio differences. This is appropriate when measurement variability is proportional to the measurement magnitude (heteroscedasticity), which is common for duration statistics such as TST, SOL, and WASO. As in the reference pipeline of Menghini et al. (2021),biasis the mean difference on the original scale and the slope is multiplied by the reference value. This is an approximation: Euser et al. (2008) derived the slope for differences relative to the mean of both scorers, and the exact back-transformed limits relative to the reference value are slightly asymmetric. WhenTrue,loa_method='auto'inreportandplot_blandaltmanwill automatically select the Euser method for all statistics, bypassing the homoscedasticity assumption test. Statistics with a value of exactly zero in either scorer (e.g. SOL for a subject who fell asleep in the first epoch) can not be log-transformed; they are excluded with a warning and keep the regular LoA methods. Passingloa_method='log'for such statistics, or whenlog_transform=False, raises aValueError. Default isFalse.
- ref_data
Notes
Sleep statistics that are identical between scorers are removed from analysis. Sessions with a missing value in either scorer (e.g.
Lat_REMfor a night without REM sleep) are removed separately for each sleep statistic, with a warning, so the number of sessions can differ between statistics.Several nights per subject. When
subjectsis given and at least one subject has several nights, the nights are not independent and the following methods are used instead (Bland & Altman, 2007; Myles & Cui, 2007; Parker et al., 2016):The SD of the differences, which defines the limits of agreement, is the total SD of the one-way random-effects ANOVA of Bland & Altman (2007), i.e. the SD of a new night of a new subject (see
variance_components). The bias is the unweighted mean of all differences, so that subjects with more nights weigh more, as in Bland & Altman (2007). This differs from the intercept of a mixed model, which weighs subjects by precision.The parametric confidence interval of the bias uses the model-based standard error of the bias, with
n_subjects - 1degrees of freedom. The confidence intervals of the limits of agreement, LoA half-width and Euser LoA slope use the MOVER method of Zou (2013).The regressions behind the proportional bias and heteroscedasticity tests, and behind the regression-based bias and LoA, are linear mixed models with a random intercept per subject, fitted with statsmodels (
pip install yasa[stats]). Their confidence intervals are always the Wald intervals of the mixed model, even withci_method="boot"(seesummary).The bootstrap confidence intervals resample subjects instead of nights.
The normality test (Shapiro-Wilk) is applied to all differences, as without repeated measures. Its p-value is only approximate when the nights are clustered within subjects, but the skewness and kurtosis gates (see
effect_size_gates) limit the consequences.
References
[Menghini2021]Menghini, L., Cellini, N., Goldstone, A., Baker, F. C., & de Zambotti, M. (2021). A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. SLEEP, 44(2), zsaa170. https://doi.org/10.1093/sleep/zsaa170
[BlandAltman2007]Bland, J. M., & Altman, D. G. (2007). Agreement between methods of measurement with multiple observations per individual. Journal of Biopharmaceutical Statistics, 17(4), 571-582. https://doi.org/10.1080/10543400701329422
[Zou2013]Zou, G. Y. (2013). Confidence interval estimation for the Bland-Altman limits of agreement with multiple observations per individual. Statistical Methods in Medical Research, 22(6), 630-642. https://doi.org/10.1177/0962280211402548
[Parker2016]Parker, R. A., Weir, C. J., Rubio, N., et al. (2016). Application of mixed effects limits of agreement in the presence of multiple sources of variability: exemplar from the comparison of several devices to measure respiratory rate in COPD patients. PLoS ONE, 11(12), e0168321. https://doi.org/10.1371/journal.pone.0168321
[MylesCui2007]Myles, P. S., & Cui, J. (2007). Using the Bland-Altman method to measure agreement with repeated measures. British Journal of Anaesthesia, 99(3), 309-311. https://doi.org/10.1093/bja/aem214
Examples
>>> import pandas as pd >>> import yasa >>> >>> # Generate fake reference and observed datasets with similar sleep statistics >>> ref_scorer = "Henri" >>> obs_scorer = "Piéron" >>> ref_hyps = [yasa.simulate_hypnogram(tib=600, scorer=ref_scorer, seed=i) for i in range(20)] >>> obs_hyps = [h.simulate_similar(scorer=obs_scorer, seed=i) for i, h in enumerate(ref_hyps)] >>> # Generate sleep statistics from hypnograms using EpochByEpochAgreement >>> eea = yasa.EpochByEpochAgreement(ref_hyps, obs_hyps) >>> sstats = eea.get_sleep_stats() >>> # Create SleepStatsAgreement instance >>> ssa = yasa.SleepStatsAgreement(sstats) >>> summary = ssa.summary(ci_method="param").round(1) # 24 columns, 3 per variable >>> summary[["bias_mean", "loa_lower", "loa_upper"]].head(3) variable bias_mean loa_lower loa_upper interval center lower upper center lower upper center lower upper sleep_stat %N1 0.3 -1.1 1.6 -5.5 -7.9 -3.1 6.1 3.7 8.5 %N2 -0.2 -3.2 2.8 -12.9 -18.1 -7.6 12.4 7.2 17.6 %N3 1.4 -3.2 5.9 -17.7 -25.6 -9.8 20.4 12.6 28.3
>>> ssa.report(ci_method="param").head(3)[["Bias [95% CI]", "LoA [95% CI]"]]
>>> ssa.assumptions["constant_bias"].head(3).round(3) metric slope pvalue r2 passed method sleep_stat %N1 0.370 0.181 0.097 True param %N2 0.553 0.017 0.279 False regr %N3 0.613 0.131 0.122 True param
>>> ssa.assumptions.xs("method", level="metric", axis=1).head(3) assumption normal constant_bias homoscedastic sleep_stat %N1 param param param %N2 param regr param %N3 param param param
>>> new_hyps = [h.simulate_similar(scorer="Kelly", seed=i) for i, h in enumerate(obs_hyps)] >>> new_sstats = pd.Series(new_hyps).map(lambda h: h.sleep_statistics()).apply(pd.Series) >>> new_sstats[["N1", "TST", "WASO"]].round(1).head(5) N1 TST WASO 0 42.5 439.5 147.5 1 84.0 550.0 38.5 2 53.5 489.0 103.0 3 57.0 469.5 120.0 4 71.0 531.0 69.0
>>> new_stats_calibrated = ssa.calibrate(new_sstats[["N1", "TST", "WASO"]], bias_method="auto") >>> new_stats_calibrated.round(1).head(5) N1 TST WASO 0 42.1 445.2 145.0 1 83.6 555.8 36.0 2 53.1 494.8 100.5 3 56.6 475.2 117.5 4 70.6 536.8 66.5
With several nights per subject, pass the subject of each night to
EpochByEpochAgreement, or directly to this class withsubjects. Here, we simulate 4 subjects with 3 nights each:>>> ref_hyps = [yasa.simulate_hypnogram(tib=480, scorer="PSG", seed=i) for i in range(12)] >>> obs_hyps = [h.simulate_similar(scorer="Device", seed=i) for i, h in enumerate(ref_hyps)] >>> subjects = [f"S{i // 3}" for i in range(12)] >>> ebe = yasa.EpochByEpochAgreement(ref_hyps, obs_hyps, subjects=subjects) >>> ssa = yasa.SleepStatsAgreement(ebe.get_sleep_stats()) >>> ssa.variance_components.loc[["TST", "WASO", "N3"]].round(1) sd_between sd_within sd_total n_subjects n_sessions sleep_stat TST 0.0 55.0 55.0 4 12 WASO 0.0 56.8 56.8 4 12 N3 6.8 41.5 42.0 4 12
Methods
__init__(ref_data[, obs_data, ref_scorer, ...])calibrate(data[, bias_method])Calibrate a
DataFrameof sleep statistics from a new scorer based on observed biases inobs_data/obs_scorer.plot_blandaltman([sleep_stats, bias_method, ...])Plot Bland-Altman agreement plots for one or more sleep statistics.
report([bias_method, loa_method, ci_method, ...])Return a human-readable
DataFramefor reporting bias, limits of agreement, and statistical assumption results, following the reporting format proposed by Menghini et al. (2021).summary([ci_method, sleep_stats])Return a
DataFramethat includes all calculated metrics:Attributes
A
pandas.DataFramewith the results of the statistical assumption tests for each sleep statistic.A long-format
pandas.DataFramecontaining all raw sleep statistics fromref_dataandobs_data, with aMultiIndexwith levelssleep_statandsession_id(or the original index name from the input data).The effect-size thresholds used together with
alphain the assumption tests.The number of sessions.
The number of unique subjects.
The name of the observed scorer.
The name of the reference scorer.
Return a list of all sleep statistics included in the agreement analyses.
Between- and within-subject components of the variability of the differences.