yasa.SleepStatsAgreement#

class yasa.SleepStatsAgreement(ref_data, obs_data, *, ref_scorer='Reference', obs_scorer='Observed', agreement=1.96, confidence=0.95, alpha=0.05, verbose=True, bootstrap_kwargs={})[source]#

Evaluate agreement between sleep statistics reported by two different scorers. Evaluation includes bias and limits of agreement (as well as both their confidence intervals), various plotting options, and calibration functions for correcting biased values from the observed scorer.

Features include:

Get summary calculations of bias, limits of agreement, and their confidence intervals.
Test statistical assumptions of bias, limits of agreement, and their confidence intervals, and apply corrective procedures when the assumptions are not met.
Get bias and limits of agreement in a string-formatted table.
Calibrate new data to correct for biases in observed data.
Return individual calibration functions.
Visualize discrepancies for outlier inspection.
Visualize Bland-Altman plots.

Warning

Experimental — the API of this class may change before the full release planned for v0.8.0. Use with caution.

See also

yasa.Hypnogram.sleep_statistics

Added in version 0.7.0.

Parameters:

ref_datapandas.DataFrame: A pandas.DataFrame with sleep statistics from the reference scorer. Rows are unique observations and columns are unique sleep statistics.
obs_datapandas.DataFrame: A pandas.DataFrame with sleep statistics from the observed scorer. Rows are unique observations and columns are unique sleep statistics. Shape, index, and columns must be identical to ref_data.
ref_scorerstr: Name of the reference scorer.
obs_scorerstr: Name of the observed scorer.
agreementfloat: Multiple of the standard deviation to plot agreement limits. The default is 1.96, which corresponds to a 95% confidence interval if the differences are normally distributed.

Note

agreement gets adjusted for regression-modeled limits of agreement.
confidencefloat: Confidence level (between 0 and 1) for the confidence intervals applied to bias and limits of agreement. Default is 0.95 (i.e., 95%). The same level is used for both parametric and bootstrapped confidence intervals.
alphafloat: Alpha cutoff used for all assumption tests.
verbosebool or str: Verbose level. Default (False) will only print warning and error messages. The logging levels are ‘debug’, ‘info’, ‘warning’, ‘error’, and ‘critical’. For most users the choice is between ‘info’ (or verbose=True) and warning (verbose=False).
bootstrap_kwargsdict: Optional keyword arguments passed to scipy.stats.bootstrap. Defaults use n_resamples=1000 and method='BCa'. The keys 'confidence_level', 'vectorized', and 'paired' cannot be overridden.

Notes

Sleep statistics that are identical between scorers are removed from analysis.

Many steps here are influenced by guidelines proposed in Menghini et al., 2021 [Menghini2021]. See https://sri-human-sleep.github.io/sleep-trackers-performance/AnalyticalPipeline_v1.0.0.html

References

[Menghini2021]

Menghini, L., Cellini, N., Goldstone, A., Baker, F. C., & de Zambotti, M. (2021). A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. SLEEP, 44(2), zsaa170. https://doi.org/10.1093/sleep/zsaa170

Examples

>>> import pandas as pd
>>> import yasa
>>>
>>> # Generate fake reference and observed datasets with similar sleep statistics
>>> ref_scorer = "Henri"
>>> obs_scorer = "Piéron"
>>> ref_hyps = [yasa.simulate_hypnogram(tib=600, scorer=ref_scorer, seed=i) for i in range(20)]
>>> obs_hyps = [h.simulate_similar(scorer=obs_scorer, seed=i) for i, h in enumerate(ref_hyps)]
>>> # Generate sleep statistics from hypnograms using EpochByEpochAgreement
>>> eea = yasa.EpochByEpochAgreement(ref_hyps, obs_hyps)
>>> sstats = eea.get_sleep_stats()
>>> ref_sstats = sstats.loc[ref_scorer]
>>> obs_sstats = sstats.loc[obs_scorer]
>>> # Create SleepStatsAgreement instance
>>> ssa = yasa.SleepStatsAgreement(ref_sstats, obs_sstats)
>>> ssa.summary().round(1).head(3)
variable   bias_intercept             bias_parm  ... loa_slope uloa_parm
interval           center lower upper    center  ...     upper    center lower upper
sleep_stat                                       ...
%N1                  -5.4 -13.9   3.2       0.3  ...       1.6       6.1   3.7   8.5
%N2                 -27.3 -49.1  -5.6      -0.2  ...       1.4      12.4   7.2  17.6
%N3                  -9.1 -23.8   5.5       1.4  ...       1.8      20.4  12.6  28.3

[3 rows x 21 columns]

>>> ssa.get_table().head(3)[["bias", "loa"]]
                      bias                            loa
sleep_stat
%N1                   0.25  Bias ± 2.46 * (-0.00 + 1.00x)
%N2         -27.34 + 0.55x   Bias ± 2.46 * (0.00 + 1.00x)
%N3                   1.38   Bias ± 2.46 * (0.00 + 1.00x)

>>> ssa.assumptions.head(3)
            unbiased  normal  constant_bias  homoscedastic
sleep_stat
%N1             True    True           True          False
%N2             True    True          False          False
%N3             True    True           True          False

>>> ssa.auto_methods.head(3)
            bias   loa    ci
sleep_stat
%N1         parm  regr  parm
%N2         regr  regr  parm
%N3         parm  regr  parm

>>> ssa.get_table(bias_method="parm", loa_method="parm").head(3)[["bias", "loa"]]
             bias            loa
sleep_stat
%N1          0.25    -5.55, 6.06
%N2         -0.23  -12.87, 12.40
%N3          1.38  -17.67, 20.44

>>> new_hyps = [h.simulate_similar(scorer="Kelly", seed=i) for i, h in enumerate(obs_hyps)]
>>> new_sstats = pd.Series(new_hyps).map(lambda h: h.sleep_statistics()).apply(pd.Series)
>>> new_sstats[["N1", "TST", "WASO"]].round(1).head(5)
     N1    TST   WASO
0  42.5  439.5  147.5
1  84.0  550.0   38.5
2  53.5  489.0  103.0
3  57.0  469.5  120.0
4  71.0  531.0   69.0

>>> new_stats_calibrated = ssa.calibrate(new_sstats[ssa.sleep_statistics], bias_method="auto")
>>> new_stats_calibrated[["N1", "TST", "WASO"]].round(1).head(5)
     N1    TST   WASO
0  42.5  439.5  147.5
1  84.0  550.0   38.5
2  53.5  489.0  103.0
3  57.0  469.5  120.0
4  71.0  531.0   69.0

Methods

`__init__`(ref_data, obs_data, *[, ...])
`calibrate`(data[, bias_method, adjust_all])	Calibrate a `DataFrame` of sleep statistics from a new scorer based on observed biases in `obs_data`/`obs_scorer`.
`get_calibration_func`(sleep_stat)	Return a function for calibrating a specific sleep statistic, based on observed biases in `obs_data`/`obs_scorer`.
`get_table`([bias_method, loa_method, ...])	Return a `DataFrame` with bias, loa, bias_ci, loa_ci as string equations.
`summary`([ci_method])	Return a `DataFrame` that includes all calculated metrics:

Attributes

`assumptions`	A `pandas.DataFrame` containing boolean values indicating the pass/fail status of all statistical tests performed to test assumptions.
`auto_methods`	A `pandas.DataFrame` containing the methods applied when `'auto'` is selected.
`data`	A long-format `pandas.DataFrame` containing all raw sleep statistics from `ref_data` and `obs_data`, with a `MultiIndex` with levels `sleep_stat` and `session_id` (or the original index name from the input data).
`n_sessions`	The number of sessions.
`obs_scorer`	The name of the observed scorer.
`ref_scorer`	The name of the reference scorer.
`sleep_statistics`	Return a list of all sleep statistics included in the agreement analyses.