yasa.compare_detection#
- yasa.compare_detection(indices_detection, indices_groundtruth, max_distance=0, *, method='distance', iou_threshold=0.2)[source]#
Determine correctness of detected events against ground-truth events.
- Parameters:
- indices_detectionarray_like
Indices of the detected events. For example, this could be the indices of the start of the spindles, or the negative peak of the slow-waves. The indices must be in samples, and not in seconds. With
method="intersection", this is instead an array of shape (n_events, 2) with the start and end of each event, in any unit (e.g. seconds).- indices_groundtrutharray_like
The ground-truth events, in the same format and unit as
indices_detection.- max_distanceint, optional
Maximum distance between indices, in samples, to consider as the same event (default = 0). For example, if the sampling frequency of the data is 100 Hz, using max_distance=100 will search for a matching event 1 second before or after the current event. Only used with
method="distance".- methodstr
How to match detected and ground-truth events.
"distance"(default): two events match if their indices are withinmax_distanceof each other. A single event can match several events."intersection": two events match if they overlap enough (seeiou_threshold). Unlike"distance", this takes the duration of the events into account. Each event can match at most one other event, and YASA keeps the pairing with the most matches. This method requires both the start and end of each event.
- iou_thresholdfloat
Minimum overlap for two events to match with
method="intersection"(default 0.2). The overlap is measured as the intersection over union (IoU), i.e. the duration of the overlap divided by the total duration covered by the two events. It ranges from 0 to 1, where 1 means that the two events are identical. Events that do not overlap never match.
- Returns:
- resultsdict
A dictionary with the comparison results:
tp: True positives, i.e. detected events that match a ground-truth event.fp: False positives, i.e. detected events with no matching ground-truth event.fn: False negatives, i.e. ground-truth events with no matching detected event.precision: Precision score, aka positive predictive value (see Notes)recall: Recall score, aka sensitivity (see Notes)f1: F1-score (see Notes)
Notes
The precision score is calculated as TP / (TP + FP), i.e. the proportion of detected events that match a ground-truth event.
The recall score is calculated as (N - FN) / N, where N is the number of ground-truth events, i.e. the proportion of ground-truth events that match a detected event. This is the same as TP / (TP + FN) when
max_distance=0ormethod="intersection".The F1-score is the harmonic mean of precision and recall.
This function is inspired by the sleepecg.compare_heartbeats function.
Examples
A simple example. Here, detected refers to the indices (in the data) of the detected events. These could be for example the index of the onset of each detected spindle. grndtrth refers to the ground-truth (e.g. human-annotated) events.
>>> from yasa import compare_detection >>> detected = [5, 12, 20, 34, 41, 57, 63] >>> grndtrth = [5, 12, 18, 26, 34, 41, 55, 63, 68] >>> compare_detection(detected, grndtrth) {'tp': array([ 5, 12, 34, 41, 63]), 'fp': array([20, 57]), 'fn': array([18, 26, 55, 68]), 'precision': 0.7142857142857143, 'recall': 0.5555555555555556, 'f1': 0.6250000000000001}
There are 5 true positives, 2 false positives and 4 false negatives. This gives a precision score of 0.71 (= 5 / (5 + 2)), a recall score of 0.56 (= 5 / (5 + 4)) and a F1-score of 0.625. The F1-score is the harmonic average of precision and recall, and should be the preferred metric when comparing the performance of a detection against a ground-truth.
Order matters! If we set detected as the ground-truth, FP and FN are inverted, and same for precision and recall. The TP and F1-score remain the same though. Therefore, when comparing two detections (and not a detection against a ground-truth), the F1-score is the preferred metric because it is independent of the order.
>>> compare_detection(grndtrth, detected) {'tp': array([ 5, 12, 34, 41, 63]), 'fp': array([18, 26, 55, 68]), 'fn': array([20, 57]), 'precision': 0.5555555555555556, 'recall': 0.7142857142857143, 'f1': 0.6250000000000001}
There might be some events that are very close to each other, and we would like to count them as true positive even though they do not occur exactly at the same index. This is possible with the max_distance argument, which defines the lookaround window (in samples) for each event.
>>> compare_detection(detected, grndtrth, max_distance=2) {'tp': array([ 5, 12, 20, 34, 41, 57, 63]), 'fp': array([], dtype=int64), 'fn': array([26, 68]), 'precision': 1.0, 'recall': 0.7777777777777778, 'f1': 0.8750000000000001}
Finally, if detected is empty, all performance metrics will be set to zero, and a copy of the groundtruth array will be returned as false negatives.
>>> compare_detection([], grndtrth) {'tp': array([], dtype=int64), 'fp': array([], dtype=int64), 'fn': array([ 5, 12, 18, 26, 34, 41, 55, 63, 68]), 'precision': 0, 'recall': 0, 'f1': 0}
Compare event intervals directly in seconds, without rounding or resampling:
>>> res = compare_detection([[0, 3], [10, 11]], [[2, 5]], method="intersection") >>> res["precision"], res["recall"] (0.5, 1.0)