Reproducing the CalMS21 dataset paper (Sun et al. 2021) from dandiset 001878¶
Paper. Sun JJ, Karigo T, Chakraborty D, Mohanty SP, Wild B, Sun Q, Chen C, Anderson DJ, Perona P, Yue Y, Kennedy A (2021). The Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions. NeurIPS 2021 Datasets and Benchmarks track. arXiv:2104.02710v4. The target names doi:10.22002/D1.1991, which is the CaltechDATA record of the dataset (CalMS21); this paper is the one that record and the dataset describe. The full text (arXiv v4, with its appendix and datasheet) was used.
Dandiset. DANDI:001878, version draft.
The dandiset has no published version, so the draft is the only one there is. It was last modified on
2026-06-25T18:32:10Z. To keep a changing draft from passing unnoticed, the helper module checks the
blob id of each asset it reads against the one recorded when this notebook was written, and stops if
it differs. The dandiset is a collection of conversions from three source datasets; for CalMS21 it
holds one sequence, the Task 1 test sequence mouse071 (pose, annotations as bouts, and the video).
Data from elsewhere. The paper deposited the whole dataset on CaltechDATA (record s0vdx-0k302,
version 1.0, the target DOI). The claims about the dataset as a whole (elsewhere) are tested on
those files, streamed from the record's zips with range requests. The link claim and one pose-error
claim (with inputs) use CaltechDATA's copy of mouse071 next to the dandiset's.
Code. The baseline code the paper links (gitlab.aicrowd.com/aicrowd/research/mab-e/mab-e-baselines) did not answer and no archived copy of the repository was found, and the top entries' code is not linked, so the model benchmarks are blocked.
| Claim | Data | Status |
|---|---|---|
| format-keypoints | dandiset | tested |
| resident-is-black | dandiset | tested |
| task1-vocab | dandiset | tested |
| task1-video | dandiset | tested |
| contact-during-social-behaviors | dandiset | tested |
| pose-errors-close-fast | dandiset | tested |
| link-mouse071 | with inputs | tested |
| pose-errors-close-fast-confidence | with inputs | tested |
| table5-task1-counts | elsewhere | tested |
| table5-task2-counts | elsewhere | tested |
| table5-task3-counts | elsewhere | tested |
| table6-task1-label-fractions | elsewhere | tested |
| fig4b-task2-label-fractions | elsewhere | tested |
| table7-task3-label-fractions | elsewhere | tested |
| dataset-size | elsewhere | tested |
| table10-task3-difficulty | elsewhere | tested |
| table1-architectures, table2-task-programming, table3-task2-style-transfer, table4-task3-new-behaviors, fig7-errors-at-transitions, fig11-annotator-similarity | elsewhere | blocked: the models' code and settings are not available |
Random numbers are drawn only by the bootstrap intervals; each claim's section has its own generator with a fixed seed.
import itertools
import time
import warnings
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import calms21
pd.set_option("display.width", 200)
pd.set_option("display.max_columns", 40)
pd.set_option("display.max_rows", 200)
warnings.filterwarnings("ignore", category=RuntimeWarning)
started_all = time.time()
started = time.time()
s = calms21.load_session()
kp, sc, n = s["keypoints"], s["scores"], s["n_frames"]
print(f"loaded {calms21.CALMS21_NWB} from dandiset {calms21.DANDISET_ID} version {calms21.DANDISET_VERSION} "
f"in {time.time() - started:.0f} s")
loaded sub-mouse071-resident-intruder/sub-mouse071-resident-intruder_ses-mouse071-task1_image.nwb from dandiset 001878 version draft in 2 s
Data check¶
Each check of the metadata against the values is stated under its output.
info = calms21.draft_info()
print("Draft modified:", info["modified"], "; recorded when written:", calms21.DRAFT_MODIFIED)
print("Published version:", info.get("dandiset", {}).get("most_recent_published_version"))
assets = calms21.list_assets()
print(f"\n{len(assets)} assets:")
print(assets[["path", "size", "blob"]].to_string(index=False))
for path, blob in calms21.BLOBS.items():
print("pinned blob unchanged:", path.split("/")[-1], bool((assets.set_index("path").blob[path]) == blob))
meta = calms21.requests.get(f"{calms21.API}/dandisets/001878/versions/draft/").json()
summ = meta["assetsSummary"]
print("\nassetsSummary: numberOfSubjects", summ["numberOfSubjects"], "; numberOfFiles", summ["numberOfFiles"],
"; species", [x["name"] for x in summ["species"]], "; approach", summ["approach"],
"; measurementTechnique", [x["name"] for x in summ["measurementTechnique"]])
print("relatedResource:", [(r["name"], r["relation"]) for r in meta["relatedResource"]])
print("license:", meta["license"])
print("\npose series, frame counts and related publications of each NWB file:")
for path in sorted(assets.path[assets.path.str.endswith(".nwb")]):
ov = calms21.pose_overview(path)
print(f" {path.split('/')[0]}: {len(ov['series'])} series {ov['series']}; frames {ov['frames']}; "
f"related_publications {ov['related_publications']}")
Draft modified: 2026-06-25T18:32:10.096594Z ; recorded when written: 2026-06-25T18:32:10.096594Z
Published version: None
6 assets:
path size blob
sub-mlost-batch2-1-mouse/sub-mlost-batch2-1-mouse_ses-batch2-1_image/4ce9fc13-9b37-48d8-9549-7eed9adcf184_external_file_0.mp4 4187160 6965313b-b732-42ba-a099-158827af0355
sub-mouse071-resident-intruder/sub-mouse071-resident-intruder_ses-mouse071-task1_image.nwb 6717539 e169899a-5a31-44d1-aa7a-93c33e5d3e73
sub-mouse-21-11-8-one-mouse/sub-mouse-21-11-8-one-mouse_ses-21-11-8-one-mouse_image.nwb 6923382 7c872f72-1ea9-4cab-aa71-698e0a6a9895
sub-mlost-batch2-1-mouse/sub-mlost-batch2-1-mouse_ses-batch2-1_image.nwb 3552734 eaab5c37-7a7f-4b65-864d-f2986df5d64e
sub-mouse-21-11-8-one-mouse/sub-mouse-21-11-8-one-mouse_ses-21-11-8-one-mouse_image/b9338a74-9eb3-4458-80ae-8d804d82969d_external_file_0.mp4 23614309 83c83d37-a3c0-4022-a32f-cfc646140637
sub-mouse071-resident-intruder/sub-mouse071-resident-intruder_ses-mouse071-task1_image/304bc772-9d15-41a5-80bd-bc57dcd5c9b9_external_file_0.mp4 184059454 73781d39-da5a-41c9-9f25-9e009ca260d1
pinned blob unchanged: sub-mouse071-resident-intruder_ses-mouse071-task1_image.nwb True
pinned blob unchanged: 304bc772-9d15-41a5-80bd-bc57dcd5c9b9_external_file_0.mp4 True
assetsSummary: numberOfSubjects 3 ; numberOfFiles 6 ; species ['Mus musculus - House mouse'] ; approach [] ; measurementTechnique ['analytical technique']
relatedResource: [('keypoint-MoSeq dataset (Weinreb et al., 2024)', 'dcite:IsDerivedFrom'), ('Mlost unsupervised behavior benchmark (2025)', 'dcite:IsDerivedFrom'), ('CalMS21 - Caltech Mouse Social Interactions (Sun et al., 2021)', 'dcite:IsDerivedFrom')]
license: ['spdx:CC-BY-4.0']
pose series, frame counts and related publications of each NWB file:
sub-mlost-batch2-1-mouse: 11 series ['bcl', 'bcr', 'bodycenter', 'earl', 'earr', 'headcenter', 'hipl', 'hipr', 'neck', 'nose', 'tailbase']; frames [14991]; related_publications ['https://doi.org/10.5281/zenodo.14382973']
sub-mouse-21-11-8-one-mouse: 9 series ['head', 'left ear', 'nose', 'right ear', 'spine1', 'spine2', 'spine3', 'spine4', 'tail']; frames [71524]; related_publications ['https://doi.org/10.5281/zenodo.10636983']
sub-mouse071-resident-intruder: 14 series ['intruder_ear_left', 'intruder_ear_right', 'intruder_hip_left', 'intruder_hip_right', 'intruder_neck', 'intruder_nose', 'intruder_tail_base', 'resident_ear_left', 'resident_ear_right', 'resident_hip_left', 'resident_hip_right', 'resident_neck', 'resident_nose', 'resident_tail_base']; frames [23810]; related_publications ['https://doi.org/10.22002/D1.1991']
The dandiset holds three NWB files with their videos, from three source datasets (keypoint-MoSeq,
the Mlost benchmark and CalMS21); only sub-mouse071-resident-intruder is CalMS21. The other two have
different pose series and frame counts (71,524 and 14,991 frames, against 23,810 here) and cite other
DOIs (all printed above), so no CalMS21 recording is deposited twice. CalMS21 is linked as IsDerivedFrom, and the
dandiset's licence is CC-BY-4.0, which matches the CaltechDATA record (shown below) but not the paper's
datasheet, which gives CC-BY-NC-SA.
print("session_id:", s["session_id"], "; identifier:", s["identifier"])
print("session_start_time:", s["session_start_time"], "; related_publications:", s["related_publications"])
print("subject:", {k: v for k, v in s["subject"].items() if k != "description"})
print("subject description:", s["subject"]["description"])
print("\npose software:", s["pose_software"], "; dimensions (rows, cols):", s["pose_dimensions"])
print("skeleton nodes:", s["skeleton_nodes"])
rates = {v["rate"] for v in s["series"].values()}
starts = {v["start"] for v in s["series"].values()}
units = {v["unit"] for v in s["series"].values()}
print("series:", len(s["series"]), "; rates", rates, "; starting times", starts, "; units", units,
"; reference frame", {v["reference_frame"] for v in s["series"].values()})
print("frames:", n, "=", round(n / calms21.RATE, 2), "s")
rng_tab = pd.DataFrame({
"x_min": kp[:, :, 0].min(axis=(0, 2)), "x_max": kp[:, :, 0].max(axis=(0, 2)),
"y_min": kp[:, :, 1].min(axis=(0, 2)), "y_max": kp[:, :, 1].max(axis=(0, 2)),
"nan": np.isnan(kp).sum(axis=(0, 2, 3))}, index=calms21.MICE)
print("\nkeypoint ranges (pixels):")
print(rng_tab.round(1).to_string())
print("\nconfidence values: finite", int(np.isfinite(sc).sum()), "of", sc.size, "; NaN", int(np.isnan(sc).sum()))
print("derived series rates:", s["centroid_speed_rate"], s["interanimal_distance_rate"],
"; lengths", len(s["centroid_speed"]), len(s["interanimal_distance"]), "; video rate", s["video_rate"])
session_id: mouse071_task1 ; identifier: calms21-task1-mouse071
session_start_time: 2024-01-01T00:00:00+00:00 ; related_publications: ['https://doi.org/10.22002/D1.1991']
subject: {'age': 'P0D/', 'sex': 'U', 'species': 'Mus musculus', 'strain': 'C57BL/6', 'subject_id': 'mouse071_resident-intruder'}
subject description: Resident-intruder dyad (two interacting mice) recorded as one session; NWB stores a single Subject, so the id concatenates the dyad and per-animal split is noted here.
pose software: MARS ; dimensions (rows, cols): [[570, 1024]]
skeleton nodes: ['resident_nose', 'resident_ear_left', 'resident_ear_right', 'resident_neck', 'resident_hip_left', 'resident_hip_right', 'resident_tail_base', 'intruder_nose', 'intruder_ear_left', 'intruder_ear_right', 'intruder_neck', 'intruder_hip_left', 'intruder_hip_right', 'intruder_tail_base']
series: 14 ; rates {30.0} ; starting times {0.0} ; units {'pixels'} ; reference frame {'(0,0) top-left of the video frame'}
frames: 23810 = 793.67 s
keypoint ranges (pixels):
x_min x_max y_min y_max nan
resident 5.0 983.5 30.0 551.0 0
intruder 5.0 991.2 9.0 563.3 0
confidence values: finite 0 of 333340 ; NaN 333340
derived series rates: 30.0 30.0 ; lengths 23810 23810 ; video rate 30.0
One session of one resident and intruder pair, 23,810 frames (13.2 min) at 30 Hz, all 14 series on the
same rate and starting time. Positions are in pixels and lie within the stated 1024 x 570 image (x 5 to
991, y 9 to 563). Every confidence value is NaN: the MARS scores that CalMS21 distributes with the pose
were not carried into the file, though the NWB field is present. The subject is a dyad stored as one
Subject with sex U, strain C57BL/6 only (the BALB/c intruder is not recorded), and age P0D/;
session_start_time is 2024-01-01T00:00, a placeholder, since CalMS21 gives no dates.
bouts = s["bouts"].sort_values("start_time").reset_index(drop=True)
print("ethogram:")
print(s["ethogram"].to_string(index=False))
print("\nannotator attribute:", repr(s["annotator"]), "; description:", s["bouts_description"])
print("bouts:", len(bouts), "; by label:", bouts.label.value_counts().to_dict())
gaps = bouts.start_time.values[1:] - bouts.stop_time.values[:-1]
print("gaps between consecutive bouts (s): min", gaps.min(), "max", gaps.max())
print("first start", bouts.start_time.iloc[0], "; last stop", round(bouts.stop_time.iloc[-1], 4),
"; frames / 30 =", round(n / 30, 4))
on_grid = np.abs(bouts[["start_time", "stop_time"]].values * 30 - np.round(bouts[["start_time", "stop_time"]].values * 30)).max()
print("largest distance of a bout edge from the 30 Hz frame grid (frames):", on_grid)
labels = calms21.frame_labels(bouts, n)
print("frames without a label:", int((labels == "").sum()))
print("frames per label:", pd.Series(labels).value_counts().to_dict())
dur = (bouts.stop_time - bouts.start_time)
print("bout duration (s) by label:")
print(dur.groupby(bouts.label).describe().round(2).to_string())
# The converter's derived series, recomputed from the pose.
cen = calms21.centroids(kp)
d_cent = np.hypot(*(cen[:, 0] - cen[:, 1]).T)
pair = kp.mean(axis=(1, 3))
v_pair = np.r_[np.nan, np.hypot(*np.diff(pair, axis=0).T) * 30]
v_res = np.r_[np.nan, np.hypot(*np.diff(cen[:, 0], axis=0).T) * 30]
print("\ninteranimal_distance minus distance between the two mice's centroids: max |diff|",
np.nanmax(np.abs(d_cent - s["interanimal_distance"])))
print("centroid_speed minus speed of the mean of all 14 keypoints: max |diff|",
np.nanmax(np.abs(v_pair - s["centroid_speed"])))
print("centroid_speed minus resident centroid speed: max |diff|", round(np.nanmax(np.abs(v_res - s["centroid_speed"])), 1))
ethogram:
behavior definition native_code
attack 0
investigation 1
mount 2
other 3
annotator attribute: 'CalMS21 annotator 0' ; description: CalMS21 task1 social-behavior annotation (single-label partition).
bouts: 170 ; by label: {'investigation': 79, 'other': 65, 'mount': 26}
gaps between consecutive bouts (s): min 0.0 max 0.0
first start 0.0 ; last stop 793.6667 ; frames / 30 = 793.6667
largest distance of a bout edge from the 30 Hz frame grid (frames): 1.8189894035458565e-12
frames without a label: 0
frames per label: {'other': 16237, 'investigation': 4679, 'mount': 2894}
bout duration (s) by label:
count mean std min 25% 50% 75% max
label
investigation 79.0 1.97 2.48 0.17 0.57 1.07 2.08 12.53
mount 26.0 3.71 2.78 0.43 1.47 2.92 5.22 10.87
other 65.0 8.33 7.07 0.03 2.53 6.97 11.13 29.23
interanimal_distance minus distance between the two mice's centroids: max |diff| 1.1368683772161603e-13
centroid_speed minus speed of the mean of all 14 keypoints: max |diff| 2.0314416815381264e-11
centroid_speed minus resident centroid speed: max |diff| 1384.2
The catalogue lists attack, investigation, mount and other with codes 0 to 3, but its definition
column is empty for all four, though the paper gives definitions (Appendix D.2). The bouts tile the
session exactly: no gaps, edges on the 30 Hz frame grid, from 0 to 793.67 s, so each frame gets one
label. This session has no attack. interanimal_distance is the distance between the two mice's
centroids (mean of seven keypoints each), as described. centroid_speed is described as the speed of
"the pose centroid (mean over keypoints)"; it is the speed of the mean of all 14 keypoints, which is the
midpoint of the two mice, not the speed of either animal, and the bouts' mean_speed, peak_speed and
immobility_fraction reduce it. Neither derived series is used below; speeds are computed per mouse.
started = time.time()
vinfo = calms21.video_info()
print("video:", vinfo)
STEP = 2 # decode at half resolution
energy = np.full(n, np.nan)
patch_rows = []
prev = None
n_decoded = 0
body = kp[:, :, :, [3, 4, 5]].mean(axis=3) # neck and hips: (frames, mouse, x/y)
for i, img in calms21.video_frames(every=1, step=STEP):
a = img.astype(np.int16)
if prev is not None and i < n:
energy[i] = np.abs(a - prev).mean()
prev = a
n_decoded += 1
if i % 30 == 0 and i < n:
row = {"frame": i, "sep": float(np.hypot(*(body[i, 0] - body[i, 1])))}
for half in (4, 6, 12): # patch half-widths in full-resolution pixels: 8, 12 and 24 px patches
for m, name in enumerate(calms21.MICE):
x, y = body[i, m] / STEP
h = half // STEP # half-width in decoded pixels; the patch is 2h decoded pixels wide
x0, y0 = int(round(x - h)), int(round(y - h)) # centred on the body centre
p = img[max(0, y0):y0 + 2 * h, max(0, x0):x0 + 2 * h]
row[f"{name}_{2 * half}"] = float(p.mean()) if p.size else np.nan
patch_rows.append(row)
patches = pd.DataFrame(patch_rows)
print(f"decoded {n_decoded} frames in {time.time() - started:.0f} s; pose frames {n}")
# Clock check: video motion energy against the speed of the two mice, over lags.
v_mice = np.nanmean([calms21.speed(cen[:, 0]), calms21.speed(cen[:, 1])], axis=0)
ok = np.isfinite(energy)
lags = np.arange(-15, 16)
xc = [stats.pearsonr(energy[ok][max(0, L):len(energy[ok]) + min(0, L)],
v_mice[ok][max(0, -L):len(v_mice[ok]) - max(0, L)])[0] for L in lags]
xc = pd.Series(xc, index=lags)
print("\ncorrelation of video motion energy with pose speed, by lag (video frame i against pose frame i - lag):")
print(xc.round(3).to_string())
print("peak at lag", int(xc.idxmax()), "frames, r =", round(xc.max(), 3), "; at lag 0 r =", round(xc[0], 3),
"; at lag +-15 r =", round(xc[-15], 3), round(xc[15], 3))
video: {'frames': 23810, 'rate': 30.0, 'width': 1024, 'height': 570, 'codec': 'h264', 'duration_s': 793.6666666666666}
decoded 23810 frames in 34 s; pose frames 23810 correlation of video motion energy with pose speed, by lag (video frame i against pose frame i - lag): -15 0.192 -14 0.203 -13 0.215 -12 0.233 -11 0.251 -10 0.273 -9 0.300 -8 0.329 -7 0.363 -6 0.402 -5 0.445 -4 0.497 -3 0.555 -2 0.624 -1 0.696 0 0.754 1 0.734 2 0.686 3 0.633 4 0.582 5 0.532 6 0.489 7 0.453 8 0.420 9 0.392 10 0.367 11 0.341 12 0.320 13 0.298 14 0.279 15 0.264 peak at lag 0 frames, r = 0.754 ; at lag 0 r = 0.754 ; at lag +-15 r = 0.192 0.264
The video is 1024 x 570 at 30 frames per second with 23,810 frames, as many as the pose. Its motion energy follows the speed of the mice most closely at lag 0 (printed above), so the video and the pose share one clock, to within a frame. The patches sampled here are used in resident-is-black.
# Clock check for the annotations: how well the nose-to-intruder distance separates investigation
# from other frames, with the labels shifted in time.
nose_d = calms21.nose_to_intruder(kp)
def auc(score, positive):
r = stats.rankdata(score)
n1, n0 = positive.sum(), (~positive).sum()
return (r[positive].sum() - n1 * (n1 + 1) / 2) / (n1 * n0)
shifts = np.r_[np.arange(-300, -30, 30), np.arange(-30, 31, 3), np.arange(60, 301, 30)]
rows = []
for k in shifts:
lab = np.roll(labels, k)
sel = np.isin(lab, ["investigation", "other"])
if k > 0:
sel[:k] = False
elif k < 0:
sel[k:] = False
rows.append({"shift_frames": k, "auc_investigation_vs_other": auc(-nose_d[sel], lab[sel] == "investigation")})
shift_tab = pd.DataFrame(rows).set_index("shift_frames")
print(shift_tab.round(3).T.to_string())
best = shift_tab.auc_investigation_vs_other.idxmax()
print("best shift", best, "frames, AUC", round(shift_tab.auc_investigation_vs_other.max(), 3),
"; at 0", round(shift_tab.loc[0].iloc[0], 3), "; at +-300 frames",
round(shift_tab.loc[-300].iloc[0], 3), round(shift_tab.loc[300].iloc[0], 3))
shift_frames -300 -270 -240 -210 -180 -150 -120 -90 -60 -30 -27 -24 -21 -18 -15 -12 -9 -6 -3 0 3 6 9 12 15 18 21 24 27 30 60 90 120 150 180 210 240 270 300 auc_investigation_vs_other 0.568 0.572 0.564 0.559 0.568 0.596 0.641 0.697 0.768 0.89 0.903 0.915 0.928 0.94 0.952 0.964 0.974 0.983 0.99 0.99 0.983 0.975 0.965 0.954 0.944 0.933 0.923 0.913 0.904 0.895 0.803 0.738 0.674 0.635 0.613 0.6 0.611 0.614 0.608 best shift 0 frames, AUC 0.99 ; at 0 0.99 ; at +-300 frames 0.568 0.608
The separation of investigation from other frames by the resident's nose distance is highest near zero shift and falls off with shifts of seconds, so the annotations share the pose's clock to within the printed resolution (3 frames near zero). This is consistent with the bouts being aligned to the pose; it does not resolve offsets of a frame or two.
started = time.time()
rec = calms21.caltech_record()
print("CaltechDATA record:", rec)
files = calms21.caltech_files()
print(files.to_string(index=False))
summaries = []
for (task, split) in calms21.CALMS21_FILES:
t0 = time.time()
df = calms21.set_summary(task, split)
summaries.append(df)
print(f"{task:9s} {split:6s} sequences {len(df):3d} frames {df.n_frames.sum():9d} ({time.time() - t0:.0f} s)")
sets = pd.concat(summaries, ignore_index=True)
print(f"streamed all sets in {time.time() - started:.0f} s")
lab = sets[sets.task != "unlabeled"]
print("\nlabelled sequences whose annotation count differs from the frame count in scores:",
int((lab.n_annotations != lab.n_frames).sum()))
kp_vs_sc = sets.n_frames_keypoints != sets.n_frames
print("sequences whose frame count in keypoints differs from the one in scores:", int(kp_vs_sc.sum()), "of", len(sets),
"; by set:", sets.assign(differs=kp_vs_sc).groupby(["task", "split"]).differs.sum().to_dict())
lab_vocab = lab.assign(vocab=lab.vocab.astype(str), annotator=lab.annotator.astype(str))
print("vocabularies and annotator ids by set:")
print(lab_vocab.groupby(["task", "split", "vocab", "annotator"]).size().to_string())
dup = sets[sets.duplicated("kp_hash", keep=False)].copy()
dup["set"] = dup.task + "/" + dup.split
groups = dup.groupby("kp_hash")
print(f"\nsequences with identical keypoints (whole field): {len(dup)}, in {groups.ngroups} groups; "
f"distinct recordings over all sets: {sets.kp_hash.nunique()} of {len(sets)} sequences")
print("groups by the sets they span:")
print(groups.set.apply(lambda g: " + ".join(sorted(set(g)))).value_counts().to_string())
frames_same = groups.n_frames.nunique().eq(1).all()
print("frame counts equal within every group:", bool(frames_same))
cross = groups.apply(lambda g: sorted(set(g.set)), include_groups=False)
leak = [h for h, sl in cross.items() if any(a.split("/")[0] == b.split("/")[0] and a != b for a in sl for b in sl)]
print("\ngroups that span the training and test split of one task:", len(leak))
for h in leak:
print(" " + " = ".join(dup[dup.kp_hash == h].key))
t1test_in_t3train = dup[dup.kp_hash.isin(dup[dup.set == "task1/test"].kp_hash) & (dup.set == "task3/train")]
t1train_in_t3test = dup[dup.kp_hash.isin(dup[dup.set == "task1/train"].kp_hash) & (dup.set == "task3/test")]
print("Task 3 training sequences that are Task 1 test sequences:", len(t1test_in_t3train), t1test_in_t3train.key.tolist())
print(" distinct recordings among them:", t1test_in_t3train.kp_hash.nunique(), "; Task 3 training sequences per recording:",
sorted(t1test_in_t3train.groupby("kp_hash").size().tolist(), reverse=True))
print("Task 3 test sequences that are Task 1 training sequences:", len(t1train_in_t3test),
f"of {int((sets.task.eq('task3') & sets.split.eq('test')).sum())}")
m071 = sets[sets.key.str.endswith("mouse071_task1_annotator1")].kp_hash.iloc[0]
print("mouse071 also deposited as:", sets[(sets.kp_hash == m071) & ~sets.key.str.endswith("mouse071_task1_annotator1")].key.tolist())
CaltechDATA record: {'doi': '10.22002/D1.1991', 'version': '1.0', 'publication_date': '2021-06-07', 'updated': '2024-08-16T01:02:14.250039+00:00', 'rights': ['cc-by-4.0']}
file size checksum
task1_videos_mp4.zip 28286565405 md5:790b2ff6054c0889c3b0112b3a12eacc
task1_videos_seq.zip 74199137301 md5:b771c359e96a2f6576151e695166652b
task2_annotation_styles.zip 912092847 md5:c97e87e13e77ffb80c073e05c05a4683
calms21_convert_to_npy.py 4390 md5:068f0362ea9879a4051b66bd60656c29
readme.md 10946 md5:ad3faeae6835747fd30fcc0c8e83ad2e
task3_new_behaviors.zip 556170163 md5:df59df02d069bab1cfc376cdc1a3925b
seq_image_extraction.zip 9157 md5:ae250529ec80fb268aa068b5bc35bda1
task1_MARS_features.zip 1831694572 md5:2a8b758dfff719a7310a24fa5a7a2435
unlabeled_videos.zip 3530737176 md5:35ab3acdeb231a3fe1536e38ad223b2e
task1_classic_classification.zip 457357950 md5:8a02654fddae28614ee24a6a082261b8
task1 train sequences 70 frames 507738 (9 s)
task1 test sequences 19 frames 262107 (6 s)
task2 train sequences 30 frames 582873 (10 s)
task2 test sequences 47 frames 954989 (13 s)
task3 train sequences 17 frames 223021 (6 s)
task3 test sequences 109 frames 713006 (10 s)
unlabeled part1 sequences 70 frames 1667948 (24 s)
unlabeled part2 sequences 70 frames 1308755 (18 s)
unlabeled part3 sequences 70 frames 1271925 (37 s)
unlabeled part4 sequences 72 frames 1703430 (23 s)
streamed all sets in 157 s
labelled sequences whose annotation count differs from the frame count in scores: 0
sequences whose frame count in keypoints differs from the one in scores: 0 of 574 ; by set: {('task1', 'test'): 0, ('task1', 'train'): 0, ('task2', 'test'): 0, ('task2', 'train'): 0, ('task3', 'test'): 0, ('task3', 'train'): 0, ('unlabeled', 'part1'): 0, ('unlabeled', 'part2'): 0, ('unlabeled', 'part3'): 0, ('unlabeled', 'part4'): 0}
vocabularies and annotator ids by set:
task split vocab annotator
task1 test {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3} 0.0 19
train {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3} 0.0 70
task2 test {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3} 1.0 13
2.0 6
3.0 4
4.0 4
5.0 20
train {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3} 1.0 6
2.0 6
3.0 6
4.0 6
5.0 6
task3 test {'approach': 1, 'other': 0} 0.0 25
{'disengaged': 1, 'other': 0} 2.0 1
{'groom': 1, 'other': 0} 6.0 13
{'intromission': 1, 'other': 0} 0.0 10
{'mount_attempt': 1, 'other': 0} 0.0 12
{'other': 0, 'sniff_face': 1} 0.0 47
{'other': 0, 'whiterearing': 1} 2.0 1
train {'approach': 1, 'other': 0} 0.0 3
{'disengaged': 1, 'other': 0} 2.0 2
{'groom': 1, 'other': 0} 6.0 2
{'intromission': 1, 'other': 0} 0.0 1
{'mount_attempt': 1, 'other': 0} 0.0 4
{'other': 0, 'sniff_face': 1} 0.0 3
{'other': 0, 'whiterearing': 1} 2.0 2
sequences with identical keypoints (whole field): 202, in 78 groups; distinct recordings over all sets: 450 of 574 sequences
groups by the sets they span:
set
task1/train + task3/test 50
task1/test + task3/test 6
task1/test + task3/train 4
task1/train + task3/test + task3/train 4
task2/test + task2/train 4
task2/train 4
task1/train + task3/train 3
task2/test 2
task1/test + task3/test + task3/train 1
frame counts equal within every group: True
groups that span the training and test split of one task: 9
task1/train/mouse026_task1_annotator1 = task3/approach/train/mouse003_task3_approach = task3/mount_attempt/test/mouse012_task3_mount_attempt = task3/sniff_face/test/mouse026_task3_sniff_face
task1/train/mouse014_task1_annotator1 = task3/sniff_face/train/mouse001_task3_sniff_face = task3/mount_attempt/test/mouse010_task3_mount_attempt
task2/annotator4/train/mouse008_task2_annotator4 = task2/annotator3/test/mouse008_task2_annotator3
task2/annotator4/train/mouse009_task2_annotator4 = task2/annotator3/test/mouse009_task2_annotator3
task1/train/mouse003_task1_annotator1 = task3/approach/train/mouse001_task3_approach = task3/intromission/test/mouse002_task3_intromission = task3/mount_attempt/test/mouse005_task3_mount_attempt = task3/sniff_face/test/mouse004_task3_sniff_face
task1/train/mouse025_task1_annotator1 = task3/approach/train/mouse002_task3_approach = task3/intromission/test/mouse003_task3_intromission = task3/sniff_face/test/mouse025_task3_sniff_face
task2/annotator3/train/mouse006_task2_annotator3 = task2/annotator4/test/mouse006_task2_annotator4
task1/test/mouse086_task1_annotator1 = task3/mount_attempt/train/mouse003_task3_mount_attempt = task3/intromission/test/mouse010_task3_intromission
task2/annotator3/train/mouse007_task2_annotator3 = task2/annotator4/test/mouse007_task2_annotator4
Task 3 training sequences that are Task 1 test sequences: 6 ['task3/groom/train/mouse002_task3_groom', 'task3/intromission/train/mouse001_task3_intromission', 'task3/mount_attempt/train/mouse001_task3_mount_attempt', 'task3/mount_attempt/train/mouse002_task3_mount_attempt', 'task3/mount_attempt/train/mouse003_task3_mount_attempt', 'task3/mount_attempt/train/mouse004_task3_mount_attempt']
distinct recordings among them: 5 ; Task 3 training sequences per recording: [2, 1, 1, 1, 1]
Task 3 test sequences that are Task 1 training sequences: 93 of 109
mouse071 also deposited as: ['task3/groom/train/mouse002_task3_groom']
Every member's decompressed size and CRC matched the zip's directory (the helper checks both). In the labelled sets the length of the annotations equals the number of frames in the scores for every sequence, the frame counts of the keypoints and the scores agree in all 574 sequences, unlabeled ones included, and the vocabularies and annotator ids are those of Appendix C (Task 3 gives annotator ids 0, 2 and 6 by behavior, which the paper does not describe). Many sequences have identical keypoints over their whole length: the same recording deposited more than once, under other names and in other tasks (printed above): 574 sequences hold 450 distinct recordings. Most duplicates are Task 1 recordings that reappear in Task 3 (93 of the 109 Task 3 test sequences are Task 1 training recordings), and some cross the training and test splits of one task: four Task 2 recordings are in the training set of one of annotators 3 and 4 and the test set of the other, and five Task 1 test recordings, mouse071 among them, appear as six Task 3 training sequences (one recording is two of them). The paper says each sequence is an independent observation (Appendix B) and does not mention this. dataset-size counts each recording once under one of its readings; the counts and fractions of Tables 5 to 7 are per set, as the paper reports them.
format-keypoints¶
Each frame holds seven MARS keypoints (nose, left ear, right ear, neck, left hip, right hip, tail base) for each of two mice, the resident (mouse 0) and the intruder (mouse 1), in pixel coordinates of a 1024 x 570 image, at 30 Hz, with a confidence score from 0 to 1 for each keypoint.
expected_names = {f"{m}_{p}" for m in calms21.MICE for p in calms21.PARTS}
found = set(s["skeleton_nodes"])
print("14 series named for the seven parts of resident and intruder:", found == expected_names, f"({len(found)} nodes)")
print("units:", units, "; rates (Hz):", rates)
in_x = (kp[:, :, 0] >= 0) & (kp[:, :, 0] <= calms21.IMAGE_W)
in_y = (kp[:, :, 1] >= 0) & (kp[:, :, 1] <= calms21.IMAGE_H)
print(f"coordinates within the image: x {in_x.mean():.4f}, y {in_y.mean():.4f} of values")
print("pose dimensions stored (rows, cols):", s["pose_dimensions"], "; video", vinfo["width"], "x", vinfo["height"])
fin = np.isfinite(sc)
print(f"confidence values present: {fin.sum()} of {sc.size}" +
(f"; range {sc[fin].min():.3f} to {sc[fin].max():.3f}" if fin.any() else ""))
positions_ok = found == expected_names and units == {"pixels"} and rates == {30.0} and in_x.all() and in_y.all()
scores_ok = fin.all() and (sc[fin] >= 0).all() and (sc[fin] <= 1).all()
print("positions meet the criterion:", positions_ok, "; confidence meets it:", scores_ok)
14 series named for the seven parts of resident and intruder: True (14 nodes)
units: {'pixels'} ; rates (Hz): {30.0}
coordinates within the image: x 1.0000, y 1.0000 of values
pose dimensions stored (rows, cols): [[570, 1024]] ; video 1024 x 570
confidence values present: 0 of 333340
positions meet the criterion: True ; confidence meets it: False
Result: partially reproduced. The file holds the 14 keypoint series named for the seven parts of the resident and the intruder, in pixels, at 30 Hz, with every coordinate inside the 1024 x 570 image. The confidence field is present for every keypoint but holds no values (0 of 333,340 finite).
resident-is-black¶
Mouse 0, the resident, is the black C57Bl/6J mouse, and mouse 1, the intruder, is the white BALB/c mouse.
use = patches[patches.sep >= 60]
print(f"frames sampled: {len(patches)}; with body centres at least 60 px apart: {len(use)}")
for size in (12, 8, 24):
darker = use[f"resident_{size}"] < use[f"intruder_{size}"]
print(f"patch {size:2d} px: resident darker in {darker.mean():.3f} of frames; median grey resident "
f"{use[f'resident_{size}'].median():.1f}, intruder {use[f'intruder_{size}'].median():.1f}")
for sep in (30, 120):
u = patches[patches.sep >= sep]
print(f"separation >= {sep} px ({len(u)} frames), 12 px patch: resident darker in "
f"{(u.resident_12 < u.intruder_12).mean():.3f}")
by_min = use.assign(minute=use.frame // 1800).groupby("minute").apply(
lambda g: pd.Series({"frames": len(g), "resident_darker": (g.resident_12 < g.intruder_12).mean()}))
print("\nby minute of the session (12 px patch):")
print(by_min.round(3).T.to_string())
fig, ax = plt.subplots(figsize=(5, 4))
ax.scatter(use.intruder_12, use.resident_12, s=6)
ax.plot([0, 255], [0, 255], "k--", lw=0.8)
ax.set_xlabel("intruder patch grey level"); ax.set_ylabel("resident patch grey level")
ax.set_xlim(0, 255); ax.set_ylim(0, 255)
print("grey levels plotted span", round(use[["resident_12", "intruder_12"]].min().min(), 1), "to",
round(use[["resident_12", "intruder_12"]].max().max(), 1), "(all inside the axes)")
plt.show()
frames sampled: 794; with body centres at least 60 px apart: 746 patch 12 px: resident darker in 1.000 of frames; median grey resident 46.4, intruder 250.9 patch 8 px: resident darker in 0.997 of frames; median grey resident 46.4, intruder 251.8 patch 24 px: resident darker in 1.000 of frames; median grey resident 47.0, intruder 247.7 separation >= 30 px (793 frames), 12 px patch: resident darker in 0.999 separation >= 120 px (620 frames), 12 px patch: resident darker in 1.000 by minute of the session (12 px patch): minute 0 1 2 3 4 5 6 7 8 9 10 11 12 13 frames 60.0 42.0 59.0 50.0 58.0 56.0 53.0 58.0 60.0 60.0 57.0 60.0 59.0 14.0 resident_darker 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 grey levels plotted span 37.1 to 254.2 (all inside the axes)
Result: reproduced. In the 746 sampled frames where the two body centres are at least 60 pixels apart, the 12-pixel patch on the resident is darker than the one on the intruder in every frame (fraction 1.000; median grey level 46.4 against 250.9). The fraction is 0.997 to 1.000 with other patch sizes (8 and 24 px) and separations, and 1.0 in every minute of the session.
task1-vocab¶
Task 1 frames are annotated by one annotator (annotator id 0) with exactly one label per frame from the vocabulary attack 0, investigation 1, mount 2 and other 3.
eth = s["ethogram"]
codes = dict(zip(eth.behavior, eth.native_code.astype(int)))
print("catalogue codes:", codes, "; equal to the paper's vocabulary:", codes == calms21.TASK1_VOCAB)
print("annotator:", repr(s["annotator"]), "; annotator id 0:", s["annotator"].strip().endswith(" 0"))
uncovered = int((labels == "").sum())
print("frames with no label:", uncovered, "of", n, "; frames with two labels: 0 (frame_labels raises on overlap)")
print("labels used in bouts but not in the catalogue:", sorted(set(bouts.label) - set(eth.behavior)))
catalogue codes: {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3} ; equal to the paper's vocabulary: True
annotator: 'CalMS21 annotator 0' ; annotator id 0: True
frames with no label: 0 of 23810 ; frames with two labels: 0 (frame_labels raises on overlap)
labels used in bouts but not in the catalogue: []
Result: reproduced. The catalogue holds attack, investigation, mount and other with codes 0 to 3, the annotator is given as CalMS21 annotator 0, and every one of the 23,810 frames has exactly one label.
task1-video¶
The source videos of Task 1 are released, recorded at 30 Hz and 1024 x 570 pixels.
print("video file:", s["video_external_file"][0].split("/")[-1])
print(f"size {vinfo['width']} x {vinfo['height']}, {vinfo['rate']} frames/s, {vinfo['frames']} frames "
f"(decoded {n_decoded}), codec {vinfo['codec']}; pose frames {n}")
print("matches 1024 x 570:", (vinfo["width"], vinfo["height"]) == (1024, 570), "; 30 Hz:", vinfo["rate"] == 30.0,
"; same frame count as the pose:", n_decoded == n)
mp4 = calms21.zip_members("task1_videos_mp4.zip")
row = mp4[mp4.member.str.contains("mouse071")]
print("\nCaltechDATA task1_videos_mp4.zip holds", int(mp4.member.str.endswith(".mp4").sum()), "videos; mouse071:")
print(row.to_string(index=False))
print("dandiset mp4 size:", int(assets.set_index("path").loc[calms21.CALMS21_MP4, "size"]), "bytes")
video file: 304bc772-9d15-41a5-80bd-bc57dcd5c9b9_external_file_0.mp4 size 1024 x 570, 30.0 frames/s, 23810 frames (decoded 23810), codec h264; pose frames 23810 matches 1024 x 570: True ; 30 Hz: True ; same frame count as the pose: True
CaltechDATA task1_videos_mp4.zip holds 89 videos; mouse071:
member size compressed crc32
task1_videos_mp4/test/mouse071_task1_annotator1.mp4 964451532 921968852 911229681
dandiset mp4 size: 184059454 bytes
Result: reproduced. The dandiset's video of mouse071 is 1024 x 570 at 30 frames per second with 23,810 frames, as many as the pose. It is 184,059,454 bytes against 964,451,532 for CaltechDATA's file of the same sequence, so it was re-encoded; the two were not compared frame by frame.
contact-during-social-behaviors¶
Close investigation, attack and mount are behaviors in which the resident is in close contact with the intruder; other frames include the animals apart.
rng = np.random.default_rng(101)
rows = []
for name in ["investigation", "attack", "mount"]:
sel = labels == name
base = np.isin(labels, [name, "other"])
if not sel.any():
rows.append({"label": name, "frames": 0})
continue
d, lo, hi = calms21.block_bootstrap_diff(nose_d, labels == name, 300, 2000, rng, include=base)
rows.append({"label": name, "frames": int(sel.sum()), "median_px": np.median(nose_d[sel]),
"other_median_px": np.median(nose_d[labels == "other"]), "diff_px": d, "ci_lo": lo, "ci_hi": hi})
contact = pd.DataFrame(rows)
print("resident nose to nearest intruder keypoint (pixels):")
print(contact.round(1).to_string(index=False))
print("\nquantiles by label (px):")
print(pd.Series(nose_d).groupby(labels).quantile([0.1, 0.5, 0.9]).unstack().round(1).to_string())
# Variants: centroid distance; by half of the session.
rng_v = np.random.default_rng(102)
for name in ["investigation", "mount"]:
base = np.isin(labels, [name, "other"])
d, lo, hi = calms21.block_bootstrap_diff(d_cent, labels == name, 300, 2000, rng_v, include=base)
print(f"variant, centroid distance, {name} minus other: {d:.1f} px (95% CI {lo:.1f} to {hi:.1f})")
half = np.arange(n) < n // 2
for h, nm in ((half, "first half"), (~half, "second half")):
med = pd.Series(nose_d[h]).groupby(labels[h]).median().round(1).to_dict()
print(f"{nm}: median nose distance by label {med}")
fig, ax = plt.subplots(figsize=(6, 3))
for name in ["other", "investigation", "mount"]:
ax.hist(nose_d[labels == name], bins=np.arange(0, 600, 10), histtype="step", density=True, label=name)
ax.set_xlabel("resident nose to intruder (px)"); ax.legend()
print("frames beyond 600 px (outside the axes):", int((nose_d > 600).sum()), "; max", round(nose_d.max(), 1))
plt.show()
resident nose to nearest intruder keypoint (pixels):
label frames median_px other_median_px diff_px ci_lo ci_hi
investigation 4679 21.4 376.0 -354.6 -408.9 -295.2
attack 0 NaN NaN NaN NaN NaN
mount 2894 15.0 376.0 -361.0 -413.6 -303.8
quantiles by label (px):
0.1 0.5 0.9
investigation 9.1 21.4 40.7
mount 6.1 15.0 24.5
other 117.3 376.0 684.7
variant, centroid distance, investigation minus other: -307.2 px (95% CI -399.8 to -239.5)
variant, centroid distance, mount minus other: -355.5 px (95% CI -448.9 to -286.5)
first half: median nose distance by label {'investigation': 21.2, 'mount': 14.8, 'other': 308.5}
second half: median nose distance by label {'investigation': 21.7, 'mount': 15.9, 'other': 415.5}
frames beyond 600 px (outside the axes): 3876 ; max 839.5
Result: reproduced. The resident's nose is a median 21.4 px from the intruder in investigation frames and 15.0 px in mount frames, against 376.0 px in other frames (differences -354.6 and -361.0 px, 95% intervals -408.9 to -295.2 and -413.6 to -303.8). The centroid distance gives the same, as does each half of the session. Attack does not occur in this session and is untested.
pose-errors-close-fast¶
Errors in pose estimation are most common on frames when the two animals are in close contact or moving very quickly.
rng = np.random.default_rng(201)
SEGMENTS = {"nose-neck": (0, 3), "neck-tail": (3, 6), "ear-ear": (1, 2), "hip-hip": (4, 5)}
lengths = {(m, seg): np.hypot(*(kp[:, m, :, a] - kp[:, m, :, b]).T)
for m in (0, 1) for seg, (a, b) in SEGMENTS.items()}
def error_flags(factor):
flags = np.zeros(n, bool)
for v in lengths.values():
q1, q3 = np.percentile(v, [25, 75])
flags |= (v < q1 - factor * (q3 - q1)) | (v > q3 + factor * (q3 - q1))
return flags
# Speed of the faster mouse: running median of each mouse's speed, then the larger of the two.
v_fast = np.maximum(pd.Series(calms21.speed(cen[:, 0])).rolling(15, center=True, min_periods=1).median().values,
pd.Series(calms21.speed(cen[:, 1])).rolling(15, center=True, min_periods=1).median().values)
# Variant in the other order: the larger of the two speeds in each frame, then the running median.
v_fast_alt = pd.Series(np.fmax(calms21.speed(cen[:, 0]), calms21.speed(cen[:, 1]))).rolling(
15, center=True, min_periods=1).median().values
def test(err, q, stat=np.mean, v=v_fast):
out = {}
close = d_cent <= np.quantile(d_cent, q)
fast = v >= np.quantile(v, 1 - q)
for nm, g in (("close", close), ("fast", fast)):
d, lo, hi = calms21.block_bootstrap_diff(err.astype(float), g, 300, 2000, rng, stat=stat)
out[nm] = (err[g].mean(), err[~g].mean(), d, lo, hi)
return out
err = error_flags(3.0)
print(f"frames flagged as pose errors (factor 3): {err.sum()} of {n} ({err.mean():.4f})")
print("per segment and mouse:", {f"{calms21.MICE[m]} {seg}": int(((v < np.percentile(v, 25) - 3 * np.subtract(*np.percentile(v, [75, 25]))) |
(v > np.percentile(v, 75) + 3 * np.subtract(*np.percentile(v, [75, 25])))).sum()) for (m, seg), v in lengths.items()})
print("close threshold (10%):", round(np.quantile(d_cent, 0.1), 1), "px; fast threshold (90%):",
round(np.quantile(v_fast, 0.9), 1), "px/s")
rows = []
for factor in (3.0, 1.5):
e = error_flags(factor)
for q in (0.10, 0.05, 0.20):
res = test(e, q)
for nm, (a, b, d, lo, hi) in res.items():
rows.append({"factor": factor, "fraction": q, "group": nm, "error_in_group": a, "error_elsewhere": b,
"diff": d, "ci_lo": lo, "ci_hi": hi})
errtab = pd.DataFrame(rows)
print("\nerror fraction in close / fast frames against the rest (main: factor 3, fraction 0.10):")
print(errtab.round(4).to_string(index=False))
rows = []
for q in (0.10, 0.05, 0.20):
a, b, d, lo, hi = test(err, q, v=v_fast_alt)["fast"]
rows.append({"fraction": q, "error_in_fast": a, "error_elsewhere": b, "diff": d, "ci_lo": lo, "ci_hi": hi})
print("\nvariant, fast frames by the running median of the larger speed (factor 3):")
print(pd.DataFrame(rows).round(4).to_string(index=False))
by_label = pd.Series(err).groupby(labels).mean()
print("\nerror fraction by annotation label:", by_label.round(4).to_dict())
by_label_n = pd.Series(err).groupby(labels).sum()
print("flagged frames by annotation label:", {k: int(v) for k, v in by_label_n.items()},
"; share of all flagged frames:", (by_label_n / err.sum()).round(3).to_dict())
frames flagged as pose errors (factor 3): 657 of 23810 (0.0276)
per segment and mouse: {'resident nose-neck': 14, 'resident neck-tail': 0, 'resident ear-ear': 113, 'resident hip-hip': 76, 'intruder nose-neck': 3, 'intruder neck-tail': 0, 'intruder ear-ear': 107, 'intruder hip-hip': 391}
close threshold (10%): 66.4 px; fast threshold (90%): 389.0 px/s
error fraction in close / fast frames against the rest (main: factor 3, fraction 0.10):
factor fraction group error_in_group error_elsewhere diff ci_lo ci_hi
3.0 0.10 close 0.1310 0.0161 0.1149 0.0636 0.1787
3.0 0.10 fast 0.0097 0.0296 -0.0199 -0.0331 -0.0081
3.0 0.05 close 0.1780 0.0197 0.1583 0.0854 0.2448
3.0 0.05 fast 0.0076 0.0286 -0.0211 -0.0338 -0.0092
3.0 0.20 close 0.0922 0.0114 0.0807 0.0497 0.1194
3.0 0.20 fast 0.0155 0.0306 -0.0151 -0.0291 -0.0029
1.5 0.10 close 0.3935 0.1579 0.2357 0.1657 0.3142
1.5 0.10 fast 0.0924 0.1913 -0.0989 -0.1345 -0.0572
1.5 0.05 close 0.4685 0.1663 0.3022 0.2107 0.3980
1.5 0.05 fast 0.0982 0.1858 -0.0876 -0.1270 -0.0362
1.5 0.20 close 0.3427 0.1411 0.2016 0.1326 0.2586
1.5 0.20 fast 0.1031 0.2010 -0.0979 -0.1266 -0.0613
variant, fast frames by the running median of the larger speed (factor 3):
fraction error_in_fast error_elsewhere diff ci_lo ci_hi
0.10 0.0097 0.0296 -0.0199 -0.0331 -0.0080
0.05 0.0076 0.0286 -0.0211 -0.0348 -0.0090
0.20 0.0189 0.0298 -0.0109 -0.0245 0.0012
error fraction by annotation label: {'investigation': 0.0453, 'mount': 0.105, 'other': 0.0087}
flagged frames by annotation label: {'investigation': 212, 'mount': 304, 'other': 141} ; share of all flagged frames: {'investigation': 0.323, 'mount': 0.463, 'other': 0.215}
Result: partially reproduced. Frames with an implausible body length are more common when the mice are closest: 0.131 of the closest 10% of frames against 0.016 of the rest (difference 0.115, 95% interval 0.064 to 0.179). They are less common, not more, in the fastest 10% of frames: 0.010 against 0.030 (difference -0.020, interval -0.033 to -0.008). Both directions hold at every threshold and factor shown. With the speed taken in the other order (larger speed per frame, then the running median) the fast frames still have fewer flags, with an interval that excludes 0 at 10% and 5% (-0.020, -0.033 to -0.008) and includes it at 20% (-0.011, -0.025 to 0.001). The error measure is a proxy chosen here, since the dandiset holds no confidence values. Mount holds the largest share of flagged frames (304 of 657, 0.463; error fraction 0.105 against 0.009 in other frames), where a hunched posture could give a real outlier; the flags are checked against the MARS confidence and the video under pose-errors-close-fast-confidence.
link-mouse071¶
The dandiset's CalMS21 sequence is the CalMS21 Task 1 test sequence mouse071, with the same keypoints, confidence scores and frame annotations.
started = time.time()
ref = calms21.caltech_sequence("task1", "test", "mouse071_task1_annotator1")
ref_kp = np.asarray(ref["keypoints"], float)
ref_sc = np.asarray(ref["scores"], float)
ref_ann = np.asarray(ref["annotations"])
print(f"CaltechDATA {ref['key']} loaded in {time.time() - started:.0f} s; metadata {ref['metadata']}")
print("dandiset session:", s["session_id"], "; frames dandiset", n, "CaltechDATA", len(ref_kp))
print("shapes:", kp.shape, ref_kp.shape)
diff = np.abs(kp - ref_kp)
print(f"keypoints: max |diff| {diff.max():.3g} px; fraction of coordinates within 0.001 px {np.mean(diff <= 1e-3):.6f}")
# Order checks: would swapping mice, axes or left/right parts fit better?
print("max |diff| with mice swapped:", round(np.abs(kp - ref_kp[:, ::-1]).max(), 1),
"; with x and y swapped:", round(np.abs(kp - ref_kp[:, :, ::-1]).max(), 1))
code_of = dict(zip(s["ethogram"].behavior, s["ethogram"].native_code.astype(int)))
dandi_codes = np.array([code_of[x] for x in labels])
agree = dandi_codes == ref_ann
print(f"frame labels agree on {agree.sum()} of {n} frames ({agree.mean():.6f})")
print("CaltechDATA label counts:", dict(zip(*np.unique(ref_ann, return_counts=True))),
"; dandiset:", dict(zip(*np.unique(dandi_codes, return_counts=True))))
if (~agree).any():
print("first disagreeing frames:", np.flatnonzero(~agree)[:20])
print(f"\nconfidence: CaltechDATA range {ref_sc.min():.3f} to {ref_sc.max():.3f}, median {np.median(ref_sc):.3f}, "
f"finite {np.isfinite(ref_sc).sum()} of {ref_sc.size}; dandiset finite {np.isfinite(sc).sum()}")
CaltechDATA task1/test/mouse071_task1_annotator1 loaded in 3 s; metadata {'annotator-id': 0, 'vocab': {'attack': 0, 'investigation': 1, 'mount': 2, 'other': 3}}
dandiset session: mouse071_task1 ; frames dandiset 23810 CaltechDATA 23810
shapes: (23810, 2, 2, 7) (23810, 2, 2, 7)
keypoints: max |diff| 0 px; fraction of coordinates within 0.001 px 1.000000
max |diff| with mice swapped: 896.5 ; with x and y swapped: 894.6
frame labels agree on 23810 of 23810 frames (1.000000)
CaltechDATA label counts: {np.int64(1): np.int64(4679), np.int64(2): np.int64(2894), np.int64(3): np.int64(16237)} ; dandiset: {np.int64(1): np.int64(4679), np.int64(2): np.int64(2894), np.int64(3): np.int64(16237)}
confidence: CaltechDATA range 0.030 to 0.980, median 0.722, finite 333340 of 333340; dandiset finite 0
Result: reproduced. The sequence matches by name and has the same 23,810 frames; every keypoint coordinate is identical to CaltechDATA's (largest difference 0 px), and swapping the mice or the axes gives differences of about 895 px, so the order is the stated one. The frame labels from the bouts agree with CaltechDATA's annotations on all 23,810 frames. CaltechDATA's confidence scores (0.030 to 0.980, all finite) have no counterpart in the dandiset, where all are NaN.
pose-errors-close-fast-confidence¶
Errors in pose estimation are most common on frames when the two animals are in close contact or moving very quickly, as shown by the MARS confidence scores.
rng = np.random.default_rng(301)
conf = ref_sc.mean(axis=(1, 2)) # mean over 14 keypoints, from CaltechDATA, matched by frame index
print(f"mean confidence per frame: median {np.median(conf):.3f}, range {conf.min():.3f} to {conf.max():.3f}")
rows = []
for q in (0.10, 0.05, 0.20):
close = d_cent <= np.quantile(d_cent, q)
fast = v_fast >= np.quantile(v_fast, 1 - q)
for nm, g in (("close", close), ("fast", fast)):
d, lo, hi = calms21.block_bootstrap_diff(conf, g, 300, 2000, rng)
rows.append({"fraction": q, "group": nm, "median_in_group": np.median(conf[g]),
"median_elsewhere": np.median(conf[~g]), "diff": d, "ci_lo": lo, "ci_hi": hi})
ctab = pd.DataFrame(rows)
print(ctab.round(4).to_string(index=False))
for q in (0.10, 0.05, 0.20):
fast = v_fast_alt >= np.quantile(v_fast_alt, 1 - q)
d, lo, hi = calms21.block_bootstrap_diff(conf, fast, 300, 2000, rng)
print(f"variant, fast by the running median of the larger speed, {q:.0%}: median confidence fast minus rest "
f"{d:.4f} (95% CI {lo:.4f} to {hi:.4f})")
per_mouse = {name: {"close": np.median(ref_sc[d_cent <= np.quantile(d_cent, 0.1), m].mean(axis=1)) -
np.median(ref_sc[d_cent > np.quantile(d_cent, 0.1), m].mean(axis=1))}
for m, name in enumerate(calms21.MICE)}
print("variant, per mouse, median confidence close minus rest (10%):", {k: round(v["close"], 4) for k, v in per_mouse.items()})
print("correlation of confidence with the error flags of pose-errors-close-fast:",
round(stats.pointbiserialr(err, conf)[0], 3))
fig, ax = plt.subplots(1, 2, figsize=(9, 3))
ax[0].scatter(d_cent[::10], conf[::10], s=2); ax[0].set_xlabel("centroid distance (px)"); ax[0].set_ylabel("mean confidence")
ax[1].scatter(v_fast[::10], conf[::10], s=2); ax[1].set_xlabel("faster mouse speed (px/s)")
ax[1].set_xlim(0, 1500)
print("every 10th frame plotted; frames above 1500 px/s (outside the right axes):", int((v_fast[::10] > 1500).sum()))
plt.show()
mean confidence per frame: median 0.735, range 0.373 to 0.817
fraction group median_in_group median_elsewhere diff ci_lo ci_hi
0.10 close 0.6491 0.7397 -0.0907 -0.1072 -0.0735
0.10 fast 0.7505 0.7323 0.0182 0.0121 0.0252
0.05 close 0.6352 0.7373 -0.1021 -0.1198 -0.0841
0.05 fast 0.7456 0.7338 0.0118 0.0040 0.0196
0.20 close 0.6741 0.7433 -0.0691 -0.0838 -0.0550
0.20 fast 0.7507 0.7299 0.0207 0.0153 0.0266
variant, fast by the running median of the larger speed, 10%: median confidence fast minus rest 0.0182 (95% CI 0.0119 to 0.0250)
variant, fast by the running median of the larger speed, 5%: median confidence fast minus rest 0.0123 (95% CI 0.0040 to 0.0207)
variant, fast by the running median of the larger speed, 20%: median confidence fast minus rest 0.0201 (95% CI 0.0142 to 0.0263)
variant, per mouse, median confidence close minus rest (10%): {'resident': np.float64(-0.0306), 'intruder': np.float64(-0.1487)}
correlation of confidence with the error flags of pose-errors-close-fast: -0.366
every 10th frame plotted; frames above 1500 px/s (outside the right axes): 0
# Check of the body-length proxy of pose-errors-close-fast. In close frames a flag may mark a real
# posture (hunched, rearing) and not a tracking error. If flagged close frames are tracking errors,
# their MARS confidence should be lower than that of unflagged close frames.
rng = np.random.default_rng(302)
close10 = d_cent <= np.quantile(d_cent, 0.1)
rows = []
for nm, g in (("close, flagged", close10 & err), ("close, not flagged", close10 & ~err),
("not close, flagged", ~close10 & err), ("not close, not flagged", ~close10 & ~err)):
rows.append({"frames": nm, "n": int(g.sum()), "median_confidence": np.median(conf[g]),
"median_conf_resident": np.median(ref_sc[g, 0].mean(axis=1)),
"median_conf_intruder": np.median(ref_sc[g, 1].mean(axis=1))})
print(pd.DataFrame(rows).round(3).to_string(index=False))
d, lo, hi = calms21.block_bootstrap_diff(conf, err, 300, 2000, rng, include=close10)
print(f"close frames, median confidence flagged minus not flagged: {d:.3f} (95% CI {lo:.3f} to {hi:.3f})")
print("close flagged frames by label:", pd.Series(labels[close10 & err]).value_counts().to_dict())
print("close frames by label:", pd.Series(labels[close10]).value_counts().to_dict())
# Eight flagged close frames, drawn at random, with the keypoints drawn on the video.
show = np.sort(rng.choice(np.flatnonzero(close10 & err), 8, replace=False))
started = time.time()
imgs = calms21.video_frames_at(show, step=1)
print(f"\nframes shown: {show.tolist()} (decoded in {time.time() - started:.0f} s)")
fig, axs = plt.subplots(2, 4, figsize=(12, 6.5))
for ax, i in zip(axs.ravel(), show):
cx, cy = kp[i].mean(axis=(0, 2))
ax.imshow(imgs[i], cmap="gray", vmin=0, vmax=255)
for m, col in ((0, "tab:orange"), (1, "tab:blue")):
ax.plot(kp[i, m, 0], kp[i, m, 1], "o", ms=3, color=col)
for a, b in SEGMENTS.values():
ax.plot(kp[i, m, 0, [a, b]], kp[i, m, 1, [a, b]], "-", lw=0.8, color=col)
ax.set_xlim(cx - 110, cx + 110); ax.set_ylim(cy + 110, cy - 110)
ax.set_title(f"frame {i}, {labels[i]}, conf {conf[i]:.2f}", fontsize=8); ax.axis("off")
plt.suptitle("flagged close frames; resident orange, intruder blue (220 px windows)")
plt.show()
frames n median_confidence median_conf_resident median_conf_intruder
close, flagged 312 0.574 0.673 0.521
close, not flagged 2069 0.657 0.720 0.606
not close, flagged 345 0.649 0.668 0.610
not close, not flagged 21084 0.740 0.746 0.747
close frames, median confidence flagged minus not flagged: -0.082 (95% CI -0.107 to -0.050)
close flagged frames by label: {'mount': 243, 'investigation': 56, 'other': 13}
close frames by label: {'mount': 1984, 'investigation': 323, 'other': 74}
frames shown: [2684, 3143, 5774, 9019, 10622, 17688, 20951, 22726] (decoded in 30 s)
In the closest 10% of frames, the 312 flagged frames have a lower MARS confidence than the 2,069 unflagged ones (median 0.574 against 0.657, difference -0.082, 95% interval -0.107 to -0.050), lowest for the intruder (0.521). 243 of the 312 are mount frames. In each of the eight flagged close frames drawn above, at least one segment drawn for a mouse runs across the other mouse or the background, which is a tracking error and not a posture (judged by eye on these eight frames only). So the flags in close frames mark mostly tracking errors, though the share of real postures among them is not measured.
Result: partially reproduced. The mean MARS confidence is lower in the closest 10% of frames (median 0.649 against 0.740, difference -0.091, 95% interval -0.107 to -0.074), most of all for the intruder (-0.149, resident -0.031). In the fastest 10% of frames it is higher, not lower (0.751 against 0.732, difference 0.018, interval 0.012 to 0.025). Both hold at the 5% and 20% thresholds, and with the speed taken in the other order (0.018, 95% interval 0.012 to 0.025, at 10%).
table5-task1-counts¶
Task 1 has 70 training sequences with 507,738 frames and 19 test sequences with 262,107 frames.
PAPER_T5_TASK1 = {"train": (507738, 70), "test": (262107, 19)}
t1 = sets[sets.task == "task1"]
rows = []
for split, (frames, seqs) in PAPER_T5_TASK1.items():
g = t1[t1.split == split]
rows.append({"split": split, "frames": int(g.n_annotations.sum()), "paper_frames": frames,
"sequences": len(g), "paper_sequences": seqs})
t5_1 = pd.DataFrame(rows)
print(t5_1.to_string(index=False))
match = (t5_1.frames == t5_1.paper_frames) & (t5_1.sequences == t5_1.paper_sequences)
print("all four counts match:", bool(match.all()))
print("Task 1 sequence length (s): min", round(t1.n_frames.min() / 30, 1), "median", round(t1.n_frames.median() / 30, 1),
"max", round(t1.n_frames.max() / 30, 1))
print("mouse071 in the test split:", bool((t1.key == "task1/test/mouse071_task1_annotator1").any()))
split frames paper_frames sequences paper_sequences train 507738 507738 70 70 test 262107 262107 19 19 all four counts match: True Task 1 sequence length (s): min 3.2 median 237.2 max 793.7 mouse071 in the test split: True
Result: reproduced. The Task 1 files hold 70 training sequences with 507,738 frames and 19 test sequences with 262,107 frames, as reported.
table5-task2-counts¶
Task 2 has, for annotators 1 to 5, 6 training sequences each with 139,112, 135,623, 107,420, 108,325 and 92,383 frames, and 13, 6, 4, 4 and 20 test sequences with 286,810, 150,919, 77,079, 76,174 and 364,007 frames.
PAPER_T5_TASK2 = { # annotator: (train frames, train sequences, test frames, test sequences)
1: (139112, 6, 286810, 13), 2: (135623, 6, 150919, 6), 3: (107420, 6, 77079, 4),
4: (108325, 6, 76174, 4), 5: (92383, 6, 364007, 20)}
t2 = sets[sets.task == "task2"].copy()
t2["group"] = t2.key.str.extract(r"task2/annotator(\d)/")[0].astype(int)
print("annotator id in the metadata equals the one in the key:", bool((t2.annotator.astype(int) == t2.group).all()))
def count_table(df, paper):
rows = []
for g, (trf, trs, tef, tes) in paper.items():
for split, pf, ps in (("train", trf, trs), ("test", tef, tes)):
sub = df[(df.group == g) & (df.split == split)]
rows.append({"group": g, "split": split, "frames": int(sub.n_annotations.sum()), "paper_frames": pf,
"sequences": len(sub), "paper_sequences": ps})
t = pd.DataFrame(rows)
t["frames_rel_diff"] = (t.frames - t.paper_frames) / t.paper_frames
return t
t5_2 = count_table(t2, PAPER_T5_TASK2)
print(t5_2.to_string(index=False))
exact = int((t5_2.frames == t5_2.paper_frames).sum() + (t5_2.sequences == t5_2.paper_sequences).sum())
within = bool((t5_2.frames_rel_diff.abs() <= 0.1).all() and
((t5_2.sequences - t5_2.paper_sequences).abs() <= 0.1 * t5_2.paper_sequences).all())
print(f"counts matching exactly: {exact} of {2 * len(t5_2)}; all within 10%: {within}")
print("\nannotator 1 training sequences (frames in scores, in keypoints, and annotations):")
a1 = t2[(t2.group == 1) & (t2.split == "train")][["key", "n_frames", "n_frames_keypoints", "n_annotations"]]
print(a1.to_string(index=False))
print("totals:", a1[["n_frames", "n_frames_keypoints", "n_annotations"]].sum().to_dict(), "; paper 139112")
annotator id in the metadata equals the one in the key: True
group split frames paper_frames sequences paper_sequences frames_rel_diff
1 train 139122 139112 6 6 0.000072
1 test 286810 286810 13 13 0.000000
2 train 135623 135623 6 6 0.000000
2 test 150919 150919 6 6 0.000000
3 train 107420 107420 6 6 0.000000
3 test 77079 77079 4 4 0.000000
4 train 108325 108325 6 6 0.000000
4 test 76174 76174 4 4 0.000000
5 train 92383 92383 6 6 0.000000
5 test 364007 364007 20 20 0.000000
counts matching exactly: 19 of 20; all within 10%: True
annotator 1 training sequences (frames in scores, in keypoints, and annotations):
key n_frames n_frames_keypoints n_annotations
task2/annotator1/train/mouse001_task2_annotator1 17353 17353 17353.0
task2/annotator1/train/mouse002_task2_annotator1 17415 17415 17415.0
task2/annotator1/train/mouse003_task2_annotator1 27461 27461 27461.0
task2/annotator1/train/mouse004_task2_annotator1 31258 31258 31258.0
task2/annotator1/train/mouse012_task2_annotator1 17415 17415 17415.0
task2/annotator1/train/mouse014_task2_annotator1 28220 28220 28220.0
totals: {'n_frames': 139122.0, 'n_frames_keypoints': 139122.0, 'n_annotations': 139122.0} ; paper 139112
Result: partially reproduced. 19 of the 20 counts match exactly. Annotator 1's training set has 139,122 frames where the table gives 139,112, a difference of 10 frames (0.007%); its 6 sequences match. The six sequences have the same frame counts in scores, keypoints and annotations (printed above); the paper gives no per-sequence counts, so which sequence the 10 frames belong to is not known.
table5-task3-counts¶
Task 3 has, for approach, disengaged, grooming, intromission, mount attempt, sniff face and white rearing, 3, 2, 2, 1, 4, 3 and 2 training sequences with 20,624, 35,751, 45,174, 19,200, 46,847, 19,244 and 36,181 frames, and 25, 1, 13, 10, 12, 47 and 1 test sequences with 126,468, 19,088, 156,664, 55,218, 85,836, 251,793 and 17,939 frames.
PAPER_T5_TASK3 = {
"approach": (20624, 3, 126468, 25), "disengaged": (35751, 2, 19088, 1), "groom": (45174, 2, 156664, 13),
"intromission": (19200, 1, 55218, 10), "mount_attempt": (46847, 4, 85836, 12),
"sniff_face": (19244, 3, 251793, 47), "whiterearing": (36181, 2, 17939, 1)}
t3 = sets[sets.task == "task3"].copy()
t3["group"] = t3.key.str.extract(r"task3/([a-z_]+)/")[0]
print("behaviors in the files:", sorted(t3.group.unique()))
print("vocabulary of each sequence names its behavior with code 1:",
bool(all(v.get(g) == 1 and v.get("other") == 0 for v, g in zip(t3.vocab, t3.group))))
t5_3 = count_table(t3, PAPER_T5_TASK3)
print(t5_3.to_string(index=False))
exact = int((t5_3.frames == t5_3.paper_frames).sum() + (t5_3.sequences == t5_3.paper_sequences).sum())
within = bool((t5_3.frames_rel_diff.abs() <= 0.1).all() and
((t5_3.sequences - t5_3.paper_sequences).abs() <= 0.1 * t5_3.paper_sequences).all())
print(f"counts matching exactly: {exact} of {2 * len(t5_3)}; all within 10%: {within}")
behaviors in the files: ['approach', 'disengaged', 'groom', 'intromission', 'mount_attempt', 'sniff_face', 'whiterearing']
vocabulary of each sequence names its behavior with code 1: True
group split frames paper_frames sequences paper_sequences frames_rel_diff
approach train 20624 20624 3 3 0.0
approach test 126468 126468 25 25 0.0
disengaged train 35751 35751 2 2 0.0
disengaged test 19088 19088 1 1 0.0
groom train 45174 45174 2 2 0.0
groom test 156664 156664 13 13 0.0
intromission train 19200 19200 1 1 0.0
intromission test 55218 55218 10 10 0.0
mount_attempt train 46847 46847 4 4 0.0
mount_attempt test 85836 85836 12 12 0.0
sniff_face train 19244 19244 3 3 0.0
sniff_face test 251793 251793 47 47 0.0
whiterearing train 36181 36181 2 2 0.0
whiterearing test 17939 17939 1 1 0.0
counts matching exactly: 28 of 28; all within 10%: True
Result: reproduced. All 28 Task 3 counts of frames and sequences match the table.
table6-task1-label-fractions¶
In the Task 1 training set, 2.76% of frames are labelled attack, 28.9% investigation, 5.64% mount and 62.7% other.
PAPER_T6 = {"attack": (2.76, 0.005), "investigation": (28.9, 0.05), "mount": (5.64, 0.005), "other": (62.7, 0.05)}
def label_counts(df):
tot = {}
for counts in df.counts:
for c, k in counts.items():
tot[c] = tot.get(c, 0) + k
return tot
def label_percent(df, vocab):
tot = label_counts(df)
allf = sum(tot.values())
return {name: 100 * tot.get(code, 0) / allf for name, code in vocab.items()}, allf
rows = []
for reading, splits in (("a: train", ["train"]), ("b: train+test", ["train", "test"])):
pct, allf = label_percent(t1[t1.split.isin(splits)], calms21.TASK1_VOCAB)
cnt = label_counts(t1[t1.split.isin(splits)])
for name, (val, tol) in PAPER_T6.items():
rows.append({"reading": reading, "label": name, "frames": cnt.get(calms21.TASK1_VOCAB[name], 0), "of": allf,
"percent": pct[name], "paper": val, "tolerance": tol,
"within": abs(pct[name] - val) <= tol, "within_10pct": abs(pct[name] - val) <= 0.1 * val})
t6 = pd.DataFrame(rows)
print(t6.round(5).to_string(index=False))
print(t6.groupby("reading")[["within", "within_10pct"]].all().to_string())
att = sum(c.get(0, 0) for c in t1[t1.split == "train"].counts)
tot = int(t1[t1.split == "train"].n_annotations.sum())
print(f"attack, training split: {att} of {tot} frames = {100 * att / tot:.6f}%; |diff| from 2.76 = {abs(100 * att / tot - 2.76):.6f}")
reading label frames of percent paper tolerance within within_10pct
a: train attack 14039 507738 2.76501 2.76 0.005 False True
a: train investigation 146615 507738 28.87611 28.90 0.050 True True
a: train mount 28615 507738 5.63578 5.64 0.005 True True
a: train other 318469 507738 62.72310 62.70 0.050 True True
b: train+test attack 26669 769845 3.46420 2.76 0.005 False False
b: train+test investigation 207890 769845 27.00414 28.90 0.050 False True
b: train+test mount 60463 769845 7.85392 5.64 0.005 False False
b: train+test other 474823 769845 61.67774 62.70 0.050 False True
within within_10pct
reading
a: train False True
b: train+test False False
attack, training split: 14039 of 507738 frames = 2.765009%; |diff| from 2.76 = 0.005009
Result: partially reproduced. Under reading (a), the training split, investigation (28.876%), mount (5.636%) and other (62.723%) are within tolerance, and attack is 2.76501% (14,039 of 507,738 frames, printed below the table), which rounds to 2.77 and lies 0.00001 beyond the tolerance of 0.005 around 2.76; truncated, it gives the reported 2.76. All four are within 10% of the reported values. Reading (b) does not match.
fig4b-task2-label-fractions¶
The five Task 2 annotators label different fractions of frames as other, investigation, mount and attack (annotator 1 60.0, 20.6, 16.2 and 3.2%; annotator 2 65.5, 25.8, 3.4 and 5.3%; annotator 3 68.9, 13.5, 9.5 and 8.1%; annotator 4 75.2, 11.6, 4.5 and 8.7%; annotator 5 69.5, 26.4, 1.0 and 3.0%).
ORDER = ["other", "investigation", "mount", "attack"]
PAPER_F4B_T2 = {1: (60.0, 20.6, 16.2, 3.2), 2: (65.5, 25.8, 3.4, 5.3), 3: (68.9, 13.5, 9.5, 8.1),
4: (75.2, 11.6, 4.5, 8.7), 5: (69.5, 26.4, 1.0, 3.0)}
TOL = 0.05
results = {}
for split_reading, splits in (("a: train", ["train"]), ("b: train+test", ["train", "test"])):
rows = []
for a, paper in PAPER_F4B_T2.items():
sub = t2[(t2.group == a) & t2.split.isin(splits)]
pct, allf = label_percent(sub, calms21.TASK1_VOCAB)
cnt = label_counts(sub)
mine = np.array([pct[c] for c in ORDER])
best = np.sort(mine)[np.argsort(np.argsort(paper))] # sorted matching minimises the largest deviation
rows.append({"annotator": a, "frames": allf, **{f"n_{c}": cnt.get(calms21.TASK1_VOCAB[c], 0) for c in ORDER},
**{c: round(v, 4) for c, v in zip(ORDER, mine)}, "paper": paper,
"largest_gap": round(np.abs(mine - paper).max(), 4),
"within_i": int((np.abs(mine - paper) <= TOL).sum()),
"within_ii": int((np.abs(best - paper) <= TOL).sum())})
tab = pd.DataFrame(rows)
results[split_reading] = tab
print(f"reading {split_reading}:")
print(tab.to_string(index=False))
print(f" values within {TOL}: class order (i) {tab.within_i.sum()} of 20, best assignment (ii) {tab.within_ii.sum()} of 20\n")
# The closest case: annotator 1 attack, training split.
sub = t2[(t2.group == 1) & (t2.split == "train")]
k, tot = label_counts(sub).get(calms21.TASK1_VOCAB["attack"], 0), int(sub.n_annotations.sum())
k_max = int(np.floor((3.2 + TOL) * tot / 100))
print(f"annotator 1 attack: {k} of {tot} frames = {100 * k / tot:.5f}%, {abs(100 * k / tot - 3.2):.5f} from 3.2; "
f"the largest count within 0.05 is {k_max} frames ({100 * k_max / tot:.5f}%, {k_max - k} more), "
f"{k_max + 1} frames give {100 * (k_max + 1) / tot:.5f}%; over Table 5's 139112 frames {100 * k / 139112:.5f}%")
reading a: train:
annotator frames n_other n_investigation n_mount n_attack other investigation mount attack paper largest_gap within_i within_ii
1 139122 83441 28628 22536 4517 59.9769 20.5776 16.1987 3.2468 (60.0, 20.6, 16.2, 3.2) 0.0468 4 4
2 135623 88848 35013 4609 7153 65.5110 25.8164 3.3984 5.2742 (65.5, 25.8, 3.4, 5.3) 0.0258 4 4
3 107420 74023 14496 10210 8691 68.9099 13.4947 9.5047 8.0907 (68.9, 13.5, 9.5, 8.1) 0.0099 4 4
4 108325 81495 12591 4827 9412 75.2319 11.6234 4.4560 8.6887 (75.2, 11.6, 4.5, 8.7) 0.0440 4 4
5 92383 64236 24403 949 2795 69.5323 26.4150 1.0272 3.0254 (69.5, 26.4, 1.0, 3.0) 0.0323 4 4
values within 0.05: class order (i) 20 of 20, best assignment (ii) 20 of 20
reading b: train+test:
annotator frames n_other n_investigation n_mount n_attack other investigation mount attack paper largest_gap within_i within_ii
1 425932 256093 96424 58965 14450 60.1253 22.6384 13.8438 3.3926 (60.0, 20.6, 16.2, 3.2) 2.3562 0 0
2 286542 195169 74719 5328 11326 68.1118 26.0761 1.8594 3.9526 (65.5, 25.8, 3.4, 5.3) 2.6118 0 0
3 184499 123693 30610 17444 12752 67.0426 16.5909 9.4548 6.9117 (68.9, 13.5, 9.5, 8.1) 3.0909 1 1
4 184499 119018 30928 16332 18221 64.5088 16.7632 8.8521 9.8759 (75.2, 11.6, 4.5, 8.7) 10.6912 0 0
5 456390 334396 106234 1479 14281 73.2698 23.2770 0.3241 3.1291 (69.5, 26.4, 1.0, 3.0) 3.7698 0 0
values within 0.05: class order (i) 1 of 20, best assignment (ii) 1 of 20
annotator 1 attack: 4517 of 139122 frames = 3.24679%, 0.04679 from 3.2; the largest count within 0.05 is 4521 frames (3.24967%, 4 more), 4522 frames give 3.25038%; over Table 5's 139112 frames 3.24702%
Result: reproduced. Under the training split with the class order of the Task 1 pie, all 20 percentages are within 0.05 of the figure's values. The closest case is annotator 1 attack, 4,517 of 139,122 frames (3.2468%, 0.047 from 3.2); it would stay within 0.05 with up to 4 more attack frames. Training and test together match 1 of 20.
table7-task3-label-fractions¶
In Task 3 the behaviors occupy 3.25% (approach), 1.47% (disengaged), 14.0% (groom), 19.3% (intromission), 0.90% (mount attempt), 4.74% (sniff face) and 9.29% (white rearing) of frames.
PAPER_T7 = {"approach": (3.25, 0.005), "disengaged": (1.47, 0.005), "groom": (14.0, 0.05), "intromission": (19.3, 0.05),
"mount_attempt": (0.90, 0.005), "sniff_face": (4.74, 0.005), "whiterearing": (9.29, 0.005)}
rows = []
for reading, splits in (("a: train", ["train"]), ("b: train+test", ["train", "test"])):
for b, (val, tol) in PAPER_T7.items():
sub = t3[(t3.group == b) & t3.split.isin(splits)]
pct, allf = label_percent(sub, {b: 1, "other": 0})
rows.append({"reading": reading, "behavior": b, "frames": label_counts(sub).get(1, 0), "of": allf,
"percent": pct[b], "paper": val, "tolerance": tol,
"within": abs(pct[b] - val) <= tol, "within_10pct": abs(pct[b] - val) <= 0.1 * val})
t7 = pd.DataFrame(rows)
print(t7.round(5).to_string(index=False))
print(t7.groupby("reading")[["within", "within_10pct"]].all().to_string())
reading behavior frames of percent paper tolerance within within_10pct
a: train approach 670 20624 3.24864 3.25 0.005 True True
a: train disengaged 524 35751 1.46569 1.47 0.005 True True
a: train groom 6319 45174 13.98813 14.00 0.050 True True
a: train intromission 3697 19200 19.25521 19.30 0.050 True True
a: train mount_attempt 423 46847 0.90294 0.90 0.005 True True
a: train sniff_face 913 19244 4.74434 4.74 0.005 True True
a: train whiterearing 3363 36181 9.29493 9.29 0.005 True True
b: train+test approach 4493 147092 3.05455 3.25 0.005 False True
b: train+test disengaged 752 54839 1.37129 1.47 0.005 False True
b: train+test groom 14859 201838 7.36184 14.00 0.050 False False
b: train+test intromission 11090 74418 14.90231 19.30 0.050 False False
b: train+test mount_attempt 896 132683 0.67529 0.90 0.005 False False
b: train+test sniff_face 15969 271037 5.89182 4.74 0.005 False False
b: train+test whiterearing 6001 54120 11.08832 9.29 0.005 False False
within within_10pct
reading
a: train True True
b: train+test False False
Result: reproduced. Under reading (a), the training split, all seven percentages are within tolerance (approach 3.249%, disengaged 1.466%, groom 13.988%, intromission 19.255%, mount attempt 0.903%, sniff face 4.744%, white rearing 9.295%).
dataset-size¶
The unlabeled set has 282 videos and about 6 million frames; the dataset holds over 1 million annotated frames, over 70 hours of tracked poses and over 10 hours of frame-by-frame annotation.
unl = sets[sets.task == "unlabeled"]
print(f"unlabeled: {len(unl)} sequences, {unl.n_frames.sum():,} frames ({unl.n_frames.sum() / 30 / 3600:.1f} h)")
print(f"unlabeled frames counted in keypoints: {unl.n_frames_keypoints.sum():,}; sequences where the two counts differ: "
f"{int((unl.n_frames_keypoints != unl.n_frames).sum())}")
labelled = sets[sets.task != "unlabeled"]
once = sets.drop_duplicates("kp_hash")
readings = {
"a: Task 1": sets[sets.task == "task1"],
"b: all labelled": labelled,
"c: all labelled, duplicates once": once[once.task != "unlabeled"],
}
rows = []
for nm, df in readings.items():
ann = df.n_frames.sum()
unl_part = unl if not nm.startswith("c") else once[once.task == "unlabeled"]
pose = ann + unl_part.n_frames.sum()
rows.append({"reading": nm, "annotated_frames": ann, "annotated_hours": ann / 30 / 3600,
"pose_frames": pose, "pose_hours": pose / 30 / 3600})
size = pd.DataFrame(rows)
print(size.round(2).to_string(index=False))
print("\nover 1 million annotated frames under:", size.reading[size.annotated_frames > 1e6].tolist())
print("over 10 hours of annotation under:", size.reading[size.annotated_hours > 10].tolist())
print("over 70 hours of pose under:", size.reading[size.pose_hours > 70].tolist())
print("282 unlabeled sequences:", len(unl) == 282, "; frames within 6 million +- 10%:", abs(unl.n_frames.sum() - 6e6) <= 6e5)
unlabeled: 282 sequences, 5,952,058 frames (55.1 h)
unlabeled frames counted in keypoints: 5,952,058; sequences where the two counts differ: 0
reading annotated_frames annotated_hours pose_frames pose_hours
a: Task 1 769845 7.13 6721903 62.24
b: all labelled 3243734 30.03 9195792 85.15
c: all labelled, duplicates once 2273674 21.05 8225732 76.16
over 1 million annotated frames under: ['b: all labelled', 'c: all labelled, duplicates once']
over 10 hours of annotation under: ['b: all labelled', 'c: all labelled, duplicates once']
over 70 hours of pose under: ['b: all labelled', 'c: all labelled, duplicates once']
282 unlabeled sequences: True ; frames within 6 million +- 10%: True
Result: reproduced. The unlabeled set has 282 sequences and 5,952,058 frames, counted the same in the keypoints and the scores. Over all labelled sets there are 3,243,734 annotated frames (30.0 h), or 2,273,674 (21.1 h) with each recording counted once, and 85.2 h or 76.2 h of pose with the unlabeled set; so "over 1 million annotated frames", "over 10 hours of annotation" and "over 70 hours of pose" hold under readings (b) and (c). Task 1 alone has 769,845 annotated frames (7.1 h), and with the unlabeled set 62.2 h of pose, so none of the three holds under reading (a).
table10-task3-difficulty¶
On Task 3, classifier performance depends on the percentage of frames in which a behavior is observed and on its average bout duration, with shorter bouts giving lower performance.
BEHAVIORS = ["approach", "disengaged", "groom", "intromission", "mount_attempt", "sniff_face", "whiterearing"]
F1_BASELINE = [0.338, 0.195, 0.289, 0.721, 0.034, 0.358, 0.430] # Table 10, baseline
F1_TASKPROG = [0.310, 0.198, 0.409, 0.609, 0.048, 0.274, 0.430] # Table 10, baseline with task programming
def exact_spearman(x, y):
# One-sided exact permutation p of Spearman's rho being at least the observed value.
rx, ry = stats.rankdata(x), stats.rankdata(y)
rho = stats.spearmanr(x, y)[0]
perms = [stats.spearmanr(rx, np.array(p))[0] for p in itertools.permutations(ry)]
return rho, float(np.mean(np.array(perms) >= rho - 1e-12))
rows, feats = [], {}
for split in ("train", "test"):
pct, bout = [], []
for b in BEHAVIORS:
g = t3[(t3.group == b) & (t3.split == split)]
p, _ = label_percent(g, {b: 1, "other": 0})
runs = [r for d in g.runs for r in d.get(1, [])]
pct.append(p[b]); bout.append(np.mean(runs) if runs else np.nan)
feats[split] = pd.DataFrame({"percent_frames": pct, "mean_bout_frames": bout}, index=BEHAVIORS)
for f1_name, f1 in (("baseline", F1_BASELINE), ("task programming", F1_TASKPROG)):
for feat in ("percent_frames", "mean_bout_frames"):
rho, p = exact_spearman(feats[split][feat].values, f1)
rows.append({"split": split, "F1": f1_name, "feature": feat, "rho": rho, "p_one_sided": p})
for split, f in feats.items():
print(f"{split} split:")
print(f.assign(F1_baseline=F1_BASELINE).round(3).to_string())
corr = pd.DataFrame(rows)
print("\nSpearman correlations across the seven behaviors (exact one-sided permutation p):")
print(corr.round(3).to_string(index=False))
train split:
percent_frames mean_bout_frames F1_baseline
approach 3.249 22.333 0.338
disengaged 1.466 7.594 0.195
groom 13.988 137.370 0.289
intromission 19.255 336.091 0.721
mount_attempt 0.903 30.214 0.034
sniff_face 4.744 24.676 0.358
whiterearing 9.295 29.500 0.430
test split:
percent_frames mean_bout_frames F1_baseline
approach 3.023 21.478 0.338
disengaged 1.194 12.667 0.195
groom 5.451 61.884 0.289
intromission 13.389 284.346 0.721
mount_attempt 0.551 24.895 0.034
sniff_face 5.980 38.116 0.358
whiterearing 14.705 29.640 0.430
Spearman correlations across the seven behaviors (exact one-sided permutation p):
split F1 feature rho p_one_sided
train baseline percent_frames 0.786 0.024
train baseline mean_bout_frames 0.321 0.249
train task programming percent_frames 0.929 0.003
train task programming mean_bout_frames 0.536 0.118
test baseline percent_frames 0.929 0.003
test baseline mean_bout_frames 0.607 0.083
test task programming percent_frames 0.857 0.012
test task programming mean_bout_frames 0.679 0.055
Result: partially reproduced. Across the seven behaviors, the baseline F1 rises with the percentage of labelled frames (Spearman 0.786, p = 0.024 on the training split; 0.929, p = 0.003 on the test split). It also rises with mean bout length, but not significantly (0.321, p = 0.249; 0.607, p = 0.083). With the task programming model's F1, the bout-length correlations are 0.536 (p = 0.118) and 0.679 (p = 0.055).
Summary¶
A failure to reproduce here may be a failure of this attempt, not of the paper: see the reasoning and the differences from the paper's method.
| Claim | Data | Outcome |
|---|---|---|
| format-keypoints | dandiset | partially reproduced (no confidence values) |
| resident-is-black | dandiset | reproduced |
| task1-vocab | dandiset | reproduced |
| task1-video | dandiset | reproduced |
| contact-during-social-behaviors | dandiset | reproduced (attack absent from the session) |
| pose-errors-close-fast | dandiset | partially reproduced (close yes, fast reversed) |
| link-mouse071 | with inputs | reproduced |
| pose-errors-close-fast-confidence | with inputs | partially reproduced (close yes, fast reversed) |
| table5-task1-counts | elsewhere | reproduced |
| table5-task2-counts | elsewhere | partially reproduced (one count off by 10 frames) |
| table5-task3-counts | elsewhere | reproduced |
| table6-task1-label-fractions | elsewhere | partially reproduced (attack 2.76501%, 0.00001 outside the tolerance) |
| fig4b-task2-label-fractions | elsewhere | reproduced |
| table7-task3-label-fractions | elsewhere | reproduced |
| dataset-size | elsewhere | reproduced |
| table10-task3-difficulty | elsewhere | partially reproduced (bout length not significant) |
| table1-architectures, table2-task-programming, table3-task2-style-transfer, table4-task3-new-behaviors, fig7-errors-at-transitions, fig11-annotator-similarity | elsewhere | blocked: methods underspecified |
How the analysis differs from the paper's. The paper describes a dataset and benchmarks models on it. The dandiset holds one of its sequences, so the claims about the dataset as a whole were tested on the CaltechDATA deposit. The paper gives no measure for "errors in pose estimation", "close contact" or "moving very quickly", nor a test of how Task 3 performance depends on frequency and bout length; the measures, thresholds and tests used here were fixed in the claims file before the analysis. Tolerances for counts and percentages of the deposited files are 0 and half a unit of the last reported digit, tighter than the protocol's default of 10%. No model was trained: the baseline code is not reachable and the paper leaves the architectures' settings to it.
What was found about the dandiset.
- The CalMS21 part of the dandiset is one sequence of the dataset, Task 1 test sequence mouse071; its keypoints and frame annotations are identical to CaltechDATA's, and its video has the same frame count, size and rate as the pose and shares its clock.
- The MARS confidence scores, which CalMS21 distributes with the pose, are missing: the field is present and every value is NaN.
- The behavior catalogue has empty definitions, though the paper defines all three behaviors.
- The subject is the dyad stored as one subject with strain C57BL/6 only, sex unknown and age
P0D/; the session start time is a placeholder (2024-01-01). The dandiset'scentroid_speedis the speed of the midpoint of the two mice, which its description does not make clear. - The video was re-encoded (184 MB against CaltechDATA's 964 MB for the same sequence); the dandiset does not say so.
- The dandiset links the CalMS21 dataset DOI but not the paper (arXiv:2104.02710).
Outside the dandiset, the CaltechDATA deposit holds many recordings more than once across tasks, including across the training and test splits, which the paper does not mention; one Table 5 count differs from the data by 10 frames; and the paper's datasheet gives a different licence (CC-BY-NC-SA) from the deposit and the dandiset (CC-BY 4.0), and says the raw videos are not included while Section 3.1 and the deposit include the Task 1 videos.
print(f"notebook ran in {(time.time() - started_all) / 60:.1f} min")
notebook ran in 4.5 min