Dandisets / 001878

Dandiset 001878

✓ Verified by an independent agent

Standardized behavioral segmentation datasets

House mouse Converted by CatalystNeuro 230 MB 6 files
  • 4 reproduced
  • 2 partially reproduced

Counted separately: 2 claims that also use data from outside the dandiset.

  • 1 reproduced
  • 1 partially reproduced

Counted separately: 14 claims on data the paper deposited in other archives.

  • 5 reproduced
  • 3 partially reproduced
  • 6 blocked

View on DANDI Paper JSON How to read outcomes

Claims

From the paper CalMS21 - Caltech Mouse Social Interactions (Sun et al., 2021)

Reproducing Sun et al. (2021) from dandiset 001878 and the CalMS21 deposit 16

Notebook reproduce_sun_2021.ipynb

  • Partially reproducedSection 3 and Figure 3; Appendix C (data format); Appendix D.1

    Each frame holds seven MARS keypoints (nose, left ear, right ear, neck, left hip, right hip, tail base) for each of two mice, the resident (mouse 0) and the intruder (mouse 1), in pixel coordinates of a 1024 x 570 image, at 30 Hz, with a confidence score from 0 to 1 for each keypoint.

    The file holds 14 keypoint series named for the seven parts of the resident and the intruder, in pixels, at 30 Hz, with every coordinate inside the 1024 x 570 image (x 5.0 to 991.2, y 9.0 to 563.3). The confidence field is present for every keypoint but all 333,340 values are NaN, so the confidence part of the claim fails.

    How the test differed from the paper’s method

    • Judged on the one CalMS21 sequence in the dandiset.
  • Partially reproducedAppendix B (Composition, errors and noise)

    Errors in pose estimation are most common on frames when the two animals are in close contact or moving very quickly.

    657 of 23,810 frames (2.8%) have a body length outside the session's quartiles widened by 3 IQR. In the closest 10% of frames (centroid distance at most 66.4 px) the fraction is 0.131 against 0.016 elsewhere (difference 0.115, 95% interval 0.064 to 0.179). In the fastest 10% (at least 389 px/s) it is 0.010 against 0.030 (difference -0.020, interval -0.033 to -0.008), the reverse of the claim. Both directions hold at 5% and 20% and with a factor of 1.5. With the speed taken as the running median of the larger speed per frame, the fast difference is -0.020 (interval -0.033 to -0.008) at 10% and -0.011 (interval -0.025 to 0.001) at 20%. Error fractions by label are 0.045 (investigation), 0.105 (mount) and 0.009 (other), so a hunched posture in mount could give real outliers. Checked against the MARS confidence of the same frames, flagged close frames have a lower confidence than unflagged close frames (median 0.574 against 0.657, difference -0.082, interval -0.107 to -0.050), and in each of eight flagged close frames drawn at random and shown with the video, at least one segment visibly runs across the other mouse or the background (judged by eye). The flags in close frames therefore mark mostly tracking errors, though the share of real postures among them is not measured.

    How the test differed from the paper’s method

    • The dandiset has no confidence values, so pose errors are counted with an anatomical proxy chosen here; the paper gives no measure of pose error.
    • The thresholds for close and fast frames were chosen here.
  • Partially reproducedAppendix B (Composition, errors and noise)uses data from outside the dandiset

    Errors in pose estimation are most common on frames when the two animals are in close contact or moving very quickly, as shown by the MARS confidence scores.

    The mean confidence per frame (CaltechDATA scores, matched frame by frame) is lower in the closest 10% of frames (median 0.649 against 0.740, difference -0.091, 95% interval -0.107 to -0.074), most of all for the intruder (-0.149 against -0.031 for the resident). In the fastest 10% of frames it is higher (0.751 against 0.732, difference 0.018, interval 0.012 to 0.025), the reverse of the claim. Both hold at 5% and 20%. The confidence correlates with the body-length error flags of pose-errors-close-fast (point-biserial r = -0.366). With the speed taken as the running median of the larger speed per frame, the fast difference is 0.018 (interval 0.012 to 0.025). The link holds in full, so the match of scores to frames is exact.

    How the test differed from the paper’s method

    • Confidence is a proxy for pose error; the paper gives no measure.
    • The confidence scores come from CaltechDATA, as the dandiset's are missing.
  • Partially reproducedTable 5; Section 3.2data in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Task 2 has, for annotators 1 to 5, 6 training sequences each with 139,112, 135,623, 107,420, 108,325 and 92,383 frames, and 13, 6, 4, 4 and 20 test sequences with 286,810, 150,919, 77,079, 76,174 and 364,007 frames.

    19 of 20 counts match exactly. Annotator 1's training set has 139,122 frames where Table 5 gives 139,112 (10 frames, 0.007%); its six sequences match. The six sequences have the same frame counts in scores, keypoints and annotations; the paper gives no per-sequence counts, so the sequence that holds the 10 frames is not known. Annotator ids in the metadata agree with those in the keys.

  • Partially reproducedTable 6; Figure 4Bdata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    In the Task 1 training set, 2.76% of frames are labelled attack, 28.9% investigation, 5.64% mount and 62.7% other.

    Under reading (a), the training split, investigation, mount and other are within half a unit of the last reported digit; attack is 14,039 of 507,738 frames, 2.765009%, 0.005009 from 2.76 against a tolerance of 0.005. It rounds to 2.77 and truncates to the reported 2.76. All four are within 10% of the reported values. Reading (b), training and test, gives 3.46, 27.00, 7.85 and 61.68% and does not match.

    How the test differed from the paper’s method

    • The tolerance of half a unit of the last digit is tighter than the protocol's default of 10%, because these are percentages of a fixed file; under the default the claim would be reproduced.
  • Partially reproducedSection 4.4; Tables 7 and 10data in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    On Task 3, classifier performance depends on the percentage of frames in which a behavior is observed and on its average bout duration, with shorter bouts giving lower performance.

    Across the seven behaviors the baseline F1 of Table 10 rises with the percentage of labelled frames (Spearman 0.786, exact one-sided p = 0.024 on the training split; 0.929, p = 0.003 on the test split) and with mean bout length, but the latter is not significant (0.321, p = 0.249; 0.607, p = 0.083). With the task programming model's F1 the bout-length correlations are 0.536 (p = 0.118) and 0.679 (p = 0.055). The direction of both parts of the statement holds; only the frequency part is significant with seven behaviors.

    How the test differed from the paper’s method

    • The paper states the dependence without a test; the correlation and the permutation test were chosen here, using the paper's published F1 scores with frequencies and bout lengths computed from the data.
  • ReproducedAppendix C; Appendix B (Collection Process); Appendix D.1

    Mouse 0, the resident, is the black C57Bl/6J mouse, and mouse 1, the intruder, is the white BALB/c mouse.

    In 746 of 794 sampled frames (every 30th) the body centres are at least 60 px apart; in all of them the 12 px patch on the resident is darker than the one on the intruder (median grey level 46.4 against 250.9). With 8 and 24 px patches the fractions are 0.997 and 1.000, with separations of 30 and 120 px 0.999 and 1.000, and 1.0 in every minute.

    How the test differed from the paper’s method

    • The coat colour test (grey level of the video at the body centre) was chosen here; the paper states the identities without a test.
  • ReproducedSection 3; Appendix C (metadata and vocab)

    Task 1 frames are annotated by one annotator (annotator id 0) with exactly one label per frame from the vocabulary attack 0, investigation 1, mount 2 and other 3.

    The catalogue holds attack 0, investigation 1, mount 2 and other 3; the bouts give the annotator as "CalMS21 annotator 0"; the 170 bouts tile the session from 0 to 793.67 s with no gaps, so all 23,810 frames have exactly one label. The session has no attack bouts.

  • ReproducedSection 3.1; Appendix D.1

    The source videos of Task 1 are released, recorded at 30 Hz and 1024 x 570 pixels.

    The video is 1024 x 570, h264, at 30 frames per second with 23,810 frames, equal to the pose. Its motion energy correlates best with the pose speed at lag 0 (r = 0.754, falling to 0.19 and 0.26 at 15 frames either way). It is 184,059,454 bytes against 964,451,532 for CaltechDATA's mouse071_task1_annotator1.mp4, so it was re-encoded; the two were not compared frame by frame.

  • ReproducedAppendix D.2 (behavior definitions)

    Close investigation, attack and mount are behaviors in which the resident is in close contact with the intruder; other frames include the animals apart.

    Median distance from the resident's nose to the nearest intruder keypoint is 21.4 px in 4,679 investigation frames and 15.0 px in 2,894 mount frames, against 376.0 px in 16,237 other frames (differences -354.6 and -361.0 px, 95% block bootstrap intervals -408.9 to -295.2 and -413.6 to -303.8, with blocks of contiguous session time). The centroid distance gives -307.2 and -355.5 px, with intervals excluding 0, and each half of the session gives the same order. Attack is absent from the session, so the attack part of the statement is untested.

    How the test differed from the paper’s method

    • The paper defines the behaviors in words; the distance measure, the comparison with other frames and the bootstrap were chosen here.
    • One session, with no attack frames.
  • ReproducedTable 5; Section 3.1; Figure 4Adata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Task 1 has 70 training sequences with 507,738 frames and 19 test sequences with 262,107 frames.

    The training file holds 70 sequences with 507,738 frames and the test file 19 with 262,107, as reported. Annotation lengths equal the frame counts of the scores in every sequence, and the frame counts of the keypoints and the scores agree in all 574 sequences of the deposit.

  • ReproducedTable 5; Section 3.3; Figure 4Adata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Task 3 has, for approach, disengaged, grooming, intromission, mount attempt, sniff face and white rearing, 3, 2, 2, 1, 4, 3 and 2 training sequences with 20,624, 35,751, 45,174, 19,200, 46,847, 19,244 and 36,181 frames, and 25, 1, 13, 10, 12, 47 and 1 test sequences with 126,468, 19,088, 156,664, 55,218, 85,836, 251,793 and 17,939 frames.

    All 28 counts match Table 5.

  • ReproducedFigure 4Bdata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    The five Task 2 annotators label different fractions of frames as other, investigation, mount and attack (annotator 1 60.0, 20.6, 16.2 and 3.2%; annotator 2 65.5, 25.8, 3.4 and 5.3%; annotator 3 68.9, 13.5, 9.5 and 8.1%; annotator 4 75.2, 11.6, 4.5 and 8.7%; annotator 5 69.5, 26.4, 1.0 and 3.0%).

    Under the training split with the class order of the Task 1 pie (other, investigation, mount, attack), all 20 percentages are within 0.05 of the figure's values; the best assignment of classes gives the same 20. The closest case is annotator 1 attack, 4,517 of 139,122 frames (3.2468%, 0.047 from 3.2), which would stay within 0.05 with up to 4 more attack frames. Training and test together match 1 of 20 under either class reading.

    How the test differed from the paper’s method

    • The Task 2 pies carry no class names; the order was read from the Task 1 pie, and the alternative reading (best assignment) was listed before the analysis. Both match.
  • ReproducedTable 7; Figure 4Bdata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    In Task 3 the behaviors occupy 3.25% (approach), 1.47% (disengaged), 14.0% (groom), 19.3% (intromission), 0.90% (mount attempt), 4.74% (sniff face) and 9.29% (white rearing) of frames.

    Under reading (a), the training split, all seven are within tolerance. Training and test together give values outside 10% for five of seven (for example groom 7.36%).

  • ReproducedAbstract; Section 3; Figure 4A; Section 5 (Discussion)data in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    The unlabeled set has 282 videos and about 6 million frames; the dataset holds over 1 million annotated frames, over 70 hours of tracked poses and over 10 hours of frame-by-frame annotation.

    The unlabeled set has 282 sequences and 5,952,058 frames (55.1 h), counted the same in the keypoints and the scores. Annotated frames are 769,845 (7.1 h) in Task 1, 3,243,734 (30.0 h) over all labelled sets, and 2,273,674 (21.1 h) with each recording counted once; tracked pose including the unlabeled set is 62.2, 85.2 and 76.2 h. So the three "over" statements (1 million annotated frames, 10 hours of annotation, 70 hours of pose) hold under readings (b) and (c) and none holds under reading (a).

    How the test differed from the paper’s method

    • The paper does not say which sets its totals cover; three readings were listed before the analysis.

Not tested in a notebook 6

6 methods underspecified
  • BlockedTable 1; Section 4.1· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Among baseline architectures trained on Task 1, the 1D convolutional network performs best (F1 0.793, MAP 0.856), ahead of fully connected (0.659), LSTM (0.675) and self-attention (0.610) models.

    Needs the four baseline networks trained on Task 1. The paper gives some settings (input window, skip factor, the ranges searched) and refers to the project code for the rest, but the code at gitlab.aicrowd.com/aicrowd/research/mab-e/mab-e-baselines did not answer, with or without the sandbox, and no copy was found in Software Heritage or on GitHub; the web archive holds its landing page only. The data are public on CaltechDATA. Training five runs of each network was also not attempted within the time limit.

  • BlockedTable 2 and Table 8; Section 4.2· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Adding task programming features learned from the unlabeled set raises Task 1 F1 from 0.793 to 0.829, and the challenge's top entry reaches 0.864.

    Needs the baseline network (code unavailable, see table1-architectures) and the challenge's top entry, whose code is not linked. The task programming code (GitHub neuroethology/TREBA) and precomputed features on CaltechDATA are public, but without the baseline classifier the comparison cannot be repeated.

  • BlockedTable 3 and Table 9; Section 4.3 and Appendix G.2· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    On Task 2 the fine-tuned baseline reaches an annotator-averaged F1 of 0.754, 0.774 with task programming features and 0.809 for the top entry, and all models do better on annotators 1 and 2 than on 3, 4 and 5.

    Needs the fine-tuned baseline and the top entry; the baseline code is unavailable (see table1-architectures) and the top entry's code is not linked.

  • BlockedTable 4 and Table 10; Section 4.4· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    On Task 3 the baseline reaches a behavior-averaged F1 of 0.338 (0.328 with task programming features, 0.319 for the top entry), far below Tasks 1 and 2.

    Needs the fine-tuned baseline with per-behavior learning rates hand-tuned on a validation set, and the top entry; neither's code is available (see table1-architectures).

  • BlockedFigure 7; Section 4.2· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    Many prediction errors of the baseline models fall around the boundaries between behaviors.

    Needs the baseline model's predictions; its code is unavailable (see table1-architectures), so the claim depends on a blocked claim. The predictions are also not deposited in either archive (data_missing).

  • BlockedFigure 11; Section 4.3 and Appendix G.2· Methods underspecifieddata in CaltechDATA doi:10.22002/D1.1991 (record s0vdx-0k302, version 1.0)

    The annotator embedding learned by the top Task 2 entry places annotators 3 and 4 close to each other and apart from annotators 1, 2 and 5.

    Needs the trained top entry's annotator embedding; the entry's code is not linked and the paper describes the model only in outline. The learned matrix is also not deposited in either archive (data_missing).

Findings about the data

What attempts found about the dandiset itself, for its owner and the archive.

  • Data missingThe MARS confidence scores, which CalMS21 distributes with every keypoint, are missing from the dandiset's CalMS21 file; the confidence field of all 14 pose series is present but all 333,340 values are NaN. CaltechDATA's copy of the same sequence has them (0.03 to 0.98).
  • Selection not documentedFor CalMS21 the dandiset holds one sequence, the Task 1 test sequence mouse071, out of 574 sequences on CaltechDATA. Its description presents a collection of datasets and does not say that only one CalMS21 sequence is included or why this one was chosen.
  • Description missingThe behavior catalogue (CalMS21_ethogram) has an empty definition for attack, investigation, mount and other, though the paper defines the three behaviors (Appendix D.2) and states that annotations refer to behaviors initiated by the resident.
  • Description missingThe two mice are stored as one subject with strain C57BL/6, sex unknown and age P0D/; the intruder's strain (BALB/c) and the residents' sex (male) are known from the paper but not recorded, and which series belongs to the black or white mouse is given only by the series names.
  • Metadata errorThe session start time is 2024-01-01T00:00, a placeholder; CalMS21 gives no recording dates.
  • Description missingThe `centroid_speed` series, and the bout columns mean_speed, peak_speed and immobility_fraction that reduce it, are the speed of the mean of all 14 keypoints, which is the midpoint of the two mice, not the speed of either animal. The description ("the pose centroid, mean over keypoints") does not make this clear.
  • OtherThe dandiset's mouse071 video (184 MB) is a re-encoding of CaltechDATA's (964 MB). It has the same frame count, size and rate, and its motion follows the pose at lag 0, but the dandiset does not say it was re-encoded.
  • Description missingThe dandiset links the CalMS21 dataset DOI (10.22002/D1.1991) as IsDerivedFrom, but not the paper that describes the dataset and its benchmarks (arXiv 2104.02710, NeurIPS 2021 Datasets and Benchmarks).
  • OtherOn CaltechDATA, 574 CalMS21 sequences hold 450 distinct recordings (identical keypoints over the whole sequence). 93 of the 109 Task 3 test sequences are Task 1 training recordings, five Task 1 test recordings (mouse071 among them) appear as six Task 3 training sequences, and four Task 2 recordings are in the training set of one of annotators 3 and 4 and the test set of the other. The paper describes each sequence as an independent observation and does not mention the overlap, and the CaltechDATA readme says that training and test sets are split by animal identity.
  • Paper inconsistentThe paper's datasheet says the dataset does not include the raw videos, while Section 3.1 says all source videos of Task 1 are released, and the deposit holds them. The datasheet also gives the licence as CC-BY-NC-SA, while the CaltechDATA record and the dandiset give CC-BY 4.0.
  • OtherTable 5 gives 139,112 training frames for Task 2 annotator 1; the deposited file holds 139,122. The other 47 counts of Table 5 match.
  • Methods not availableThe baseline code the paper links (gitlab.aicrowd.com/aicrowd/research/mab-e/mab-e-baselines) is not reachable and no archived copy of the repository was found; the code of the challenge's top entries is not linked. The paper leaves the architectures' settings to that code, so none of the benchmark results can be repeated.

Notebooks

Every registered entry that uses this dandiset. An entry can hold several notebooks, and can use more than one dandiset.

Reproduction CalMS21 mouse social behavior dataset (Sun et al. 2021)

Tests the claims of Sun et al. (2021), the paper describing the CalMS21 dataset of resident-intruder pose and behavior annotations, on dandiset 001878, which holds one CalMS21 sequence (Task 1 test sequence mouse071) with its video, and on the full dataset deposited on CaltechDATA. It checks the format, identities and annotations of the dandiset's sequence, links it to the deposit, and tests the dataset's counts and label fractions. The model benchmarks are not run. All files

Verification

  • Round 1
    2026-10-19    Agreed with 22 of 22 outcomes. Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook). Reviewed: The executed notebook of the first run with the conclusion cells and the summary removed, the helper module and the claims file (commit cf06231).
  • Round 2
    2026-10-19    Agreed with 22 of 22 outcomes. Independent agent run by CatalystNeuro (same harness, separate context, did not write the notebook). Reviewed: The full executed notebook of the second run, with its conclusion cells and summary, the helper module, the claims file and the attempt record (commit c209e2e).

Method differences

How the tests differed from the paper’s method or data, and the claims each applies to.

  • Judged on the one CalMS21 sequence in the dandiset.format-keypoints
  • The coat colour test (grey level of the video at the body centre) was chosen here; the paper states the identities without a test.resident-is-black
  • The paper defines the behaviors in words; the distance measure, the comparison with other frames and the bootstrap were chosen here.contact-during-social-behaviors
  • One session, with no attack frames.contact-during-social-behaviors
  • The dandiset has no confidence values, so pose errors are counted with an anatomical proxy chosen here; the paper gives no measure of pose error.pose-errors-close-fast
  • The thresholds for close and fast frames were chosen here.pose-errors-close-fast
  • Confidence is a proxy for pose error; the paper gives no measure.pose-errors-close-fast-confidence
  • The confidence scores come from CaltechDATA, as the dandiset's are missing.pose-errors-close-fast-confidence
  • The tolerance of half a unit of the last digit is tighter than the protocol's default of 10%, because these are percentages of a fixed file; under the default the claim would be reproduced.table6-task1-label-fractions
  • The Task 2 pies carry no class names; the order was read from the Task 1 pie, and the alternative reading (best assignment) was listed before the analysis. Both match.fig4b-task2-label-fractions
  • The paper does not say which sets its totals cover; three readings were listed before the analysis.dataset-size
  • The paper states the dependence without a test; the correlation and the permutation test were chosen here, using the paper's published F1 scores with frequencies and bout lengths computed from the data.table10-task3-difficulty

Provenance

Attempt record
001878-sun-2021-01
Paper
CalMS21 - Caltech Mouse Social Interactions (Sun et al., 2021)
Date
2026-10-19
Made by
An agent, run by CatalystNeuro
Dandiset version
draft
Harness
Claude Code (GitHub Actions)
Model
claude-opus-5-5
Protocol
pilot-7

Transcripts of the agent runs

Each run’s prompt, messages and tool calls, with long output cut and the text of the paper left out, and its final report.

← Dandiset 001754Dandiset 001888 →