DANDI:000004

Reproducing Chandravadia et al. (2020) from dandiset 000004 24

Notebook reproduce_chandravadia_2020.ipynb

• ✕Not reproducedData Records, interval group, paragraph 1

  In the trials table, new_old_labels_recog is 0 for old stimuli (seen during encoding) and 1 for new stimuli.

  New_old_labels_recog provides the ground truth label of whether the trial showed a new or old stimulus during the recognition phase (0 is old, 1 is new).

  Data Records, interval group, paragraph 1

  The labels are coded the other way. In 79 of 87 sessions every label-1 image and no label-0 image was shown during encoding; in no session do label-0 images match the encoding block. Label 1 draws "old" responses (4 to 6) more often than label 0 in 86 of 87 sessions (medians 0.70 and 0.24, sign test p = 1.1e-24). In the 8 sessions with short encoding blocks, 52% to 98% of the label-1 images are among the stored encoding trials and no label-0 image is. The paper's code reads 1 as old, and every analysis here does the same; with the stated coding the mean AUC is 0.247.

• ✕Not reproducedData Records, general group, paragraph 1; Methods, data acquisition, paragraph 1

  The electrodes table gives each electrode's brain area, MNI coordinates, impedance, the filter applied before spike sorting (300 to 3000 Hz) and the original hardware channel.

  General/extracellular_ephys contains information about the electrodes recorded from, including their location (brain area and coordinates), impedance, and filters used ... The origChannel (a custom column) refers to the hardware channel that was used to record from this electrode.

  Data Records, general group, paragraph 1; Methods, data acquisition, paragraph 1

  Location, filtering (300-3000Hz for all) and origChannel are filled for all 887 electrodes. The impedance column is stored as NaN (float64) for all 887. Coordinates are missing for 99 electrodes in sessions 7, 20, 21, 23, 24, 50, 52 and 2004; where given, the sign of x matches the hemisphere of the location for all 788.

• ✕Not reproducedTechnical Validation, proportion of selective cells, paragraphs 2 and 3; Figure 5f

  146 of the 1863 cells (7.8%) are memory selective, by a two-tailed bootstrap comparison of mean firing rates (1,000 runs, P < 0.05) between the 50 novel and 50 familiar recognition trials in a 1 s window starting 200 ms after stimulus onset.

  To select for an MS cell, we compared the firing rate between the 50 novel and 50 familiar trials during the retrieval period ... 146 cells qualified as MS cells (7.8%)

  Technical Validation, proportion of selective cells, paragraphs 2 and 3; Figure 5f

  The Methods' test (all 50 new and 50 old trials, 200 to 1200 ms) selects 110 of 1864 units (5.9%), against the paper's 146 (7.8%). With labels shuffled (10 per unit, 18640 tests) 5.39% are selected (95% CI 5.07% to 5.73%); 110 is not distinguishable from this null (one-sided binomial p = 0.18) and only just above 5% (p = 0.044). A second bootstrap seed gives 118 (6.3%), also outside tolerance; 22 units are selected under one seed only. Correct trials only in the Methods' window give 143 and 140 under the two seeds, and the code's choices (correct trials only, 200 to 1700 ms) 147 and 149, all within tolerance (147 against 5%, p = 7.1e-08). With labels shuffled among the correct trials in the code's window the null is 5.58% (95% CI 5.26% to 5.92%), and 147 exceeds it (104.1 expected, one-sided binomial p = 2.5e-05). All trials in the code's window give 108 and 109. The paper's number therefore follows its code's restriction to correct trials, which the Methods do not mention, and the outcome comes from that disagreement, not from the data. It is judged on the Methods' procedure, as the protocol requires. Per-unit p-values of the 288 units selected under some reading or seed are printed.

  How the test differed from the paper’s method

  • The paper's code restricts the test to correct trials and uses a 1.5 s window; the Methods describe all 50 and 50 trials and a 1 s window. The Methods give the main result.
  • The bootstrap uses the code's algorithm with a fixed seed.
• ✕Not reproducedTechnical Validation, proportion of selective cells, paragraph 3; Figure 5f

  26 cells (1.4%) are both visually and memory selective.

  26 cells qualified as both VS and MS cells (1.4%) (Fig. 5f).

  Technical Validation, proportion of selective cells, paragraph 3; Figure 5f

  Under the Methods' tests 12 of 1864 units (0.6%) are both VS and MS, against 19.6 expected if the two were independent and the paper's 26 (1.4%); 14 with the second bootstrap seed. The code's choices give 26 (1.4%) under both seeds. Counting duplicated spike trains once, 11 of 1809. As for ms-cells-proportion, the outcome comes from the Methods disagreeing with the code.

  How the test differed from the paper’s method

  • Depends on ms-cells-proportion, whose main test follows the Methods and not the code.
• ◐Partially reproducedOnline-only Table 1

  Online-only Table 1 gives the age and sex of every subject; there are 25 female and 34 male subjects.

  Total Female: 25 ... Total Male: 34

  Online-only Table 1

  Age and sex match for 56 of 59 subjects. C24 is 57 in the files and 47 in the table, H33 is 30 and 29, and H48 is female aged 32 in the files and male aged 54 in the table. The files have 26 female and 33 male subjects (table 25 and 34), within tolerance. Each subject has one age and one sex across its files. The diagnosis text differs for 19 of 59 subjects. For 8 the files give a broader category; for 6 (H10, H16, H17, H19, C58, T89) they give Lateral Frontal, which adds Lateral to the table's Frontal except for H17 (Left Inferior Frontal in the table). Five differ in substance, C61 (Left Lateral Frontal in the files, Left Mesial Temporal in the table), C34 (Bilateral Independent Temporal against Bilateral Mesial Temporal), C56 and H51, where the files leave out part of the table's diagnosis (orbitofrontal, temporal), and T90, where they add Bilateral. All 87 files have the age and sex of their section in the code's session list, defineNOsessions_release.ini, as printed in the notebook.

• ◐Partially reproducedMethods, Task, paragraph 1; Data Records, stimuli group, paragraph 1; Technical Validation, proportion of selective cells, paragraph 2

  Each session has 100 encoding trials showing 100 novel images and 100 recognition trials, of which 50 show new and 50 old images, and the images come from 5 visual categories.

  In the encoding phase, subjects were presented with 100 novel images ... in the recognition phase, subjects were presented with 50 “novel” images and 50 “old” images ... (there are 5 different categories in each experiment)

  Methods, Task, paragraph 1; Data Records, stimuli group, paragraph 1; Technical Validation, proportion of selective cells, paragraph 2

  All 87 sessions have 100 recognition trials with 50 per label, 5 categories (ids and names) and one stored image per trial timestamped at its onset. 79 sessions have 100 encoding trials with 100 distinct images; sessions 18, 20, 21, 23, 26, 27 and 44 have 50 and session 123 has 97. In these eight sessions 2% to 48% of the images shown as old have no encoding trial in the file. The images are stored as one ImageSeries per file, not as stimuli_learn_x and stimuli_recog_x.

• ◐Partially reproducedData Records, acquisition group, paragraphs 1 and 2; Table 1

  The events stream holds the TTL markers 55, 1, 2, 3, 20 or 21, 31 to 36, 6 and 66 with the meanings of Table 1, and the events and experiment_ids streams have one entry each, in the same order.

  Both streams include the same number of entries in the same order. ... The following TTL values are used: Start of Experiment (55), Stimulus Onset (1), Stimulus Offset (2), Question Screen Onset (3), New/Old Response (20 or 21), Confidence of Response (31–36), End of Trial (6), End of Experiment (66).

  Data Records, acquisition group, paragraphs 1 and 2; Table 1

  Events and experiment_ids have the same length and timestamps in all 87 files, and 17019 of 17047 trials show the codes 1, 2, 3, a response and 6 in order. 78 files meet every condition. Codes not in Table 1 occur in sessions 43 (98, 99), 58 (23), 93 (99), 123 (0), 127 (0), 128 (4), 2001 (22, 23), 2002 (23) and 2003 (23). Seven files (sessions 92, 93, 96, 97, 99, 100, 101) have no start code 55 and session 2001 has a third end code 66; these do not enter the criterion. In every trial that shows the sequence the response is of its phase's kind (20 or 21 in learning, 31 to 36 in recognition), 17019 of 17019. In 6 files 1 to 18 trials have a response event that differs from the trials table's response_value; the analysis uses response_value.

  How the test differed from the paper’s method

  • The criterion did not give a number for most files in a partial result; 78 of 87 was taken as most.
• ◐Partially reproducedData Records, acquisition group, paragraph 3; Table 2

  The learning block is labelled with one of 80, 83 or 88 and the recognition block with one of 81, 84 or 89.

  The learning block is labeled with only one of the following: 80, 83, or 88, while the recognition block is labeled with only one of the following: 81, 84, or 89

  Data Records, acquisition group, paragraph 3; Table 2

  83 of 87 files use the ids of Table 2. Session 55 uses 80 and 82, session 77 uses 81 and 82, and sessions 100 and 105 use 82 and 83 (learning and recognition, identified by their responses). The files' data_collection field states the same ids.

• ◐Partially reproducedData Records, structure of the NWB file, paragraph 2

  To protect patient information, session_start_time keeps only the year and month of the experiment, with the day set to the first of the month for all sessions.

  in the field session_start_time, we set only the year and month of the experiment but defaulted the actual day of the experiment to the first of the month for all sessions.

  Data Records, structure of the NWB file, paragraph 2

  86 of 87 files start on the first of a month; session 48 starts on 2010-03-13. Session 68 is dated 2000-01-01, outside the 2006 to 2019 span of the others.

• ◐Partially reproducedData Records, units group, paragraph 1

  The units table holds spike_times, waveform_mean_encoding, waveform_mean_recognition (sampled at 100 kHz), origClusterID, SNR, IsolDist and electrodes, with one electrode per neuron.

  We use the following columns: spike_times_index, spike_times, waveform_mean_encoding, waveform_mean_recognition, origClusterID, SNR, IsolDist, and electrodes. ... (sampled at 100 kHz) ... here this link is 1:1 since each neuron is only visible on one channel.

  Data Records, units group, paragraph 1

  All 87 files hold every column (IsolDist is named IsolationDist) and link each unit to one electrode. The stated waveform sampling rate is 98400 Hz, within tolerance of 100 kHz, with 256 samples. SNR is finite for 1778 of 1864 units (11 infinite, 75 NaN); the isolation distance is NaN for 493 units and above 1e4 for 4 (up to 8.08e+32). Spike times are not in increasing order for 13 units in sessions 21, 58, 68, 70, 73, 77 and 128, and units 35 and 37 of session 58 hold every spike time twice.

• ◐Partially reproducedMethods, data acquisition, paragraph 2

  Clusters were accepted as units only when fewer than 3% of their interspike intervals were shorter than 3 ms.

  To classify the detected clusters as putative units, we assessed the following criteria: ... (3) violation of the refractory period (<3% of the spikes have an ISI of less than 3 ms)

  Methods, data acquisition, paragraph 2

  1858 of 1864 units (99.7%) have fewer than 3% of interspike intervals below 3 ms (median 0.19%). Six units are at or above 3%, four at or above 5% and two above 20% (maximum 50.1%); the two above 20% are the session 58 units whose spike times repeat. With the spike times as stored, 1860 units pass. All 77 TWH units pass.

• ◐Partially reproducedTechnical Validation, proportion of selective cells, paragraph 3; Figure 5f (the values as written into plotFigure5.ipynb at commit 1d95162)

  The percentages of VS, MS and dual cells are comparable between CS sessions (18.4%, 7.7% and 1.4%) and the new TWH sessions (19.4%, 10.4% and 1.3%).

  We found that the percentages of VS, MS, and dual VS/MS cells were comparable to previously reported percentages in our locally recorded dataset (Fig. 5f)

  Technical Validation, proportion of selective cells, paragraph 3; Figure 5f (the values as written into plotFigure5.ipynb at commit 1d95162)

  Fisher's exact test finds no difference between sites (VS p = 0.360, MS p = 0.215, dual p = 1.000). Under the Methods' tests CS has 17.6%, 5.8% and 0.7% and TWH 22.1%, 9.1% and 0.0%; only CS VS is within tolerance. The code's choices give CS 18.4%, 7.8% and 1.4% (328, 139 and 25 of 1787) and TWH 19.5%, 10.4% and 1.3% (15, 8 and 1 of 77), matching Figure 5f to within 0.1 point, so the partial outcome comes from the Methods disagreeing with the code. With 77 TWH units the 95% intervals of the TWH percentages are 13.4% to 33.0% (VS), 3.7% to 17.8% (MS) and 0.0% to 4.7% (dual), so the Fisher tests have little power.

  How the test differed from the paper’s method

  • The paper names no test for "comparable"; Fisher's exact test was fixed in the criterion.
  • The expected values come from plotFigure5.ipynb, which writes them into the plot as constants.
• ◐Partially reproducedFaraut et al. 2018, Scientific Data 5, 180010 (doi:10.1038/sdata.2018.10), Table 1checks the link to the paper’s data

  The sessions of the earlier release (Faraut et al. 2018) are found in this dandiset under the same subject, session id and task variant.

  64 of the 65 sessions of Table 1 of Faraut et al. 2018 (98.5%) are in the dandiset by session id, all under the same subject and task variant. Session 98 of subject H48 is not; its section in the code's session list is commented out. Age and sex agree for 39 of the 42 matched subjects; C24, H33 and H48 differ, the same three that differ from Online-only Table 1, whose values the 2018 table shares.

  How the test differed from the paper’s method

  • The claim was added after the first verification round, to link the earlier release's session list to the dandiset for previous-release-additions.
• ◐Partially reproducedBackground & Summary, paragraph 3; Technical Validation, proportion of selective cells, paragraph 3uses data from outside the dataset

  Relative to the earlier release of this task's data (Faraut et al. 2018), this release adds 23 sessions, 17 subjects and 288 neurons.

  we added 17 subjects and 288 neurons, including from a new study site, relative to our previously released dataset ... this dataset includes recordings from 23 new sessions

  Background & Summary, paragraph 3; Technical Validation, proportion of selective cells, paragraph 3

  The dandiset has 23 sessions not in the earlier list (14 CS, 9 TWH) from 17 subjects with no earlier session, the paper's numbers. The new sessions hold 325 units, outside the tolerance of 288; the sessions also in the earlier release hold 1539 units, where Faraut et al. 2018 gives 1576 neurons for 65 sessions. 1576 + 288 = 1864, the dandiset's unit count, so the paper's 288 matches a difference of totals and not a count of the new sessions. The differences of the stated totals are 22 sessions, 17 subjects and 287 neurons. The link holds for 64 of 65 earlier sessions; the missing session 98 does not change which sessions or subjects are new, and may account for part of the difference in the earlier sessions' unit count.

  How the test differed from the paper’s method

  • The earlier release's session list is Table 1 of Faraut et al. 2018, not its Dryad archive, which refuses automated download (HTTP 403 on the file stream, and its API asks for a bearer token).
  • In the first version of this attempt the claim was blocked for want of the list; it was tested after the first verification round.
• ✓ReproducedAbstract; Technical Validation, proportion of selective cells, paragraph 3

  The dataset holds 1863 single neurons recorded from the medial temporal lobes of 59 subjects.

  we present a NWB-formatted dataset of 1863 single neurons recorded from the medial temporal lobes of 59 human subjects

  Abstract; Technical Validation, proportion of selective cells, paragraph 3

  1864 unit rows over 87 files (paper 1863, one more) and 59 subjects. With the 55 units of sessions 44 and 55 counted once, as their spike trains repeat those of sessions 43 and 54, 1809. Both readings are within tolerance. CS sessions hold 1787 units and TWH sessions 77. The paper gives no count per session or site, so the extra unit cannot be identified; no unit is empty (the fewest spikes is 22).

  How the test differed from the paper’s method

  • The exact count differs from the paper's by one unit; the tolerance is the protocol's 10%.
• ✓ReproducedMethods, Subjects, paragraph 1; Online-only Table 1

  The patients performed 87 sessions, listed by subject and session id in Online-only Table 1 (the Methods give 89).

  In total, we recorded from patients across 89 sessions (see Online-only Table 1) ... Total Sessions: 87

  Methods, Subjects, paragraph 1; Online-only Table 1

  The 87 (subject, session) pairs of the files equal those of Online-only Table 1, none missing and none extra. The Methods' 89 sessions disagree with the table and the files. Every task variant of the table corresponds to one stimulus set in the files.

• ✓ReproducedAbstract; Figure 5a

  All neurons were recorded from the medial temporal lobe, in the amygdala and the hippocampus.

  1863 single neurons recorded from the medial temporal lobes of 59 human subjects ... Summary of recording locations. ... (Pink: Amygdala, Yellow: Hippocampus).

  Abstract; Figure 5a

  All 1864 units are on electrodes in the right amygdala (594), left amygdala (498), right hippocampus (407) or left hippocampus (365).

• ✓ReproducedTechnical Validation, behaviour, paragraph 1; Figure 5b and 5c

  Subjects distinguished new from old images; the behavioural AUC averaged over all sessions is 0.74 ± 0.10.

  We used the area under the curve (AUC) of the ROC to quantify the ability of subjects to successfully differentiate between new and old stimuli (Fig. 5b). The average AUC across all sessions was 0.74 ± 0.10

  Technical Validation, behaviour, paragraph 1; Figure 5b and 5c

  Mean AUC 0.753, SD 0.094 over 87 sessions (range 0.445 to 0.919; 6 sessions below 0.6, one at or below 0.5). Against chance (0.5), t = 25.1, p = 2.2e-41. With sessions 44 and 55 dropped, 0.754 and 0.095. Recomputed with the label coding the paper states (0 = old), the mean AUC is 0.247 (0.081 to 0.555).

  How the test differed from the paper’s method

  • The tolerance of the mean is the stated spread (0.10), as the protocol prescribes; the spread itself was held to 10%.
• ✓ReproducedTechnical Validation, behaviour, paragraph 1; Figure 5c

  Performance was similar at the two study sites, with an average AUC of 0.74 ± 0.09 for CS sessions and 0.69 ± 0.13 for TWH sessions.

  performance was similar across the two study sites, with an average AUC for CS and TWH sessions of 0.74 ± 0.09 and 0.69 ± 0.13, respectively

  Technical Validation, behaviour, paragraph 1; Figure 5c

  CS 0.760 ± 0.085 over 78 sessions and TWH 0.688 ± 0.138 over 9, all within tolerance. Welch's t-test between sites p = 0.161 (difference 0.072).

• ✓ReproducedTechnical Validation, behaviour, paragraph 1; Figure 5b

  The ROC curve is asymmetric (p < 0.05), as expected for declarative memory.

  Also, the shape of the ROC curve was asymmetric (Fig. 5b, p < 0.05), as expected for declarative memories

  Technical Validation, behaviour, paragraph 1; Figure 5b

  What the paper reports

  • p < 0.05

  Six sessions (59, 60, 93, 99, 119, 2008) have all five interior z(FA) or z(H) points equal, so their fitted slope is not an estimate (negative in four, 0 in one); they are left out. Over the other 81 sessions the z-ROC slope has mean 0.792, median 0.795, SD 0.289, and is below 1 in 64. One-sample t-test against 1, t = -6.49, p = 6.6e-09; Wilcoxon W = 487, p = 3.3e-08. With all 87 sessions, mean 0.726, below 1 in 70, t-test p = 1.631e-09, Wilcoxon p = 1.586e-09. The pooled ROC has slope 0.816. Per-session AUC and slopes are printed.

  How the test differed from the paper’s method

  • The paper names no measure or test for asymmetry; the z-ROC slope of the paper's code and a t-test against 1 were fixed in the criterion.
  • Sessions whose interior z-ROC points do not allow a fit are left out of the mean and the tests, after the second verification round; all 87 sessions are a variant.
• ✓ReproducedTechnical Validation, behaviour, paragraph 1; Figure 5d

  Accuracy is significantly higher in high than in low confidence trials (P < 0.001, paired t test) across all sessions.

  Accuracy was significantly larger in high compared to low confidence trials (P < 0.001) (Fig. 5d) across all sessions.

  Technical Validation, behaviour, paragraph 1; Figure 5d

  What the paper reports

  • paired t test, p < 0.001

  Fixed split, high 80.1% against low 60.9% over 84 sessions, high above low in 73, paired t-test p = 3.2e-16. Dynamic split, 78.3% against 60.4%, paired p = 1.2e-13. Sessions 93, 99 and 2008 have no responses in one group and are left out. The unpaired test of the code gives p = 3.4e-17 and 1.3e-15.

  How the test differed from the paper’s method

  • Three sessions without responses in one confidence group are left out; the paper says all sessions.
• ✓ReproducedTechnical Validation, proportion of selective cells, paragraphs 2 and 3; Figure 5f

  343 of the 1863 cells (18.4%) are visually selective, by a one-way ANOVA (P < 0.05) of the firing rate in recognition trials, in a 1 s window starting 200 ms after stimulus onset, across the 5 visual categories.

  To select for VS cells, we performed a 1 × 5 ANOVA (one way, P < 0.05) ... Out of these 1863 cells (Fig. 5e), 343 cells qualified as VS cells (18.4%)

  Technical Validation, proportion of selective cells, paragraphs 2 and 3; Figure 5f

  Methods' window (200 to 1200 ms), 332 of 1864 units (17.8%), within tolerance. The code's window (200 to 1700 ms) gives 343 of 1864 (18.4%), the paper's count. Counting duplicated spike trains once, 326 of 1809 (18.0%). Shuffled categories give 4.7% (10 shuffles per unit). At p < 0.01, 8.6%; at p < 0.1, 25.6%. Among sessions with at least 20 units, 0.0% to 35.7%.

  How the test differed from the paper’s method

  • The main result uses the Methods' 1 s window; the paper's code uses 1.5 s, shown as a variant.
• ✓ReproducedTechnical Validation, proportion of selective cells, paragraph 3; Online-only Table 1

  The dataset includes 9 sessions from a new institution, Toronto Western Hospital.

  this dataset includes recordings from 23 new sessions, including 9 sessions from a new institution (Toronto Western Hospital, TWH)

  Technical Validation, proportion of selective cells, paragraph 3; Online-only Table 1

  Sessions 2001, 2002, 2003, 2004, 2005, 2007, 2008, 2009 and 2011, from 8 subjects, with 77 units; all name Toronto Western Hospital as institution.

• ✓ReproducedFigure 4

  Neuron 132-177-4-RA (session 132, channel 177, cell 4, right amygdala) is visually selective, responding most to images of animals.

  An example VS neuron selective for the visual category of animals is shown at the bottom (The ID of this neuron 132-177-4-RA corresponds to: session ID – channel number – cell number – brain area).

  Figure 4

  Session 132 has one unit on channel 177, at electrode row 4 in the right amygdala, which matches the label 132-177-4-RA. It is the file's unit at 0-based index 4 of 8, so the label's cell number equals both its electrode row and its 0-based unit index. ANOVA across categories p = 1.2e-22 (Methods' window) and 7.9e-19 (code's window); mean rate 6.75 Hz for animals against 0.25 to 0.85 Hz for the other four categories.