Dandiset 000947

Reproducing Schwab et al. (2020) from dandiset 000947 18

Notebook reproduce_schwab_2020.ipynb

• ✕Not reproducedResults, first section; Table 1

  209 single units met the criteria to be included as GPi neurons (105 in NHP G, 104 in NHP I) and 185 as VLa neurons (66 and 119).

  The dandiset rule gives 268 GPi units (Gaia 137, Isis 131) and 279 VLa units (78 and 201), against 209 and 185 (218 in the text). Counting the same channel and sort letter on one day once gives 132 and 186; the paper's records found in the dandiset are 199 and 165. That merge joins some units the paper kept apart (47 of the 117 merged units spanning several files hold more than one of the paper's records). 518 of the 547 units are in files the paper used. Taking the rate over the length of the raw recording instead of the span of the spike train would leave out 2 more GPi units. The dandiset holds more units than the paper analysed and does not say which were kept or which units in consecutive files are one neuron.

  How the test differed from the paper’s method

  • Units are selected by the dandiset's location labels and the Methods' rate and trial criteria; the paper's anatomical criteria (within 0.5 mm of an M1-responsive site, the VLa border) cannot be applied from the files.
  • Each file's unit is counted once; the paper pools a neuron recorded over consecutive files.
• ✕Not reproducedResults, second section; Table 1

  Monophasic changes are more common in VLa than in GPi (88.9% against 66.7% of responses; chi-squared p = 2.8e-12), and polyphasic responses more common in GPi.

  Monophasic responses are 71.0% in GPi and 76.7% in VLa, chi-squared p = 0.051, and VLa is outside tolerance. With the paper's records 69.0% and 81.2% (p = 0.00036), both within tolerance, so the outcome depends on the selection of units. With one record per presumed neuron 62.6% and 74.5% (p = 0.0023), VLa outside tolerance.

  How the test differed from the paper’s method

  • Unit selection by the dandiset rule (see table1-unit-counts).
• ✕Not reproducedResults, third section; Fig 3A

  Peri-movement changes begin later in GPi than in VLa (median onsets -47 and -90 ms; rank sum p = 2.4e-5).

  Median onsets of all changes are -48 ms in GPi (641) and -67 ms in VLa (613); GPi is later by 19 ms without significance (rank sum p = 0.079, one-sided p = 0.04), and the VLa median is outside tolerance. In Gaia the medians are equal (-48.5 and -48 ms). With one record per presumed neuron -73 and -72 ms (p = 0.95), with one value per presumed neuron -47 and -53 ms (p = 0.44). With the paper's records -49 and -85 ms (p = 0.004), which meets the criterion, so the outcome depends on the selection of units. At alpha 0.0001 -42.5 and -63 ms (p = 0.052).

  How the test differed from the paper’s method

  • Unit selection by the dandiset rule (see table1-unit-counts).
• ✕Not reproducedResults, third section; Fig 3C

  VLa decreases precede GPi increases (median onsets -108 and -62 ms; rank sum p = 0.01).

  VLa decreases begin at a median of -19 ms (244) and GPi increases at -69 ms (377); VLa does not lead (rank sum p = 0.17). In Gaia VLa decreases come later (12.5 against -75 ms, p = 0.0028). With the paper's records VLa leads by 13 ms without significance (-86 against -73 ms, p = 0.2). The paper itself gives -62 ms for GPi increases here and -76 ms in Table 1. With one record per presumed neuron VLa decreases come later (-8 against -92 ms, p = 0.0082), and with one value per presumed neuron too (20.5 against -45 ms, p = 0.021). With random alignment the detector finds a decrease in 33.7% of VLa unit-directions (44.2% in the data), with a median onset of -8 ms, so many VLa decreases here may be chance detections, which would pull their median late. At alpha 0.0001 the medians are -21.5 (VLa) and -57.5 ms (GPi), p = 0.3, and chance still gives a decrease in 30.8% of VLa unit-directions (42.0% in the data), so the stricter threshold does not remove the chance decreases.

  How the test differed from the paper’s method

  • The ISI-based detection finds more decreases than the paper's on the same units (330 against 286 on the linked units), which weighs on this comparison.
• ✕Not reproducedResults, fourth section; Fig 4A-F; Table 2

  Short-time-scale spike-to-spike correlations between simultaneously recorded GPi and VLa units are rare (3.5% of 427 pairs over whole recordings, 2.2% of 183 at rest, 5.2% of 172 during movement) and their peaks are not larger than those of jittered controls (p = 0.975, 0.675 and 0.998).

  By the KS permutation test the peak CCFs exceed those of jittered controls over whole recordings (338 pairs, p = 0.003) and during movement (306 pairs, p = 0.045), not at rest (298 pairs, p = 0.705). Significant pairs are 5.6%, 3.4% and 6.5%, none above 5% by a binomial test, and mostly positive (15 of 19 whole, 15 of 20 movement). The shift is small (median peak 0.042 against a control median of 0.038). The part of the statement that correlations are rare holds; the part that peaks are not larger than controls does not. Keeping only pairs with an approximate c0 of at most 0.3 leaves 332, 121 and 172 pairs (paper 427, 183 and 172) with KS p = 0.003, 0.333 and 0.025; one file per pair of presumed neurons gives 0.003, 0.627 and 0.037. The significant peaks fall at lags spread over 0 to 10 ms, and GPi and VLa spikes share a sample as often as independent trains would (11,376 against 11,435 expected), so shared artefacts do not explain the excess. Spike times are at the sample resolution (24414.0625 Hz) in every file. With the paper's records, paired when a GPi and a VLa record share a file (329, 298 and 301 pairs), the peaks exceed the controls over whole recordings (p = 0.005) and at rest (p = 0.033), not during movement (p = 0.203), so the outcome would be the same. Neither pair set matches the paper's 427 whole-recording pairs (338 on the dandiset rule, 329 on the paper's records); this attempt did not find out how the paper counted its pairs.

  How the test differed from the paper’s method

  • The paper's exclusion of pairs with a minimum detectable correlation c0 above 0.3 is not applied in the main test, which the criterion fixed before the analysis; the paper's simulation is not in its deposit. It is shown as a variant, with c0 approximated by a simulation on the CCF counts (independent Poisson counts at the pair's expected count, plus an injected peak).
  • Only VLa spikes are jittered; the Methods do not say which train.
• ✕Not reproducedResults, fourth section; Fig 4G-H; Table 2

  Trial-to-trial noise correlations between GPi and VLa are rare (5.4% of 332 pairs at rest, 7.2% of 372 during movement), not different from shuffled controls (p = 0.07 and 0.388), and equally often positive and negative.

  Absolute noise correlations exceed trial-shuffled controls at rest and during movement (KS permutation p < 0.001 in both). 10.1% and 10.8% of pair-targets (17.5% and 18.6% of pairs) are significant against 5.4% and 7.2% in the paper. Signs are balanced (38 and 29 at rest, p = 0.33; 39 and 33 during movement, p = 0.56), as the paper reports. Spike counts drift with trial time in many units (VLa 36.1% and 38.3% of pair-targets, Spearman p < 0.05), but with counts linearly detrended the excess remains (9.3% and 9.7% significant, KS p < 0.001), and so it does against each pair-target's own shuffles (10.8% and 14.2%), with one file per pair of presumed neurons (KS p < 0.001) and with the paper's records (9.1% and 7.4% significant, KS p < 0.001 in both).

  How the test differed from the paper’s method

  • The paper's test statistic for noise correlations is not printed in its code; the KS permutation test on absolute correlations was fixed in the criterion before the analysis.
  • Significance uses the 2.5 to 97.5 percentiles of the shuffles pooled over pairs, as the paper's code does per time point.
• ✕Not reproducedResults, fifth section; Fig 5A-C; Table 2

  At rest, GPi burst onsets, burst offsets and pause onsets are each followed by a small, slow decrease of the population VLa firing rate.

  The population VLa rate does not fall below the control band within 300 ms of GPi burst onsets (150 pairs, lowest point 2.5% below baseline), burst offsets (150, 1.9%) or pause onsets (110, 1.9%). Bursts here last 67.1 ms on average and pauses 145.3 ms, against 24 and 195 ms in the paper, and 50.7% of pairs show a change after burst onsets against 11.6% in the paper. The same per-pair test on the shifted controls gives 47.5%, 47.8% and 47.0%, so the test as the Methods describe it is at chance here, and the paper's per-pair test must differ. Computed as Fig5_S14.m does (each pair in percent of its baseline, band shifted over 0 to 800 ms), the population rate does not fall below the band either (lowest 98.1%, 98.4% and 96.9%). The Methods' rule on noisy baselines leaves out more than half of the pairs with at least 5 events (157 of 307 for burst onsets and offsets, 143 of 253 for pause onsets). With the paper's records (131, 132 and 86 pairs kept) the rate does not fall below the band either. Because the burst detector and the per-pair test differ from the paper's, this outcome says little about the paper's finding.

  How the test differed from the paper’s method

  • Burst and pause detection is a reimplementation of the Legendy and Salcman surprise method; the paper's implementation is not in its deposit (Fig5_S14.m plots stored averages and builds the control band; it holds no burst detector and no per-pair test).
  • A first run used log10 in the surprise and gave bursts far longer than the paper's; the natural log, as in the Methods' formula in the paper's MATLAB, was then used. The notebook shows the durations with log10 (bursts 254.0 ms, pauses 237.0 ms) as a variant.
  • The control shifts all events of a pair by one random offset of 2 to 20 s and uses VLa rate in rest samples only; the paper does not describe its control for Fig 5.
• ◐Partially reproducedResults, first section; Fig 2A

  The firing rate of most units ramps during the start-position hold period (97% of GPi and 98% of VLa units with a significant linear trend); slopes are distributed symmetrically around zero, without a difference between GPi and VLa (rank sum p = 0.29).

  98.5% of GPi and 98.6% of VLa units have a significant control-period slope (97.0% and 97.5% at alpha 0.01), and 49.9% of significant slopes are positive. GPi and VLa slopes differ (mean -0.598 against 0.233 spikes/s2, rank sum p = 0.042), where the paper finds no difference (p = 0.29); they also differ with one record per presumed neuron (p = 0.013) and with one mean slope per presumed neuron (p = 0.033). With the paper's records the fractions are 99.5% and 98.8% and the slopes do not differ (p = 0.187), which would meet the criterion, so the outcome depends on the selection of units. With every trial aligned to a random time, the same slope test finds a significant slope in 98.4% of GPi and 99.0% of VLa units, so the fractions do not show ramps beyond what noise in a smoothed function gives; the paper's code uses the same test.

  How the test differed from the paper’s method

  • The criterion did not define a partial outcome; the outcome is partial because the first part (ramps are common) holds and the second (no difference between areas) does not.
• ◐Partially reproducedResults, second section; Table 1; Fig 2B

  Monophasic increases are the most common form of response in both GPi and VLa (48.7% and 65.3% of responses), and increases make up most detected changes (61.5% and 68.8%, biphasic responses counted twice), more than decreases (chi-squared p = 1.2e-17).

  Monophasic increase is the most common form in GPi (46.9%) and VLa (50.9%), and increases are 58.8% and 60.2% of changes (746 of 1254 overall, binomial p = 1.9e-11). The GPi values are within tolerance; the VLa values are below 65.3% and 68.8% by more than 10%, also with the paper's records (54.1% and 61.4%), with one record per presumed neuron (50.1% and 60.3%) and at alpha 0.0001 (51.9% and 60.1%). With random alignment the detector finds a decrease in 25.2% (GPi) and 33.7% (VLa) of unit-directions and an increase in 14.8% and 16.2%, so chance detections add mostly decreases.

  How the test differed from the paper’s method

  • Detrending of the control period and the ISI function are implemented here from the Methods; the paper's own implementation of these steps is not in its deposit.
• ◐Partially reproducedResults, second section; Table 1

  The mean integrated change in firing over the peri-movement epoch is positive in both GPi and VLa (3.89 and 1.75 spikes), significantly different from zero.

  The mean integrated change is positive in GPi (3.07 spikes, signed rank p = 1.1e-10) and VLa (1.50 spikes, p = 3.7e-18), both more than 10% below 3.89 and 1.75. With the paper's records 3.42 and 1.97, with one record per presumed neuron 2.13 and 1.34.

  How the test differed from the paper’s method

  • Table 1 gives the unit as spikes x 10^3; the code's computation gives spikes, the reading used.
• ◐Partially reproducedResults, second section; Methods

  A majority of GPi neurons (109 of 206) and some VLa neurons (56 of 179) respond significantly more strongly to one movement direction.

  122 of 264 responsive GPi units (46.2%) and 66 of 272 VLa units (24.3%) are directional, chi-squared p = 1.7e-07; both fractions are more than 10% below 52.9% and 31.3%. With the paper's records 48.2% and 26.8%, with one record per presumed neuron 53.8% and 21.9% (p = 9.8e-09).

• ✓ReproducedData Availability; Dryad doi:10.5061/dryad.0cfxpnvxm, Gaia_Analy.mat and Isis_Analy.matuses data from outside the dandiset

  The units the paper analysed can be found in the dandiset, and the dandiset's spike times and trials reproduce the per-unit peri-movement activity the paper's code stored.

  Records are matched by file, channel id and sort letter. 364 of the 394 records the paper's code keeps (92.4%) are found in every file they span (Isis GPi 104 of 104, Gaia GPi 95 of 105, Gaia VLa 60 of 66, Isis VLa 105 of 119). For 724 matched unit-directions the SDF from the dandiset correlates with the stored SDF at a median r of 0.9992, above 0.9 for 94.6%. Valid trial counts equal the paper's for 62.6%. Detection agrees on the presence of an increase for 90.9% and of a decrease for 80.7% of unit-directions. 25 matched units the paper calls VLa are labelled "Thalamus" in the dandiset, and 97 records span 2 to 5 files. On the side of the outside data, the paper's list gives 209 GPi and 185 VLa units, as Table 1 does.

  How the test differed from the paper’s method

  • Trial exclusion follows the Methods (6 median absolute differences from the mean); the paper's stored trial counts differ for 37% of unit-directions, by a median of 0.
• ✓ReproducedResults, first section; Table 1

  The resting firing rate of GPi neurons (mean 70.1 spikes/s) is significantly higher than that of VLa neurons (13.5 spikes/s), and the rates are consistent between the two animals.

  Mean rest rate over unit-directions 75.65 spikes/s in GPi (531) and 12.78 in VLa (552), GPi higher with rank sum p = 1.2e-171. The animals differ slightly in GPi (73.17 and 78.25, p = 0.041), where the paper reports no difference; not in VLa (p = 0.53). With one record per presumed neuron the animals do not differ in GPi (74.23 and 74.21, p = 0.69). With the paper's records 64.26 and 13.53, with one record per presumed neuron 74.22 and 12.77.

  How the test differed from the paper’s method

  • Unit selection by the dandiset rule (see table1-unit-counts).
• ✓ReproducedResults, first section; Table 1

  Action potentials of GPi neurons are shorter than those of VLa neurons (trough to peak 0.24 against 0.50 ms; rank sum p = 1e-33).

  Mean trough-to-peak width 0.247 ms in GPi and 0.509 ms in VLa, 532 of 547 units measured, rank sum p = 2.9e-37. One width is 0 and 9 reach the end of the 1.2 ms window; without them the means are 0.249 and 0.499 ms (p = 1.9e-35). No file gives one channel id to two electrodes, so each unit's channel names one electrode.

  How the test differed from the paper’s method

  • Widths come from the dandiset's 200 Hz high-pass filtered traces (the paper filtered at 300 Hz) and from the spikes in one stored chunk (about 3.2 s, median 273 spikes) per file and series.
• ✓ReproducedResults, second section; Table 1

  Nearly all units change their discharge around movement for at least one direction (GPi 99%, 206 of 209; VLa 88.1%, 163 of 185 in Table 1, 94% in the text).

  264 of 268 GPi (98.5%) and 272 of 279 VLa units (97.5%) respond in at least one direction. VLa matches the text's 94% but not Table 1's 88.1%; the criterion accepts either reading. With alpha 0.0001 the fractions with an increase alone are 81.0% and 77.4%. With every trial aligned to a random time 62.3% of GPi and 70.8% of VLa units are responsive, so the data are well above chance; with one record per presumed neuron the fractions are 100.0% and 98.4%.

  How the test differed from the paper’s method

  • The paper gives 88.1% (Table 1) and 94% (text) for VLa; both readings were listed and only the text's matches.
• ✓ReproducedResults, third section; Fig 3B

  GPi decreases lag behind VLa increases (median onsets -27 and -88 ms; rank sum p = 4.9e-4), against the gating prediction.

  GPi decreases begin at a median of -29.5 ms (264) and VLa increases at -84 ms (369), rank sum p = 0.00021. The GPi median is 0.2 ms inside its tolerance (-29.7 to -24.3 ms). Gaia 0 against -95 ms (p = 0.0014), Isis -44 against -81 ms (p = 0.055). The lag also holds with the paper's records (-36 against -85 ms, p = 0.00069), with one record per presumed neuron (-46.5 against -92 ms, p = 0.011), with one value per presumed neuron (-1.5 against -71.2 ms, p = 0.00074) and at alpha 0.0001 (-30 against -81.5 ms, p = 0.00054), but in all four the GPi median is outside its tolerance. With the paper's records the outcome would be partially reproduced, so it depends on the selection of units, and the match of the medians is fragile.

• ✓ReproducedResults, third section

  Comparing every GPi response with every VLa response, 96% of comparisons conflict with gating (178,846 of 187,154); only 4% have a GPi onset 0 to 50 ms before the VLa onset with the opposite sign.

  95.4% of 392,933 comparisons conflict with gating (95.6% with the paper's records, 95.1% with one record per presumed neuron). With onsets shuffled across unit-directions the fraction is 95.1% (95.0 to 95.2), so the number does not distinguish the data from random onsets; it follows the gating window (98.0% for 0 to 20 ms, 86.6% for 0 to 200 ms).

• ✓ReproducedResults, Task performance; S2 Fig

  Reaction times do not differ between the animals or reach directions; movement durations are longer for reaches to the right target, and NHP G moves more slowly than NHP I.

  In the 165 files that hold the units analysed, reaction times differ neither between the animals (F = 0.02, p = 0.89) nor between directions (F = 0.69, p = 0.41), and movement durations are longer to the right target (Gaia 0.273 against 0.216 s, Isis 0.195 against 0.174 s; p = 5.8e-38) and longer in Gaia (p = 7.3e-56). Over all 250 pre-MPTP files with trials reaction times differ between the animals (F = 15.19, p = 0.00011); over the files of the paper's records they do not (F = 3.79, p = 0.053). The paper also reports a significant animal by direction interaction for reaction times (p = 0.002), which is not part of the claim; here p = 0.12.

  How the test differed from the paper’s method

  • The first version used all 250 pre-MPTP files with trials, including files with motor cortex, SMA or VLp units only, and gave a reaction-time difference between the animals. After round 1 the test was restricted to the 165 files with units of the dandiset rule, the files of the paper's data as the dandiset identifies them; the outcome changed from partially reproduced to reproduced. All files and the files of the paper's records are shown as variants.
  • Movement durations exist for 20 of the 91 Gaia files used.
  • The observations are file and direction medians (330 for RT); the paper's ANOVA has 452 error degrees of freedom and does not say what its observations are. No file set here gives 452 (326 on the files with units, 496 on all files, 320 on the files of the paper's records), and the reaction-time animal effect depends on the files used.