DANDI:000147

Reproducing Guan, Aflalo et al. (2022) from dandiset 000147 14

Notebook reproduce_guan_2022.ipynb

• ✕Not reproducedFigure 1-figure supplement 6b

  A diagonal LDA classifier could not classify the cue location from neural activity in the attempted-movement window.

  A linear classifier (diagonal linear discriminant analysis [LDA]) could not classify the gaze location from neural activity during the attempted-movement period.

  Figure 1-figure supplement 6b

  Diagonal LDA classifies the 12 cue locations at 11.5% against a chance level of 8.3% (binomial p = 1e-12), above all 500 label shuffles across trials (highest 10.0%) and all 500 shuffles within finger (highest 10.2%). The authors' classifier gives 11.2%, and removing each finger's mean activity first gives 11.7%. Five of the ten sessions are above chance at p < 0.05. With folds that never split a screen the accuracy is 11.1%, above all 200 shuffles of location within screens (highest 9.8%), and with folds that never split a run it is 11.3%. Every screen cues each location once; location and position within the screen are weakly associated over all trials (chi-square p = 0.037). The cued finger is classified at 93.4% from the same features. With folds that never split a run, 11.3% is above all 200 shuffles of location within runs (highest 9.8%). Position within the screen is itself decoded at 10.5%, above all 200 shuffles within screens (highest 9.5%), but after removing each position's mean activity location is still classified at 11.1%.

  How the test differed from the paper’s method

  • Eight stratified folds within each session; the paper does not say how this classifier was cross-validated, and the authors' code uses eight folds for its classifiers.
  • The paper shows a confusion matrix without an accuracy or a test, so the comparison is with its statement that the location could not be classified.
• ✓ReproducedResults, "Intracortical recordings during finger flexion", paragraphs 1 and 3

  Over 10 sessions the recordings yielded 95.8 (SD 6.7) sorted neurons and 408 (SD 40.8) trials of the main finger task per session.

  We recorded single and multineuron activity (95.8 ± standard deviation [SD] 6.7 neurons per session over 10 sessions) ... We repeated this online-control finger flexion task over multiple sessions (408 ± SD 40.8 trials/session over 10 sessions)

  Results, "Intracortical recordings during finger flexion", paragraphs 1 and 3

  The ten files hold 95.8 units per session (SD 6.7) and 408.0 trials per session (SD 40.8), 4080 trials in all, as reported.

• ✓ReproducedFigure 1b, Figure 1-figure supplement 3, Results, "Accurately decoding fingers from PPC single-neuron activity", paragraph 1

  Online BCI classification of the cued finger was 86% accurate (SD 4% across sessions, weighted by trials) over 4016 trials in 10 sessions, against a chance level of 17%. No-Go trials of the first session are excluded because the classifier could not output No-Go that day.

  High classification accuracy during online control (86% ± SD 4% over 10 sessions; chance = 17%) (Figure 1b and Figure 1—figure supplement 3)

  Figure 1b, Figure 1-figure supplement 3, Results, "Accurately decoding fingers from PPC single-neuron activity", paragraph 1

  The recorded decoder outputs match the cued finger on 85.96% of the 4016 scored trials, with a trial-weighted SD across sessions of 3.6%. Shuffling the outputs gives 16.9%, the paper's chance level of 17%.

• ✓ReproducedResults, "Representational structure did not trend toward task optimum", paragraph 2; Figure 1b

  Most (70%) of the online classification errors were middle-ring or ring-pinky confusions.

  The majority (70%) of the online classification errors were middle-ring or ring-pinky confusions (Figure 1b).

  Results, "Representational structure did not trend toward task optimum", paragraph 2; Figure 1b

  Of the 379 online errors between two fingers, 69.7% confuse middle with ring or ring with pinky. Counting errors that involve No-Go, the fraction is 46.8% of 564 errors, or 59.2% of the 446 errors on finger trials.

  How the test differed from the paper’s method

  • The paper does not say which errors form the denominator. Only errors between two fingers, leaving out No-Go, give the paper's 70%.
• ✓ReproducedFigure 1-figure supplement 4a, Results, "Accurately decoding fingers from PPC single-neuron activity", paragraph 1, Methods "Finger classification"

  Offline classification of the cued finger from sorted firing rates in the window 0.5 to 1.5 s after the cue, with diagonal LDA and leave-one-out cross-validation within each session, was 92% accurate (SD 2% across sessions) over 4080 trials.

  offline cross-validated classification (92% ± SD 2%; Figure 1—figure supplement 4a) demonstrated that the finger representations were reliable and linearly separable.

  Figure 1-figure supplement 4a, Results, "Accurately decoding fingers from PPC single-neuron activity", paragraph 1, Methods "Finger classification"

  Diagonal LDA with leave-one-out cross-validation within sessions gives 93.4% (weighted SD 1.6%) over 4080 trials. The classifier and eight stratified folds of the authors' code give 92.4% (SD 1.7%).

  How the test differed from the paper’s method

  • The paper's methods say leave-one-out; the authors' code uses eight folds, a mean-count cutoff of 0.5, z-scoring and LDA with identity covariance. Both are reported, and the measured value is the leave-one-out one.
• ✓ReproducedResults, "Accurately decoding fingers from PPC single-neuron activity", paragraph 2, Figure 1-figure supplement 5, Methods "Single-unit tuning to finger flexion"

  Most (89%) neurons were significantly tuned to at least one finger movement, by a two-tailed t-test of each finger's firing rate in the 0.5 to 1.5 s window against No-Go trials, p < 0.05 after FDR correction, with neurons below 0.1 Hz excluded and neurons pooled across sessions.

  At the single-neuron level, most (89%) neurons were significantly tuned to individual finger-press movements (significance threshold: p < 0.05, FDR-corrected) (Figure 1—figure supplement 5).

  Results, "Accurately decoding fingers from PPC single-neuron activity", paragraph 2, Figure 1-figure supplement 5, Methods "Single-unit tuning to finger flexion"

  With FDR correction over all units and fingers pooled, 87.9% of 958 units are tuned to at least one finger. FDR within each session gives 87.5%, within each unit 83.9%, and no correction 91.7%. No unit fell below the 0.1 Hz cutoff, whether the rate is taken in the window over all trials (lowest 0.113 Hz) or over the whole recording (lowest 0.147 Hz).

  How the test differed from the paper’s method

  • The FDR family is not stated in the paper; the measured value pools all units and fingers.
  • Student's t-test with equal variances; the paper does not say which form of the t-test.
• ✓ReproducedFigure 2d and 2e, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 3

  In the session-averaged representational dissimilarity matrix (cross-validated Mahalanobis distance, window 0.5 to 1.5 s, No-Go excluded), the thumb is distinct, while the middle, ring and pinky fingers are close.

  For visual intuition, we also projected the representational structure to two dimensions in Figure 2e, which shows that the thumb is distinct, while the middle, ring, and pinky fingers are close in neural space.

  Figure 2d and 2e, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 3

  In the session-averaged RDM, max-scaled, the distances among middle, ring and pinky are 0.21 to 0.48 and every distance involving the thumb is 0.64 or more. This holds in 9 of the 10 sessions on their own.

  How the test differed from the paper’s method

  • The noise covariance uses OAS shrinkage, as in the authors' code; the paper names Ledoit-Wolf.
• ✓ReproducedFigure 2f, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 3uses data from outside the dataset

  The measured RDMs match the able-bodied PC-IP fMRI model better than the unstructured (pairwise-equidistant) model, by whitened unbiased cosine similarity (WUC), p = 5.7e-5.

  The measured representational structure matched the able-bodied representational structure significantly over the unstructured model (p = 5.7 × 10–5, two-tailed t-test) (Figure 2f)

  Figure 2f, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 3

  What the paper reports

  • two-tailed t-test, p = 5.7 × 10⁻⁵ (n = 1000 bootstrap samples over 10 sessions)

  Mean WUC is 0.881 with the able-bodied PC-IP model and 0.781 with the unstructured model. The PC-IP model is higher in all 10 sessions; the bootstrap 95% interval of the difference is 0.074 to 0.127, and the bootstrap t-test gives p = 4.4e-5 (paper 5.7e-5).

  How the test differed from the paper’s method

  • The model RDMs are read from the authors' code repository at the commit archived with the paper, not from the dandiset.
  • The p-value comes from 1000 resamples of the sessions (mean difference over its bootstrap SD, nine degrees of freedom); the paper used rsatoolbox's bootstrap t-test.
• ✓ReproducedFigure 2-figure supplement 5a, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 5uses data from outside the dataset

  The measured RDMs match the able-bodied motor cortex (MC) fMRI model better than the able-bodied PC-IP fMRI model (p = 1.9e-6), and the WUC with the MC model is close to the noise ceiling.

  PC-IP BCI finger representations matched the able-bodied fMRI finger representational structure in the MC ... even better than that of able-bodied PC-IP (Figure 2—figure supplement 5). The WUC similarity with the MC RDM was close to the noise ceiling

  Figure 2-figure supplement 5a, Results, "Finger representational structure matches the structure of able-bodied individuals", paragraph 5

  What the paper reports

  • two-tailed t-test, p = 1.9 × 10⁻⁶ (n = 1000 bootstrap samples over 10 sessions)

  Mean WUC with the MC model is 0.969, higher than with the PC-IP model in all 10 sessions (difference 0.088, bootstrap 95% interval 0.060 to 0.112, p = 7.9e-5; paper 1.9e-6). It is 0.015 below the lower noise ceiling of 0.984, and at or above its own session's ceiling in 1 of 10 sessions.

  How the test differed from the paper’s method

  • The model RDMs are read from the authors' code repository, as for fig2f-able-bodied-over-unstructured.
  • The criterion's margin of 0.05 below the noise ceiling is a choice; the paper says only "close to".
• ✓ReproducedResults, "Representational structure did not trend toward task optimum", paragraph 1

  The finger RDMs were consistent across sessions, with an average pairwise correlation between session RDMs of r = 0.90 (SD 0.04, min 0.83, max 0.99).

  Indeed, participant NS's finger RDMs were largely consistent across different recording sessions (average pairwise correlation, excluding the diagonal: r = 0.90 ± SD 0.04, min 0.83, max 0.99).

  Results, "Representational structure did not trend toward task optimum", paragraph 1

  The 45 pairs of session RDMs correlate at a mean Pearson r of 0.928 (SD 0.041, min 0.817, max 0.990), against 0.90 (SD 0.04, min 0.83, max 0.99). The mean is 0.028 above the paper's, inside the tolerance of 0.03.

• ✓ReproducedFigure 3a, Results, "Representational structure did not trend toward task optimum", paragraph 1uses data from outside the dataset

  Over sessions the model fit did not trend from the able-bodied model toward the unstructured model (session by model interaction t(6) = 0.50, one-tailed p = 0.32).

  However, we did not find conclusive evidence for a trend from the able-bodied model toward the unstructured model (linear-model session × model interaction: t(6) = 0.50, one-tailed t-test p = 0.32 ... (Figure 3a).

  Figure 3a, Results, "Representational structure did not trend toward task optimum", paragraph 1

  What the paper reports

  • one-tailed t-test of the linear-model session × model interaction, t(6) = 0.50, BF = 0.66, p = 0.32

  The session by model interaction is t(16) = 1.47, one-tailed p = 0.080 toward the unstructured model (paper t(6) = 0.50, p = 0.32). The WUC with the PC-IP model alone declines across sessions (slope -0.009 per session, p = 0.038); the slopes for the unstructured and MC models are not significant (p = 0.20 and 0.23). In other specifications the trend is also not significant. The same t read with 6 degrees of freedom gives p = 0.096; the difference between the unstructured and PC-IP fits regressed on session gives t(8) = 1.30, p = 0.116; with the MC model in place of PC-IP the slope is negative (p = 0.84).

  How the test differed from the paper’s method

  • Ordinary least squares on the twenty session-by-model WUC values, with sixteen residual degrees of freedom. The paper's t(6) implies a different model or degrees of freedom, which it does not describe, and its code for this test is not in the repository.
  • Three specifications are shown, since the paper's t(6) is not explained.
  • No Bayes factor is computed.
• ✓ReproducedFigure 3b, Results, "Representational structure did not trend toward task optimum", paragraph 2, Methods "Measuring changes in the representational structure"

  The distances between the most confused finger pairs (middle-ring and ring-pinky) did not increase across sessions (one-tailed p > 0.99) or across runs within sessions (one-tailed p = 0.67).

  the middle-ring and ring-pinky distances did not increase over the course of the experiment (across sessions: t(8) = −4.5, one-tailed t-test p > 0.99 ... across runs within sessions: t(82) = −0.45, one-tailed t-test p = 0.67

  Figure 3b, Results, "Representational structure did not trend toward task optimum", paragraph 2, Methods "Measuring changes in the representational structure"

  What the paper reports

  • across sessions: one-tailed t-test, t(8) = -4.5, BF = 0.03, p > 0.99
  • across runs within sessions: one-tailed t-test, t(82) = -0.45, BF = 0.12, p = 0.67

  For the middle-ring and ring-pinky distances of 85 per-run RDMs, the session term is t(8) = -3.56 (paper -4.5) and the run term t(83) = -1.05 (paper -0.45). In all six estimates the coefficients are negative and the one-tailed p for an increase is 0.73 or more.

  How the test differed from the paper’s method

  • The paper's code for this regression is not in its repository. The regression follows the methods' formula on per-run RDMs, with degrees of freedom of sessions minus two for the session term and runs minus two for the run term, as the paper's t(8) and t(82) suggest.
  • The dandiset has 85 runs; the paper's t(82) implies 84.
  • Per-run RDMs use five folds of eight trials each, so a fold's noise covariance rests on one or two trials per finger and inflates the distances (mean 5.06 against 0.68 for session RDMs). A variant with the session's noise covariance (mean 0.76) is reported.
  • No Bayes factors are computed.
• ✓ReproducedFigure 3-figure supplement 1, Results, "Representational structure did not trend toward task optimum", paragraph 2

  Across all finger pairs the inter-finger distances did not increase; they decreased slightly across sessions (t(8) = -4.0, two-tailed p = 0.004) and across runs within sessions (t(82) = -2.4, p = 0.019), with very small effect sizes (Cohen's f2 of 0.008 and 0.005).

  Inter-finger distances instead decreased slightly (across sessions: t(8) = −4.0, two tailed t-test p = 0.004; across runs within sessions: t(82) = −2.4, two-tailed t-test p = 0.019), although the effect size was very small (across sessions: Cohen's f2 = 0.008 ...

  Figure 3-figure supplement 1, Results, "Representational structure did not trend toward task optimum", paragraph 2

  What the paper reports

  • across sessions: two-tailed t-test, t(8) = -4.0, Cohen's f2 = 0.008, p = 0.004
  • across runs within sessions: two-tailed t-test, t(82) = -2.4, Cohen's f2 = 0.005, p = 0.019

  Over all ten pairs the session and run coefficients are negative in all six estimates, and Cohen's f2 is 0.006 for session and 0.012 for run in the main regression, and 0.017 and 0.013 with the session's noise covariance (paper 0.008 and 0.005). In the main regression the session decrease is t(8) = -2.28, p = 0.052, and the run decrease t(83) = -3.12, p = 0.0025. With the session's noise covariance they are t(8) = -3.76, p = 0.0055, and t(83) = -3.29, p = 0.0015, against the paper's t(8) = -4.0, p = 0.004, and t(82) = -2.4, p = 0.019.

  How the test differed from the paper’s method

  • The same regression and choices as fig3b-confused-pairs-no-increase.
  • The session decrease is significant at 0.05 only in the variant with the session's noise covariance; the pre-registered criterion asks for negative coefficients and small effect sizes, not significance. The per-run fold covariance inflates the distances of the main regression, so the variant with the session's covariance is the better estimate of the session term.
• ⊘BlockedFigure 1-figure supplement 6a· Metadata insufficient

  Linear regression could not decode the cue's screen coordinates from neural activity in the attempted-movement window; cross-validated r2 was close to 0 in every session.

  Linear regression could not decode target location [x, y] coordinates from the neural activity during the attempted-movement period. Violin plot shows that cross-validated regression r2 values are close to 0 across sessions

  Figure 1-figure supplement 6a

  The trials table gives the cue location as an index from 1 to 12 on the grid of three rows and four columns, described only as a screen location index, with no map to the grid or to screen coordinates. As a check, the coordinates were regressed on spike counts under the two plain orders of the index. Ordinary least squares gives a cross-validated r2 below 0 for both coordinates in all ten sessions under both orders. Ridge regression gives r2 up to 0.097 for x and 0.104 for y, with both coordinates at or below 0.05 in 5 (row order) or 6 (column order) of the ten sessions. The result depends on the regression as well as on the unknown layout.

  How the test differed from the paper’s method

  • The check assumes the index runs along the rows or down the columns; neither is documented.
  • The paper does not say whether its regression was regularised, so both ordinary least squares and ridge regression are shown, each with eight unshuffled folds per session.