Neural Prosthetics IV: Pushing preferred directions around Beata Jarosiewicz Laboratory of Andrew Schwartz

The explanatory gap Absolute gap: Why is neural activity accompanied by any conscious experience at all? Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? ‘Red’ 670 nm wavelength light Neural activity resulting from 670 nm light

The explanatory gap Absolute gap: Why is neural activity accompanied by any conscious experience at all? Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? ‘Red’ 670 nm wavelength light Neural activity resulting from 670 nm light absolute gap

The explanatory gap Absolute gap: Why is neural activity accompanied by any conscious experience at all? Comparative gap: Why does certain neural activity give rise to, say, visual rather than auditory experience? Or to the experience of red instead of green? ‘Red’ ‘Green’ 670 nm wavelength light Neural activity resulting from 670 nm light 530 nm wavelength light Neural activity resulting from 530 nm light comparative gap absolute gap

Cortical “deference” vs. “dominance” from “Neural Plasticity and Consciousness” (Hurley & Noë, Biology & Philosophy, 2003)

Cross-modal remapping: “vOICe” Left and Right Image is scanned left to right, one snapshot per second. So left and right get encoded by time (and stereo). Dark and Light Brightness is encoded by loudness: the brighter the visual pattern, the louder the sound. Up and Down Elevation is encoded by pitch: the higher the position of the visual pattern, the higher the pitch. Movie by Alex Storer, BU

Cross-modal remapping: “vOICe” 2000: “Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.” 2002: “Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.” - Pat Fletcher, long-term vOICe user

Cross-modal remapping: “vOICe” 2000: “Well the other day I was again washing dishes. I had let the water out of the sink and turn to get a towel to dry my hands. Then when I turn back to rinse the sink I was stunned to see the sink in a “depth” like image. I stepped away from the sink and walked slowly up to it again to see if my mind was playing tricks on me. No, the feeling of seeing depth in the sink bowl was still there.” 2002: “Just sound?.... No, It is by far more, it is sight! There IS true light perception generated by the vOICe. When I am not wearing the vOICe the light I perceive from a small slit in my left eye is a grey fog. When wearing the vOICe the image is light with all the little greys and blacks. Yet a definite light image.” - Pat Fletcher, long-term vOICe user

What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way we can experimentally change how the activity of particular neurons is interpreted downstream, and test whether they can change to acquire their newly-assigned function?

What caused the switch from cortical dominance to cortical deference? Maybe what’s important is how the information is interpreted downstream. Is there a way to experimentally manipulate the way neural activity is interpreted downstream, and test whether the neurons “oblige”?

There is a way, with brain control! With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output. This allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contribute to the object’s movement, and to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

3D center-out task under “brain control” Cosine tuning with (intended) movement direction Population Vector Algorithm

1)Adaptive: Each cell’s tuning function (PD and modulation depth) was estimated by iteratively regressing firing rate against target direction as the monkey performed the brain control task. Once the monkey’s performance stabilized, the tuning functions obtained from this iterative regression were frozen and used for decoding (“dPDs”). 2) Control 1 (C1): Monkey performed brain control using the decoding parameters obtained from the adaptive session. 3) Perturbation (P): A subset (~25%) of the recorded units (the “perturbed units”) were given reassigned dPDs by rotating their original dPDs 90 degrees about the x, y, or z axis (chosen randomly each day). 4) Control 2 (C2): The perturbation was removed (i.e. the original dPDs were reinstated). Experimental setup: 4 sessions

Direction of perturbation Deflection in early trajectory Monkey learns to control cursor by end of Perturbation session C1 Early P Late P

Possible neural bases 1)The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. 2)The perturbed units could have shifted their actual PDs toward their reassigned dPDs. 3)The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Possible neural bases 1)The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. 2)The perturbed units could have shifted their actual PDs toward their reassigned dPDs. 3)The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation.

Possible neural bases 1)The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. 2)The perturbed units could have shifted their actual PDs toward their reassigned dPDs. 3)The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation. Target Deflection caused by perturbation Re-aiming direction

Testing the possible neural bases To obtain each unit’s “measured” tuning function in each session, the same linear regression was done on all firing rates vs. target directions in that session (but post-hoc): measured PD = direction for which its firing rate was the highest (direction at the peak of the cosine fit) measured modulation depth = the magnitude of the fitted cosine (the difference between the peak firing rate and the baseline firing rate)

Modulation depths did not change

Possible neural bases 1)The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. 2)The perturbed units could have shifted their actual PDs toward their reassigned dPDs. 3)The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation. Target Deflection caused by perturbation Re-aiming direction

Testing for shift in measured PDs O azimuth M C1 0 0 MRMR R elevation M M

C1 -> P P -> C2

“Dose-response”

Possible neural bases 1)The perturbed units could have suppressed their contribution to the PV by firing at baseline rate everywhere; i.e. by decreasing their modulation depths. 2)The perturbed units could have shifted their actual PDs toward their reassigned dPDs. 3)The monkey could have “re-aimed” the cursor to offset the perturbation caused by the reassignment, disregarding the relative contributions of the perturbed vs. unperturbed units to the error. This would appear as a shift in all (perturbed and unperturbed) measured PDs in the direction of the perturbation. Target Deflection caused by perturbation Re-aiming direction

Conclusions This study demonstrates selective functional plasticity, even when only a global feedback signal is available. How can the brain do that? Perhaps noise allows it to explore synaptic weight spaces that optimize cursor control? We still don’t know where or how the changes took place that underlie the PD shifts in motor cortex. Further studies using the brain control paradigm promise a deeper understanding of plasticity as a neural substrate of learning. In combination with the dominance/deference studies, this result might have interesting implications for “neural coding”: perhaps it isn’t as important what “activates” a neuron (i.e. its “tuning function”) – as how that neuron’s activity is used downstream.

Thank you! Rob Kass Meel Velliste George Fraser Chance Spalding Ingrid Albrecht Nathaniel Daw Valerie Ventura NIH-NINDS-NO1-NS

Introduction Many studies have shown that neurons can change their “coding” properties in association with learning. In motor cortex, neurons can change their tuning functions when monkeys adapt to perturbations that interfere with the execution (e.g. Bizzi and collaborators) or visual feedback (e.g. Wise, Paz, Vaadia) of their movements. With the brain control paradigm, we know the precise relationship between the activity of each of the neurons under study and the “behavioral” output. The brain control paradigm also allows for a new kind of perturbation that can target selected subsets of neurons by altering the way those neurons contributes to the object’s movement; this makes it possible to test whether neurons can change their activity specifically and selectively when the monkey learns to compensate for the perturbation.

Cursor deflection resulting from perturbation ExpectedActual Rotation axes: red = X green = Y blue = Z

Expected cursor perturbations rotation axis: X = red Y = green Z = blue

The “Population Vector” algorithm: = firing rate at time t = baseline firing rate = cell’s preferred direction = (intended) movement velocity at time t = angle between the movement direction and the cell’s preferred direction = normalized preferred direction of cell i = maximum firing rate of cell i Estimating preferred directions:

Re-aiming occurs later in trajectory