Notes for week 8

Barbara

• First, there is *an older article by Mink in Progress in Neurobiology. I include it because it contains some of the basic physiology data, and contains a very stripped-down model that characterizes the BG as a feed forward system whose job it is to inhibit behavior until released. Subsequent models employ more detailed anatomy and posit more complicated views of BG function.
• Central to all reinforcement models is how to evaluate the outcome of an action — how does a brain determine "success" or "failure" and how would "reward" or "punishment" be operationalized? I don't think there is anything remotely like a satisfactory answer to this, but Berridge and Robinson, *Parsing reward TINS 2003 has some good insights.
• *Frank and O'Reilly, in Science pull apart "Go" and "No-go" into reward and punishment learning respectively, which depends on the background of inhibition of behavior being the resting state of the BG; this has some interesting information about how Parkinson's patients are able to make reward and punishment assessments.
• Finally, there are now a large number of additional basal ganglia/ decision/sequencing models appearing in the computational literature. *Bar-Gad et al. in Progress in Neurobiology 2003 highlight dimensionality reduction, pointing out the funneling down of neuronal numbers from striatum to globus pallidus. *Joel et al. in Neural Networks (2002) have a nice review of "actor-critic" models and reinforcement learning. The abstract of a current paper by Bogacz and Gurney: *The Basal Ganglia and Cortex Implement Optimal Decision Making Between Alternative Actions

  Neurophysiological studies have identified a number of brain regions critically involved in solving the problem of action selection or decision making. In the case of highly practiced tasks, these regions include cortical areas hypothesized to integrate evidence supporting alternative actions and the basal ganglia, hypothesized to act as a central switch in gating behavioral requests. However, despite our relatively detailed knowledge of basal ganglia biology and its connectivity with the cortex and numerical simulation studies demonstrating selective function, no formal theoretical framework exists that supplies an algorithmic description of these circuits. This article shows how many aspects of the anatomy and physiology of the circuit involving the cortex and basal ganglia are exactly those required to implement the computation defined by an asymptotically optimal statistical test for decision making: the multihypothesis sequential probability ratio test (MSPRT). The resulting model of basal ganglia provides a new framework for understanding the computation in the basal ganglia during decision making in highly practiced tasks. The predictions of the theory concerning the properties of particular neuronal populations are validated in existing experimental data. Further, we show that this neurobiologically grounded implementation of MSPRT outperforms other candidates for neural decision making, that it is structurally and parametrically robust, and that it can accommodate cortical mechanisms for decision making in a way that complements those in basal ganglia.

Shimon

Last week: trying to understand the cortex - cerebellum loop

• Kawato (1999) and inverse problems:
• forward internal models can predict sensory consequences from efference copies of issued motor commands;
• inverse internal models can calculate necessary feedforward motor commands from desired trajectory information.

This week: trying to understand the cortex - basal ganglia loop

The basic explanatory move:

The details:

• Mink (1996)
• Fig.5: !!!
• Ruppin et al (2002); Bar-Gad et al (2003)
• curse of dimensionality
• reward-controlled PCA for dimensionality reduction
• Bar-Gad: Figure 2
"The RDDR model suggests that the basal ganglia play a role in extraction and pre-processing of information from the whole cortex. Why is it computationally useful? First, it allows for the transmission of large amounts of information within a limited number of axons. Bar-Gad et al. hypothesize that the basal ganglia perform dimensionality reduction of widespread cortical neural activity representing the present state of the animal. The reduced information is projected to the frontal cortex that uses it for planning future actions. The RDDR network thus enables the exposure of neurons in the executive regions of the frontal cortex to maximal incoming cortical information using the anatomically limited number of synapses that each frontal neuron can receive. Second, the RDDR network provides a vehicle by which RL may be carried out in the brain in a central, parsimonious location, by allowing the appetitive value of stimuli to guide their storage and representation. Such selective RDDR storage tends to bias the overall network's response towards rewarded input stimuli."
• Berridge and Robinson (2003)
Neuroscience folks and computation people speak different languages; compare Ruppin et al. (dimensionality reduction, etc.) to Berridge and Robinson (stuck in Pavlov's times and mindset). Here's Berridge and Robinson: "The challenge is to identify how different brain circuits mediate different psychological components of reward, and how these components interact." No, the challenge is to understand the computations that have to do with reward and to identify the brain circuits that support them.
An explanatory gem from B&R: "Learning processes can be either associative or cognitive."
• Frank et al. (2004)
• Separate GO and NO-GO systems;
• the former in charge of learning from positive outcomes, the latter of learning from negative outcomes;
• each can be implemented by the RDDR model (see above);
• stimulus/response-selective neurons within each system!
• Bogacz and Gurney (2007)
• Two roles for basal ganglia:
1. "In the acquisition or learning phase, actions appropriate in a given behavioral state (i.e., a combination of ongoing behavior and stimulus) are being developed that are usually driven by external reward. In this phase, the main difficulty lies in finding the optimal policy: the mapping between states and actions that maximizes reward. Consistent with this requirement, there is a great deal of evidence that the basal ganglia act as a substrate for reinforcement learning."
2. "In contrast, in the proficient phase, the mapping between stimulus and an appropriate response is well established, and the requirement is one of performing action selection or decision making. This consists of identifying the current behavioral state and executing the known appropriate action as soon as a certain level of confidence in the identification is reached. There is a great deal of evidence for the involvement of the basal ganglia in action selection."
• "We have argued that between bouts of learning -- that is, during proficient phases of activity -- the primary computational role of the basal ganglia is to act as an action selection mechanism, mediating resolution of the action selection problem by gating behavioral requests. It is this mode of basal ganglia operation that we address in this article."
  [NOTE: this is similar to Kawato's model of the cerebellar loop]
• Task: decide which of the x_1,2 has the highest mean. Evidence accumulation through a random walk. When to stop? An applicable criterion: Sequential Probability Ratio Test (SPRT) -- track the difference between the two means. "The SPRT is optimal in the following sense: among all decision methods allowing a certain probability of error, it requires the shortest period of sampling the x_i -- that is, it minimizes decision time (Wald & Wolfowitz, 1948)" Multiple SPRT: "The MSPRT (Baum & Veeravalli, 1994) is equivalent to the following decision criterion at time T: for each alternative i, compute the conditional probability of hypothesis H_i given sensory inputs so far, P_i (T) = P(H_i | input(T)), and execute an action as soon as any of the P_i(T) exceeds a certain decision threshold." "MSPRT can be viewed as a Bayesian method of decision making, as it is based on evaluating conditional probabilities using the Bayes theorem."
• O'Reilly (2006)
"The digital features of the PFC and associated areas help to broaden the horizons of naturally parochial neural networks. The dynamic gating mechanisms work more like a post-office, with the basal ganglia reading the zip code of which PFC stripe to update, whereas the PFC cares more about the contents of the package. Furthermore, the binary rulelike representations in the PFC are more symbol-like. Thus, perhaps a fuller understanding of this synthesis of analog and digital computation will finally unlock the mysteries of human intelligence."
• sample-and-hold functionality