Archive for the ‘Reward’ Category

A Brief Introduction to Reinforcement Learning

Monday, June 2nd, 2008

Computational models that are implemented, i.e., written out as equations or software, are an increasingly important tool for the cognitive neuroscientist.  This is because implemented models are, effectively, hypotheses that have been worked out to the point where they make quantitative predictions about behavior and/or neural activity.

In earlier posts, we outlined two computational models of learning hypothesized to occur in various parts of the brain, i.e., Hebbian-like LTP (here and here) and error-correction learning (here and here). The computational model described in this post contains hypotheses about how we learn to make choices based on reward.

The goal of this post is to introduce a third type of learning: Reinforcement Learning (RL).  RL is hypothesized by a number of cognitive neuroscientists to be implemented by the basal ganglia/dopamine system.  It has become somewhat of a hot topic in Cognitive Neuroscience and received a lot of coverage at this past year’s Computational Cognitive Neuroscience Conference. (more…)

The role of reward and cognitive control in decision making

Monday, September 24th, 2007

Here’s an exchange of emails between PL and MC on a recently published paper (Balleine et al., 2007).

Email 1 (from PL):
Have a look at this introductory paragraph from a recent (Aug 2007) J Neurosci article by Balleine, Delgado and Hikosaka. What do they mean by “cognition” here?

The Role of the Dorsal Striatum in Reward and Decision-Making
To choose appropriately between distinct courses of action requires the ability to integrate an estimate of the causal relationship between an action and its consequences, or outcome, with the value, or utility, of the outcome. Any attempt to base decision-making solely on cognition fails fully to determine action selection because any information, such as “action A leads to outcome O,” can be used both to perform A and to avoid performing A. It is interesting to note in this context that, although there is an extensive literature linking the cognitive control of executive functions specifically to the prefrontal cortex (Goldman-Rakic, 1995; Fuster, 2000), more recent studies suggest that these functions depend on reward-related circuitry linking prefrontal, premotor, and sensorimotor cortices with the striatum (Chang et al., 2002; Lauwereyns et al., 2002; Tanaka et al.,2006).


Email 2 (from MC):

It sounds like they are distinguishing cognition from reward processing. I’m not sure why, since ‘cognition’ typically encompasses reward processing now days.

The distinction I think they’re really trying to make is between cognitive control and reward processing. Given that, it’s still a ridiculous paragraph. Why must it be either cognitive control or reward processing? It’s likely (no, virtually certain!) that the two interact during reward-based decision making. For instance, O’Reilly’s stuff shows how this might happen.

Another problem with this paragraph: They equate causal knowledge with cognitive control. Well-known causal knowledge doesn’t involve cognitive control! For instance, routine decision making would involve lower perceptuo-motor circuits, and if it involved differential reward then reward circuits would be engaged as well. Cognitive control has little/no role here.

When cognitive control is involved it’s probably doing a lot more than just retrieving causal relations from semantic memory. For instance, perceptual decision making studies show that cognitive control is involved even in deciding what is being perceived when uncertainty arises.

I guess what they’re trying to do is show that cognitive control doesn’t explain all of decision making since there must be a reward component as well. Perhaps this is a good point to make; they just didn’t do it well.


Email 3 (from PL):

Ahhh, ok I think I see now what they’re trying to say.  It really just struck me as an excessively divisive statement to start out what appeared to be an interesting article.  Can you say “flamebait”?  Perhaps they’re trying to be provocative.

- PL & MC