资源环境科技发展态势分析平台

Since its introduction, the reward prediction error theory of dopamine has explained a wealth of empirical phenomena, providing a unifying framework for understanding the representation of reward and value in the brain(1-3). According to the now canonical theory, reward predictions are represented as a single scalar quantity, which supports learning about the expectation, or mean, of stochastic outcomes. Here we propose an account of dopamine-based reinforcement learning inspired by recent artificial intelligence research on distributional reinforcement learning(4-6). We hypothesized that the brain represents possible future rewards not as a single mean, but instead as a probability distribution, effectively representing multiple future outcomes simultaneously and in parallel. This idea implies a set of empirical predictions, which we tested using single-unit recordings from mouse ventral tegmental area. Our findings provide strong evidence for a neural realization of distributional reinforcement learning.

Analyses of single-cell recordings from mouse ventral tegmental area are consistent with a model of reinforcement learning in which the brain represents possible future rewards not as a single mean of stochastic outcomes, as in the canonical model, but instead as a probability distribution.

Detection of dopamine dips by neurons that express dopamine D2 receptors in the striatum is used to refine generalized reward conditioning mediated by dopamine D1 receptors.

Dopamine D2 receptors (D2Rs) are densely expressed in the striatum and have been linked to neuropsychiatric disorders such as schizophrenia(1,2). High-affinity binding of dopamine suggests that D2Rs detect transient reductions in dopamine concentration (the dopamine dip) during punishment learning(3-5). However, the nature and cellular basis of D2R-dependent behaviour are unclear. Here we show that tone reward conditioning induces marked stimulus generalization in a manner that depends on dopamine D1 receptors (D1Rs) in the nucleus accumbens (NAc) of mice, and that discrimination learning refines the conditioning using a dopamine dip. In NAc slices, a narrow dopamine dip (as short as 0.4 s) was detected by D2Rs to disinhibit adenosine A(2A) receptor (A(2A)R)-mediated enlargement of dendritic spines in D2R-expressing spiny projection neurons (D2-SPNs). Plasticity-related signalling by Ca2+/calmodulin-dependent protein kinase II and A(2A)Rs in the NAc was required for discrimination learning. By contrast, extinction learning did not involve dopamine dips or D2-SPNs. Treatment with methamphetamine, which dysregulates dopamine signalling, impaired discrimination learning and spine enlargement, and these impairments were reversed by a D2R antagonist. Our data show that D2Rs refine the generalized reward learning mediated by D1Rs.