The exploration/exploitation tradeoff - pursuing a known reward vs. sampling from lesser known options in the hope of finding a better payoff - is a fundamental aspect of learning and decision making. In humans, this has been studied using multi-armed bandit tasks. The same processes have also been studied using simplified probabilistic reversal learning (PRL) tasks with binary choices. Our investigations suggest that protocols previously used to explore PRL in mice may prove beyond their cognitive capacities, with animals performing at a no-better-than-chance level. We sought a novel probabilistic learning task to improve behavioral responding in mice, whilst allowing the investigation of the exploration/exploitation tradeoff in decision making. To achieve this, we developed a two-lever operant chamber task with levers corresponding to different probabilities (high/low) of receiving a saccharin reward, reversing the reward contingencies associated with levers once animals reached a threshold of 80% responding at the high rewarding lever. We found that, unlike in existing PRL tasks, mice are able to learn and behave near optimally with 80% high/20% low reward probabilities. Altering the reward contingencies towards equality showed that some mice displayed preference for the high rewarding lever with probabilities as close as 60% high/40% low. Additionally, we show that animal choice behavior can be effectively modelled using reinforcement learning (RL) models incorporating learning rates for positive and negative prediction error, a perseveration parameter, and a noise parameter. This new decision task, coupled with RL analyses, advances access to investigate the neuroscience of the exploration/exploitation tradeoff in decision making.

Adjusting behavior to changed environmental contingencies is critical for survival, and reversal learning provides an experimental handle on such cognitive flexibility. Here, we investigate reversal learning in larval Drosophila Using odor-taste associations, we establish olfactory reversal learning in the appetitive and the aversive domain, using either fructose as a reward or high-concentration sodium chloride as a punishment, respectively. Reversal learning is demonstrated both in differential and in absolute conditioning, in either valence domain. In differential conditioning, the animals are first trained such that an odor A is paired, for example, with the reward whereas odor B is not (A+/B); this is followed by a second training phase with reversed contingencies (A/B+). In absolute conditioning, odor B is omitted, such that the animals are first trained with paired presentations of A and reward, followed by unpaired training in the second training phase. Our results reveal "true" reversal learning in that the opposite associative effects of both the first and the second training phase are detectable after reversed-contingency training. In what is a surprisingly quick, one-trial contingency adjustment in the Drosophila larva, the present study establishes a simple and genetically easy accessible study case of cognitive flexibility.

It has been proposed that reversal learning is impaired following damage to the orbitofrontal and ventromedial frontal cortex (OFC/VMFC) and to the medial temporal lobe (MTL), including the hippocampal formation. However, the exact characteristics of the MTL-associated reversal learning deficit are not known. To investigate this issue, we assessed 30 newly diagnosed patients with amnestic mild cognitive impairment (aMCI) and 30 matched healthy controls. All patients fulfilled the aMCI criteria of the Mayo Clinic Alzheimer's Disease Research Center and underwent head magnetic resonance imaging that confirmed MTL atrophy. Reversal learning was assessed using a novel reinforcement learning task. Participants first acquired and then reversed stimulus-outcome associations based on negative and positive feedback (losing and gaining points). Stimuli consisted of a cue (geometric shapes) and a spatial context (background color or pattern). Neuropsychological assessment included tasks related to the MTL (paired associates learning), dorsolateral prefrontal cortex (DLPFC) (extradimensional shift, One-touch Stockings of Cambridge), and OFC/VMFC (Holiday Apartment Task). Results revealed that, relative to controls, patients with aMCI exhibited a marked reversal learning deficit, which was highly selective for the reversal of context. The acquisition of stimulus-outcome associations and cue reversal learning were spared. Performance on the context reversal learning task significantly correlated with the right hippocampal volume. In addition, patients with aMCI had deficits on tests related to DLPFC but not to OFC/VMFC. However, DLPFC dysfunctions were not associated with context reversal learning. These results suggest that MTL deficits in aMCI selectively affect context reversal learning when OFC/VMFC functions are spared. This deficit is not influenced by the valence of the outcome (positive or negative feedback) and by executive dysfunctions.

Biologically plausible modeling of behavioral reinforcement learning tasks has seen great improvements over the past decades. Less work has been dedicated to tasks involving contingency reversals, i.e., tasks in which the original behavioral goal is reversed one or multiple times. The ability to adjust to such reversals is a key element of behavioral flexibility. Here, we investigate the neural mechanisms underlying contingency-reversal tasks. We first conduct experiments with humans and gerbils to demonstrate memory effects, including multiple reversals in which subjects (humans and animals) show a faster learning rate when a previously learned contingency re-appears. Motivated by recurrent mechanisms of learning and memory for object categories, we propose a network architecture which involves reinforcement learning to steer an orienting system that monitors the success in reward acquisition. We suggest that a model sensory system provides feature representations which are further processed by category-related subnetworks which constitute a neural analog of expert networks. Categories are selected dynamically in a competitive field and predict the expected reward. Learning occurs in sequentialized phases to selectively focus the weight adaptation to synapses in the hierarchical network and modulate their weight changes by a global modulator signal. The orienting subsystem itself learns to bias the competition in the presence of continuous monotonic reward accumulation. In case of sudden changes in the discrepancy of predicted and acquired reward the activated motor category can be switched. We suggest that this subsystem is composed of a hierarchically organized network of dis-inhibitory mechanisms, dubbed a dynamic control network (DCN), which resembles components of the basal ganglia. The DCN selectively activates an expert network, corresponding to the current behavioral strategy. The trace of the accumulated reward is monitored such that large sudden deviations from the monotonicity of its evolution trigger a reset after which another expert subnetwork can be activated-if it has already been established before-or new categories can be recruited and associated with novel behavioral patterns.

The intermediate filament protein nestin is expressed by neural stem cells, but also by some astrocytes in the neurogenic niche of the hippocampus in the adult rodent brain. We recently reported that nestin-deficient (Nes-/-) mice showed increased adult hippocampal neurogenesis, reduced Notch signaling from Nes-/- astrocytes to the neural stem cells, and impaired long-term memory. Here we assessed learning and memory of Nes-/- mice in a home cage set up using the IntelliCage system, in which the mice learn in which cage corner a nose poke earns access to drinking water. Nes-/- and wildtype mice showed comparable place learning assessed as the incorrect corner visit ratio and the incorrect nose poke ratio. However, during reversal place learning, a more challenging task, Nes-/- mice, compared to wildtype mice, showed improved learning over time demonstrated by the incorrect visit ratio and improved memory extinction over time assessed as nose pokes per visit to the previous drinking corner. In addition, Nes-/- mice showed increased explorative activity as judged by the increased total numbers of corner visits and nose pokes. We conclude that Nes-/- mice exhibit improved reversal place learning and memory extinction, a finding which together with the previous results supports the concept of the dual role of hippocampal neurogenesis in cognitive functions.

Frontotemporal Dementia (FTD) is a neurodegenerative disorder that results in disinhibition and difficulty with flexible responding when provided feedback. Inflexible responding is observed early in the course of the illness and contributes to the financial and social morbidities of FTD. Reversal learning is an established cognitive paradigm that indexes flexible responding in the face of feedback signaling a change in reinforcement contingencies, with components of reversal learning associated with specific neurotransmitter systems. The objective of the study was to evaluate the neural mechanisms underlying impaired flexible behavioural responding in FTD using a reversal learning paradigm combined with fMRI.

Reversal learning assays are commonly used across a wide range of taxa to investigate associative learning and behavioural flexibility. In serial reversal learning, the reward contingency in a binary discrimination is reversed multiple times. Performance during serial reversal learning varies greatly at the interspecific level, as some animals adopt a rule-based strategy that enables them to switch quickly between reward contingencies. A larger relative brain size, generating enhanced learning ability and increased behavioural flexibility, has been proposed to be an important factor underlying this variation. Here, we experimentally tested this hypothesis at the intraspecific level. We used guppies (Poecilia reticulata) artificially selected for small and large relative brain size, with matching differences in neuron number, in a serial reversal learning assay. We tested 96 individuals over 10 serial reversals and found that learning performance and memory were predicted by brain size, whereas differences in efficient learning strategies were not. We conclude that variation in brain size and neuron number is important for variation in learning performance and memory, but these differences are not great enough to cause the larger differences in efficient learning strategies observed at higher taxonomic levels.

Electroencephalogram (EEG) studies suggest an association between beta (13-30 Hz) power and reversal learning performance. In search for direct evidence concerning the involvement of beta oscillations in reversal learning, transcranial alternating current stimulation (tACS) was applied in a double-blind, sham-controlled and between-subjects design. Exogenous oscillatory currents were administered bilaterally to the frontal cortex at 20 Hz with an intensity of 1 mA peak-to-peak and the effects on reward-punishment based reversal learning were evaluated in hundred-and-eight healthy volunteers. Pre- and post-tACS resting state EEG recordings were analyzed. Results showed that beta-tACS improved rule implementation during reversal learning and decreases left and right resting-state frontal theta/beta EEG ratios following tACS. Our findings provide the first behavioral and electrophysiological evidence for exogenous 20 Hz oscillatory electric field potentials administered over to the frontal cortex to improve reversal learning.

In reversal learning tasks, the behavior of humans and animals is often assumed to be uniform within single experimental sessions to facilitate data analysis and model fitting. However, behavior of agents can display substantial variability in single experimental sessions, as they execute different blocks of trials with different transition dynamics. Here, we observed that in a deterministic reversal learning task, mice display noisy and sub-optimal choice transitions even at the expert stages of learning. We investigated two sources of the sub-optimality in the behavior. First, we found that mice exhibit a high lapse rate during task execution, as they reverted to unrewarded directions after choice transitions. Second, we unexpectedly found that a majority of mice did not execute a uniform strategy, but rather mixed between several behavioral modes with different transition dynamics. We quantified the use of such mixtures with a state-space model, block Hidden Markov Model (block HMM), to dissociate the mixtures of dynamic choice transitions in individual blocks of trials. Additionally, we found that blockHMM transition modes in rodent behavior can be accounted for by two different types of behavioral algorithms, model-free or inference-based learning, that might be used to solve the task. Combining these approaches, we found that mice used a mixture of both exploratory, model-free strategies and deterministic, inference-based behavior in the task, explaining their overall noisy choice sequences. Together, our combined computational approach highlights intrinsic sources of noise in rodent reversal learning behavior and provides a richer description of behavior than conventional techniques, while uncovering the hidden states that underlie the block-by-block transitions.

Each day, we make hundreds of decisions. In some instances, these decisions are guided by our innate needs; in other instances they are guided by memory. Probabilistic reversal learning tasks exemplify the close relationship between decision making and memory, as subjects are exposed to repeated pairings of a stimulus choice with a reward or punishment outcome. After stimulus-outcome associations have been learned, the associated reward contingencies are reversed, and participants are not immediately aware of this reversal. Individual differences in the tendency to choose the previously rewarded stimulus reveal differences in the tendency to make poorly considered, inflexible choices. Lesion studies have strongly linked reversal learning performance to the functioning of the orbitofrontal cortex, the hippocampus, and in some instances, the amygdala. Here, we asked whether individual differences in the microstructure of the uncinate fasciculus, a white matter tract that connects anterior and medial temporal lobe regions to the orbitofrontal cortex, predict reversal learning performance. Diffusion tensor imaging and behavioral paradigms were used to examine this relationship in 33 healthy young adults. The results of tractography revealed a significant negative relationship between reversal learning performance and uncinate axial diffusivity, but no such relationship was demonstrated in a control tract, the inferior longitudinal fasciculus. Our findings suggest that the uncinate might serve to integrate associations stored in the anterior and medial temporal lobes with expectations about expected value based on feedback history, computed in the orbitofrontal cortex.

The dopaminergic system is firmly implicated in reversal learning but human measurements of dopamine release as a correlate of reversal learning success are lacking. Dopamine release and hemodynamic brain activity in response to unexpected changes in action-outcome probabilities are here explored using simultaneous dynamic [11C]Raclopride PET-fMRI and computational modelling of behavior. When participants encounter reversed reward probabilities during a card guessing game, dopamine release is observed in associative striatum. Individual differences in absolute reward prediction error and sensitivity to errors are associated with peak dopamine receptor occupancy. The fMRI response to perseverance errors at the onset of a reversal spatially overlap with the site of dopamine release. Trial-by-trial fMRI correlates of absolute prediction errors show a response in striatum and association cortices, closely overlapping with the location of dopamine release, and separable from a valence signal in ventral striatum. The results converge to implicate striatal dopamine release in associative striatum as a central component of reversal learning, possibly signifying the need for increased cognitive control when new stimuli-responses should be learned.

Our world is full of uncertainty. Animals, including humans, need to behave flexibly to adjust to ever-changing environments. Reversal learning tasks have been used to assess behavioral flexibility in many species. However, there are some limitations in the traditional free-moving methodology, including (1) sessions to train the animals, (2) within-session number of trials associated with reversals, (3) factors of physical movement unrelated to the task in the maze or operant box, and (4) incompatibility with techniques, such as two-photon imaging. Therefore, to address these limitations, we established a novel spatiotemporal Pavlovian head-fixed reversal learning task for mice. Six experimentally naive adult C57BL/6J mice were used in this study. First, we trained head-fixed mice on a fixed-time schedule task. Sucrose solution was delivered every 10 s with a single drinking spout placed within the licking distance of the mice. After the mice showed anticipatory licking toward the timing of sucrose solution delivery, we began training the mice on the fixed-time schedule reversal learning task with two licking spouts. In this task, sucrose solution was delivered through one of the two drinking spouts. The rewarding spout was switched every 10 trials. Mice quickly learned to switch anticipatory licking to the rewarding side of the spouts, suggesting that they learned this head-fixed reversal learning task. Using the head-fixed experimental design, behavioral measures can be simplified by eliminating the complex behavioral sequences observed in free-moving animals. This novel head-fixed reversal learning task is a useful assay for studying the neurobiological mechanism of behavioral flexibility that is impaired in various psychopathological conditions.

Progressively improving performance in a serial reversal learning (SRL) test has been associated with higher cognitive abilities and has served as a measure for cognitive/behavioral flexibility. Although the cognitive and sensory abilities of marine mammals have been subject of extensive investigation, and numerous vertebrate and invertebrate species were tested, SRL studies in aquatic mammals are sparse. Particularly in pinnipeds, a high degree of behavioral flexibility seems probable as they face a highly variable environment in air and underwater. Thus, we tested four harbor seals in a visual two-alternative forced-choice discrimination task and its subsequent reversals. We found significant individual differences in performance. One individual was able to solve 37 reversals showing progressive improvement of performance with a minimum of 6 errors in reversal 33. Two seals mastered two reversals, while one animal had difficulties in learning the discrimination task and failed to complete a single reversal. In conclusion, harbor seals can master an SRL experiment; however, the performance is inferior to results obtained in other vertebrates in comparable tasks. Future experiments will need to assess whether factors such as the modality addressed in the experiment have an influence on reversal learning performance or whether indeed, during evolution, behavioral flexibility has not specifically been favored in harbor seals.

Reversal learning is thought to involve an extinction-like process that inhibits the expression of the initial learning. However, behavioral evidence for this inhibition remains difficult to interpret as various procedures have been employed to study reversal learning. Here, we used a discrimination task in rats to examine whether the inhibition produced by reversal learning is as sensitive to the passage of time as the inhibition produced by extinction. Experiment 1 showed that when tested immediately after reversal training, rats were able to use the reversed contingencies to solve the discrimination task in an outcome-specific manner. This ability to use outcome-specific information was lost when a delay was inserted between reversal training and test. However, interpretation of these data was made difficult by a potential floor effect. This concern was addressed in Experiment 2 in which it was confirmed that the passage of time impaired the ability of the rats to use the reversed contingencies in an outcome-specific manner to solve the task. Further, it revealed that the delay between initial learning and test was not responsible for this impairment. Additional work demonstrated that solving the discrimination task was unaffected by Pavlovian extinction but that the discriminative stimuli were able to block conditioning to a novel stimulus, suggesting that Pavlovian processes were likely to contribute to solving the discrimination. We therefore concluded that the expression of reversal and extinction learning do share the same sensitivity to the effect of time. However, this sensitivity was most obvious when we assessed outcome-specific information following reversal learning. This suggests that the processes involved in reversal learning are somehow distinct from those underlying extinction learning, as the latter has usually been found to leave outcome-specific information relatively intact. Thus, the present study reveals that a better understanding of the mechanisms supporting reversal training requires assessing the impact that this training exerts on the content of learning rather than performance per se.

Cognitive flexibility helps us to navigate through our ever-changing environment and has often been examined by reversal learning paradigms. Performance in reversal learning can be modeled using computational modeling which allows for the specification of biologically plausible models to infer psychological mechanisms. Although such models are increasingly used in cognitive neuroscience, developmental approaches are still scarce. Additionally, though most reversal learning paradigms have a comparable design regarding timing and feedback contingencies, the type of feedback differs substantially between studies. The present study used hierarchical Gaussian filter modeling to investigate cognitive flexibility in reversal learning in children and adolescents and the effect of various feedback types. The results demonstrate that children make more overall errors and regressive errors (when a previously learned response rule is chosen instead of the new correct response after the initial shift to the new correct target), but less perseverative errors (when a previously learned response set continues to be used despite a reversal) adolescents. Analyses of the extracted model parameters of the winning model revealed that children seem to use new and conflicting information less readily than adolescents to update their stimulus-reward associations. Furthermore, more subclinical rigidity in everyday life (parent-ratings) is related to less explorative choice behavior during the probabilistic reversal learning task. Taken together, this study provides first-time data on the development of the underlying processes of cognitive flexibility using computational modeling.

Impairments in flexible goal-directed decisions, often examined by reversal learning, are associated with behavioral abnormalities characterized by impulsiveness and disinhibition. Although the lateral orbital frontal cortex (OFC) has been consistently implicated in reversal learning, it is still unclear whether this region is involved in negative feedback processing, behavioral control, or both, and whether reward and punishment might have different effects on lateral OFC involvement. Using a relatively large sample (N = 47), and a categorical learning task with either monetary reward or moderate electric shock as feedback, we found overlapping activations in the right lateral OFC (and adjacent insula) for reward and punishment reversal learning when comparing correct reversal trials with correct acquisition trials, whereas we found overlapping activations in the right dorsolateral prefrontal cortex (DLPFC) when negative feedback signaled contingency change. The right lateral OFC and DLPFC also showed greater sensitivity to punishment than did their left homologues, indicating an asymmetry in how punishment is processed. We propose that the right lateral OFC and anterior insula are important for transforming affective feedback to behavioral adjustment, whereas the right DLPFC is involved in higher level attention control. These results provide insight into the neural mechanisms of reversal learning and behavioral flexibility, which can be leveraged to understand risky behaviors among vulnerable populations.

Mindfulness-based interventions are thought to reduce compulsive behavior such as overeating by promoting behavioral flexibility. Here the main aim was to provide support for mindfulness-mediated improvements in reversal learning, a direct measure of behavioral flexibility. We investigated whether an 8-week mindful eating intervention improved outcome-based reversal learning relative to an educational cooking (i.e., active control) intervention in a non-clinical population. Sixty-five healthy participants with a wide BMI range (19-35 kg/m2), who were motivated to change their eating habits, performed a deterministic reversal learning task that enabled the investigation of reward- and punishment-based reversal learning at baseline and following the intervention. No group differences in reversal learning were observed. However, time invested in the mindful eating, but not the educational cooking intervention correlated positively with changes in reversal learning, in a manner independent of outcome valence. These findings suggest that greater amount of mindfulness practice can lead to increased behavioral flexibility, which, in turn, might help overcome compulsive eating in clinical populations.

The neural mechanisms that allow animals to adapt their previously learned associations in response to changes in the environment remain poorly understood. To probe the synaptic mechanisms that mediate such adaptive behavior, we trained mice on an auditory-motor reversal task, and tracked changes in the strength of corticostriatal synapses associated with the formation of learned associations. Using a ChR2-based electrophysiological assay in acute striatal slices, we measured the strength of these synapses after animals learned to pair auditory stimuli with specific actions. Here, we report that the pattern of synaptic strength initially established by learning remains unchanged even when the task contingencies are reversed. Our findings reveal that synaptic changes associated with the initial acquisition of this task are not erased or overwritten, and that behavioral reversal of learned associations may recruit a separate neural circuit. These results suggest a more complex role of the striatum in regulating flexible behaviors where activity of striatal neurons may vary given the behavioral contexts of specific stimulus-action associations.

Although several studies showed adverse neurotoxic effects of melamine on hippocampus (HPC)-dependent learning and reversal learning, the evidence for this mechanism is still unknown. We recently demonstrated that intra-hippocampal melamine injection affected the induction of long-term depression, which is associated with novelty acquisition and memory consolidation. Here, we infused melamine into the HPC of rats, and employed behavioral tests, immunoblotting, immunocytochemistry and electrophysiological methods to sought evidence for its effects on cognitive flexibility. Rats with intra-hippocampal infusion of melamine displayed dose-dependent increase in trials to the criterion in reversal learning, with no locomotion or motivation defect. Compared with controls, melamine-treated rats avoided HPC-dependent place strategy. Meanwhile, the learning-induced BDNF level in the HPC neurons was significantly reduced. Importantly, bilateral intra-hippocampal BDNF infusion could effectively mitigate the suppressive effects of melamine on neural correlate with reversal performance, and rescue the strategy bias and reversal learning deficits. Our findings provide first evidence for the effect of melamine on cognitive flexibility and suggest that the reversal learning deficit is due to the inability to use place strategy. Furthermore, the suppressive effects of melamine on BDNF-mediated neural activity could be the mechanism, thus advancing the understanding of compulsive behavior in melamine-induced and other neuropsychiatric disorders.

Behavioral flexibility enables the ability to adaptively respond to changes in contingency requirements to maintain access to desired outcomes, and deficits in behavioral flexibility have been documented in many psychiatric disorders. Previous research has shown a correlation between behavioral flexibility measured in a reversal learning test and Syn3, the gene encoding synapsin III, which negatively regulates phasic dopamine release. Syn3 expression in the hippocampus, striatum, and neocortex is reported to be negatively correlated with reversal learning performance, so here, we used a global knock-out line to investigate reversal learning in mice homozygous wild type, heterozygous null, and homozygous null for the Syn3 gene. Compared with wild-type animals, we found a reversal-specific effect of genetic Syn3 deficiency that resulted in a greater proportional increase in trials required to reach a preset performance criterion during contingency reversal, despite no observed genotype effects on the ability to acquire the initial discrimination. Behavioral flexibility scores, which quantified the likelihood of switching subsequent choice behavior following positive or negative feedback, became significantly more negative in reversal only for Syn3 homozygous-null mice, suggesting a substantial increase in perseverative behavior in the reversal phase. Syn3 ablation reduced the number of anticipatory responses made per trial, often interpreted as a measure of waiting impulsivity. Overall, Syn3 expression negatively affected behavioral flexibility in a reversal-specific manner but may have reduced waiting impulsivity.