Synaptic connections between hippocampal mossy fibers (MFs) and CA3 pyramidal neurons are essential for contextual memory encoding, but the molecular mechanisms regulating MF-CA3 synapses during memory formation and the exact nature of this regulation are poorly understood. Here we report that the activity-dependent transcription factor Npas4 selectively regulates the structure and strength of MF-CA3 synapses by restricting the number of their functional synaptic contacts without affecting the other synaptic inputs onto CA3 pyramidal neurons. Using an activity-dependent reporter, we identified CA3 pyramidal cells that were activated by contextual learning and found that MF inputs on these cells were selectively strengthened. Deletion of Npas4 prevented both contextual memory formation and this learning-induced synaptic modification. We further show that Npas4 regulates MF-CA3 synapses by controlling the expression of the polo-like kinase Plk2. Thus, Npas4 is a critical regulator of experience-dependent, structural, and functional plasticity at MF-CA3 synapses during contextual memory formation.