DNA polymerase ε (POLE) is a four-subunit complex and the major leading strand polymerase in eukaryotes. Budding yeast orthologs of POLE3 and POLE4 promote Polε processivity in vitro but are dispensable for viability in vivo. Here, we report that POLE4 deficiency in mice destabilizes the entire Polε complex, leading to embryonic lethality in inbred strains and extensive developmental abnormalities, leukopenia, and tumor predisposition in outbred strains. Comparable phenotypes of growth retardation and immunodeficiency are also observed in human patients harboring destabilizing mutations in POLE1. In both Pole4-/- mouse and POLE1 mutant human cells, Polε hypomorphy is associated with replication stress and p53 activation, which we attribute to inefficient replication origin firing. Strikingly, removing p53 is sufficient to rescue embryonic lethality and all developmental abnormalities in Pole4 null mice. However, Pole4-/-p53+/- mice exhibit accelerated tumorigenesis, revealing an important role for controlled CMG and origin activation in normal development and tumor prevention.