Dark matter (DM) could be a relic of freeze-in through a light mediator, where the DM is produced by extremely feeble, IR-dominated processes in the thermal Standard Model plasma. In the simplest viable models with the DM mass below the MeV scale, the DM has a small effective electric charge and is born with a nonthermal phase-space distribution. This DM candidate would cause observable departures from standard cosmological evolution. In this work, we combine data from the cosmic microwave background (CMB), Lyman-α forest, quasar lensing, stellar streams, and Milky Way satellite abundances to set a lower limit on freeze-in DM masses up to ∼ 20 keV, with the exact constraint depending on whether the DM thermalizes in its own sector. We perform forecasts for the CMB-S4 experiment, the Hydrogen Epoch of Reionization Array, and the Vera Rubin Observatory, finding that freeze-in DM masses up to ∼ 80 keV can be explored. These cosmological probes are highly complementary with proposed direct-detection efforts to search for this DM candidate.