Astrophysical observations of neutron stars probe the structure of dense nuclear matter and have the potential to reveal phase transitions at high densities. Most recent analyses are based on parametrized models of the equation of state with a finite number of parameters and occasionally include extra parameters intended to capture phase transition phenomenology. However, such models restrict the types of behavior allowed and may not match the true equation of state. We introduce a complementary approach that extracts phase transitions directly from the equation of state without relying on, and thus being restricted by, an underlying parametrization. We then constrain the presence of phase transitions in neutron stars with astrophysical data. Current pulsar mass, tidal deformability, and mass-radius measurements disfavor only the strongest of possible phase transitions (latent energy per particle ). Weaker phase transitions are consistent with observations. We further investigate the prospects for measuring phase transitions with future gravitational-wave observations and find that catalogs of events will (at best) yield Bayes factors of in favor of phase transitions even when the true equation of state contains very strong phase transitions. Our results reinforce the idea that neutron star observations will primarily constrain trends in macroscopic properties rather than detailed microscopic behavior. Fine-tuned equation of state models will likely remain unconstrained in the near future.