Electron-positron annihilation largely occurs in local thermal and chemical equilibrium after the neutrinos fall out of thermal equilibrium and during the big bang nucleosynthesis (BBN) epoch. The effects of this process are evident in BBN yields as well as the relativistic degrees of freedom. We self-consistently calculate the collision integral for electron-positron creation and annihilation using the Klein-Nishina amplitude and appropriate statistical factors for Fermi blocking and Bose enhancement. Our calculations suggest that this annihilation freezes out when the photon-electron-positron-baryon plasma temperature is approximately 16 keV, after which its rate drops below the Hubble rate. In the temperature regime near 16 keV, we break the assumption of chemical equilibrium between the electrons, positrons, and photons to independently calculate the evolution of the chemical potentials of the electrons and positrons while computing the associated collision integrals at every time step. We find that the electron and positron chemical potentials deviate from the case with chemical equilibrium. While our results do not affect the interpretation of precision cosmological measurements in elucidating the standard cosmological model, these out of equilibrium effects may be important for testing physics beyond the standard model.