With a little digging, you can discover that the Standard Atmosphere can be traced back to 1920. The constant lapse rate of 6.5° per km in the troposphere was suggested by Prof. Toussaint, on the grounds that
… what is needed is … merely a law that can be conveniently applied and which is sufficiently in concordance with the means adhered to. By this method, corrections due to temperature will be as small as possible in calculations of airplane performance, and will be easy to calculate. …
The deviation is of some slight importance only at altitudes below 1,000 meters, which altitudes are of little interest in aerial navigation. The simplicity of the formula largely compensates this inconvenience.
The above quotation is from the paper by
Gregg (1920). The early motivations for this simplified model were evidently the calibration of aneroid altimeters for aircraft, and the construction of firing tables for long-range artillery, where air resistance is important.
Unfortunately, it is precisely the inaccurate region below 1000 m that is most important for refraction near the horizon. However, the Toussaint lapse rate, which Gregg calls "arbitrary", is now embodied in so many altimeters that it cannot be altered: all revisions of the Standard Atmosphere have preserved it.
Therefore, the Standard Atmosphere is really inappropriate for astronomical refraction calculations.
A more realistic model would include the diurnal changes in the boundary layer; but these are still so poorly understood that no satisfactory basis seems to exist for realistic refraction tables near the horizon.