You're right, with one huge caveat - humidity. In humid climates it is better to keep the windows closed to regulate the indoor temperature by keeping the indoors dry. Open windows lead to mold growth inside, especially in ventilation ducts, and condensation on pipes leading to rot and more mold.
But, yes, in desert climates this works great. My mother does this now, and on 110 F days she never needs the AC!