Without eggs, the temp inside the egg box and outside the egg box should be the same if it's had time to equalize. The temp inside the egg box is what's important, but you need to know why they're different. My guess is you have a heat gradient where the higher region of the incubator is warmer than the lower region (stating the obvious = heat rises). Do you have any kind of fan circulating the air in the incubator to keep the temps stable and equal in all regions?
Just reread your post and you don't have a fan - that will create uneven hot spots in the inc.