The common olive (olea europaea) is not winter hardy in zone 4 and prospers only in a dry, Mediterranean-like climate in zones 8 to 10. In colder climates, the tree may be grown in a greenhouse or a container that is moved indoors to a cool and protected spot in the winter.  Winter tempertures need to stay in the 40 to 50 degree F. range, with at least two months kept below 50 degrees in order to  generate flowering and fruit.

Courtesy of The NYBG Plant Information Service