During the first four centuries after Joshua, the Israelites did not occupy the city of Jerusalem. In fact, most of the land on which the city of Jerusalem sits was rural.
On one of the mountains, there was a small enclave of Jebusites who were not Israelites.
Soon after Saul's death, David decided to take that fortress. One of the accounts is found in 2 Samuel 5:6-8. In that account, it is recorded that the fortress was taken by Israelites climbing up a water shaft.
A gentile fortress dating back to the time before Israel's conquest has been excavated in the part of Jerusalem known as the "City of David." It is called "The Spring Citadel" because it was located over the spring which the previous occupants used to draw water.