A recent skill

According to those who study this anthropology thing, humans have only been building permanent dwellings for the last ten thousand years - the same time they started farming.
Which raises the question, wtf were they doing for the previous 290,000 years? In all that time did no one ever think of planting more of the stuff that tasted good and maybe building a home nearby?