Your first chance to buy a house will be in Whiterun. After completing the quest offered by the Jarl, located in Dragonsreach, you will be able to buy a house (Breezehome) for 5000 gold from his steward. It's the second house on your right when you enter Whiterun.
If you want to buy a (more expensive) house in another Hold, you will need to complete a quest for the respective holds' Jarl, just like in Whiterun. Do their quest and the house will be available for purchase from their steward for the following prices:
- Vlindrell Hall, Markarth (8000 gold)
- Honeyside, Riften (8000 gold)
- Hjerim, Windhelm (12000 gold)
- Proudspire Manor, Solitude (25000 gold)