If you could swing a 15 year mortgage, you'd have your current place paid off free and clear by the time you're 47. That would put you on track for an early retirement, and still allow you to keep the place you have now. Depending on where things are at that point, you could make the decision to keep it or downsize then.
On the other hand, being debt free opens up a lot of options too. If you're a good saver and investor, you'd pretty much guarantee an early retirement by downsizing now and socking away as much as you can.
Of course a lot depends on what you mean by early retirement. I'm assuming you mean getting out of the rat race sometime in your early fifties. Given where life expectancies are today, that would most likely give you a thirty or forty year retirement. That's a long time, and what you need to be shooting for from a financial standpoint.
Either way, the fact that you're planning that far ahead tells me you'll probably do just fine as far as getting out early.
