It's a tough one. The TZ-10 is a cracking little camera, I really can't recommend it enough as an all-rounder. I don't know the specs of the TZ-20, I've not actually been and looked but I imagine it'll be "more evolution than revolution" as reviewers like to say.
The Fuji X100 is gorgeous but it's very very very very very expensive for what it is - like an iPhone for camera nerds I suppose, which is probably why I want one!
The others that I like are the high end models. The Panasonic LX3 and LX5 really appeal but they're not as all round flexible as something like the TZ's and other similar models.
I think they key is to know the limitations of what you're buying and as an enthusiast you know what those limitations are. In effect, you're not buying blindly like you're average punter on the high street. Make an educated decision and you pretty much can't go wrong.