Regardless of the brand there is one thing to consider: does your camera body limit you right now? Is there something you cannot do because it cannot? I do not mean it does but a new one does it better, I mean is there something you want to do that you need a new body for?
If the answer is no, then buy new lenses
Now, to your brand. I have heard that the Olympus's models have a problem with noise. I have heard this from people with them and read it in reviews. If that noise keeps you from doing something (see, I am going back to my point above) then yes it may be time to upgrade.
When you switch camera brands, I am sure they will all feel alien. I am sure someone that had done so can give you some help with that.