My Dr told me I have to stay in town AT LEAST one week after surgery, that day he'll take off my cast and then I can fly home. I was thinking of doing the same thing your thinking but he basically forbids me to fly out any sooner. After that you should be fine. I posted this same question not too long ago. You should check it out to see the responses I got.
Hope this helps