I have the same problem, but only when I play in Bridgeport. If you download Barnacle Bay from the Store, you can do all the celebrity stuff there with no lag. It sucks that you have to shell out more money just to get this game to work, but it's pretty fun when it works!
I am having this trouble too. If my computer meets all the system requirements for the game, I should not have to run it at a sub-par performance level. This makes me pretty upset.