It's important to note that the site is a Shadow action, not a passive effect. If your opponent doesn't announce that he's taking the action and tells you he's done with his Shadow phase, then he can't reduce the move limit.
Additionally, there's nothing stopping the Shadow player from reducing the move limit multiple times. Even if you'd made the move limit +10, the Shadow Player could make it -500, because the site doesn't put a restriction on how low you can drop the Move Limit (this is because, of course, in order for the Shadow player to have used the site, the FP player must have already moved once, so they're still guaranteed to get at least one move out of their turn).