If 10 waves go by in a second it matters if they are 1 foot waves or 10 foot waves.
The size of the waves is literally irrelevant. You just need some feature to track, it could be part of a small wave or big wave. Speed is not being determined by how many waves pass per second but how fast that particular feature travels relative to the FOV.
Edit: Perhaps you could argue the size of the wave determines in part how fast they travel or something. But I think the motions of the waves here are rather random and slow compared to the panning speed. And still irrelevant to my main point that other solutions should be tried before the FOV change.