Yes, pushing a car require time, but the block slowed its rotation over time.
Let's re-phrase the analogy a little, as it's something people can usually imagine, we've all pushed a car at one time or another.
Consider there's a car (an small one, VW Golf or so)
It's rolling toward you on a flat road at 2mph. You try to stop it. You can probably do it within a few seconds.
Now imagine the same car, but this time it's on a 45 degree slope. It's rolling down towards you at 2mph. How long does it take you to stop it?
You can't stop it. It would run you over, barely slowing at all, and within a few seconds would be doing 30mph, and still going.
Extend that to the vertical. You're on a cliff. The car is suspended an inch above your head. Someone cuts the rope.
In all cases the car has the same mass. But on the flat ground, you are just fighting inertia and friction. When gravity gets involved, then you lose.
So arresting rotation require minimal effort. A car that is slowly spinning on a turntable is no harder to stop than a car slowly moving on flat ground (in fact it's quite a bit easier, if you measure speed along the outer radius). But arresting downward motion under gravity is vastly harder. It requires that you exert a force greater than that of gravity.
Let's put some figures on that. A car weighs 3,000 pounds. Let's say you are an average person, strong enough to deadlift 100 pounds, and can apply the same strength to pushing the car. So that means that you are exerting a force on the car 1/30th that of gravity's pull on the car.
So force required to push or turn something slowly is vastly less than the force required to stop something falling. We are using very rough figures here, but 1/30th for pushing, or maybe 1/50 for turning.
So given that the block stopped turning over a second or so does not mean that very much force was needed. Indeed it indicates that the resistance of the floors below was only a small fraction of the force of the block under gravity. Hence the resulting very little difference in acceleration due to gravity as the block fell. The building could no more slow the block than you could slow the Volkswagon rolling down a steep hill.