It probably varies. In my experience working construction this was absolutely not true. There was definitely some down time here and there but the majority of days we worked close to flat out from 7-4:30 (2 15min breaks + 30min lunch). Ditto for summer I spent working on an assembly line where there was even less downtime (although it was not nearly as physically demanding).
Frankly, working a manual labour job would be an eye opening experience to a lot of white collar workers.
Definitely, and probably better for their health, too. Government workers should IMO also have worked a physically demanding job at the same (low) pay rates, to go a bit off-topic here. People shouldn't be allowed to decide for others if they haven't stood in their shoes IMO.
Frankly, working a manual labour job would be an eye opening experience to a lot of white collar workers.