I once was working for a UK govt department as a contractor and was between tasks with nothing to do. The management thought it was easier to keep me on than to hire again when the work increased, so I was there with nothing to do. My manager and everyone around me knew I had nothing to do, but I still found it quite stressful doing things like reading Slashdot while at work (it was a long time ago). In the end I started my own project using the work development tools so it looked like I was working as normal, but after a while that was also too much so I found another contract.

I do agree about extra stress in open plan workspaces. I'm the kind of person that can't block out people around me like I notice some other people do, so everyone that walks past my desk I feel the need to glance up at them. In coffee shops I prefer the seating with my back to the wall. Music can be used to block out some of that distraction, but not all of it.