Being small, lightweight and silent actually makes a good case for their use in many situations. Impact against aircraft becomes less probable and a lot less dangerous; silence undermines the case against disruption and noise pollution.
Drone seem like a very simple matter to me, it seems americans are ignoring a long of history with model aircraft and simply common sense, specially when it comes to privacy concerns. Simply making drones unregulated wouldn't mean the city would be dominated by them overnight. Common sense snooping regulations should apply. Do you frequently see people snooping their neighbors intrusively? Why would they use a drone for that? Common sense should dictate the operator is responsible for it's misuse and any accident.
In Brazil you are essentially free to use drones however you wish; there's no relevant regulation so you'd have to do something really bad to have someone come after you. Yet there's no drone abuse, and plenty of cool applications. I've been to a bike race some time ago and the finishing line was filmed by a drone, it was pretty cool. People are using drones for mapping, agriculture, media, etc. I haven't heard of a single problem with aviation or privacy.
Fairly small RC aircraft have occasionally killed people. So it's not so much safe as it is simply rarity that's allowed RC aircraft to be unregulated.
IMO, if your willing to fly it into yourself at full speed it's safe enough to fly without any regulation. More dangerous than that and basic certification seems like a reasonable precaution.
If you hit someone with a frying pan it's considered assault with a deadly weapon so it's not considered 'safe' just stationary without direct attention. You basically have to have to attack someone for frying pan's or Pencils to hurt someone, where all it takes is intention or any number of mechanical / software failures to cause harm with a drone which is a vary different situation. It's the difference between a car and a gun and you need a licence to drive a car on the public roads.