Whenever I'm in one of my 'should I really be here?' moments, having someone ask a difficult question (ie. difficult for them) usually tears me back out of it.

We all have different influence and interests in our lives. You can know more about X than anyone else in the room and not even know that Y existed. The important thing is to make sure you interact with enough Ys to broaden your horizons.