I am basing my belief that this is all (or almost all, at least) hysteria and misidentifications of normal aircraft on the evidence that we have. The evidence that we have shows either things that are too far away/to low quality video to tell what is shown, or shows normal aircraft. None of it shows us mystery drones. I don't give any weight one way or another to claims made by the military or anybody else based on evidence that we are not shown.
If New Jersey was swarming with car-sized drones, there'd still be videos of planes and such, people misidentify stiff -- but SOME of the footage would show the car-sized mystery drones. NONE does.
I don't see any way that this approach on our part is dangerous. IF the military or anybody else has better evidence, they can ignore us and act on what they have. (But notice, as far as has been seen so far they are not actively doing anything -- drones are not being chased by jet fighters or shot down -- if they were, all those people out videoing airplanes would be catching that exciting footage, too.) I do see a real danger that continuing hysteria on the topic is going to result in aircraft being fired on by "defend Ameroica, take matters into your own hands, the government is incompetent so WE'LL do it" types that are egging each other on in social media and comments on the videos. Most likely the overwhelming majority of those folks are all talk, and just spouting off to impress each other. But the danger that some of them are not, and might do something stupid based on a story with no substance behind it, is there.