I've answered my fair share of "what's your least favourite thing about unschooling?" questions over the years (most recently in an interview I did for a new homeschooling magazine to be published through Apple Newsstand), and my answer has always been other people. People who aren't unschoolers, don't understand unschooling, and can make your life difficult because of it.
The anonymous critics mean very little, as do the random strangers you somehow end up discussing the subject with, no matter how much you may try and avoid it.
Yet what does mean something, and when it can start to feel kind of scary, is when someone has something you need, and you fear your educational background could cause them to make getting what you need more difficult.
I'm talking about the potential employer asking you questions about your education, even though you have all the experience needed for the job you're applying for. Or when something goes wrong in a big way, and you call the police, knowing that even though you're the victims in whatever happened, that you'll get grilled on the fact your children don't go to school. Or you're looking for a therapist, and fear as soon as they find out you didn't go to school, they'll start trying to blame whatever struggles you're dealing with on your lack of "proper" education and strange upbringing.
It can cause a lot of worry, wondering if someone important, someone with the power to provide you with much-needed support (or money, or information) will start asking those questions, and then start treating you through the lens of their own pre-conceived biases and ignorance about unschooling (and home learning in a broader sense).
Sure, there are other difficulties possibly related to unschooling that can make things unpleasant sometimes. There are times I feel insecure about what I know and have learned, but only in the way I think everyone, regardless of education, does: if you feel someone knows more than you, seems "smarter," it can be easy to feel some insecurity. But generally I feel really good about my skills, knowledge, and accomplishments. They're not the same as other peoples, but that's the whole point. Each person is unique, with their own strengths and skills. Most of the time, I know mine are valuable.
So it's just other people I worry about. People who might not see my strengths or value my experience, because they got side-tracked as soon as they heard that whole "didn't go to school" thing and are no longer focusing on anything else. Sometimes that fear is unfounded, but sometimes, well sometimes it isn't.
Regardless, us unschoolers usually manage fine despite that. But that doesn't mean those types of attitudes don't sometimes make things harder, make you hesitate before seeking help, and create a little kernel of fear when a person who can provide you with important services does that double take we all know well: "wait, you were homeschooled?"
Which is why, until our culture stops thinking schooling is the be all and end all when it comes to "getting an education" and becoming a "productive member of society," the reactions of others will remain my least favourite thing about unschooling.