Research from Edinburgh University shows you are not alone.
A new report into the social networking site shows that adding family members and work colleagues as friends on the site can cause a lot of anxiety.
Researchers at the university's business school found the more social circles a person is linked with online, the more likely it is that social media will be a source of stress.
Ben Marder, author of the report and early career fellow in marketing, said: "Facebook used to be like a great party for all your friends where you could dance, drink and flirt.
"But now with your mum, dad and boss there the party becomes an anxious event full of potential social landmines.
"In life, you change your persona around different people, from your parents to your friends or your boss, but on Facebook there's only one presentation of yourself and the fear of not meeting everyone's expectations can cause a great deal of stress.
"People are often aware of this when they make a post online, but your friends might not be so concerned about what your boss or your mum think when they tag you in a photograph or post on your wall."
The report – which surveyed more than 300 people – shows that on average Facebook users are friends with seven different social circles.
The most common group is friends known offline (97% added them as friends online), followed by extended family (81%), siblings (80%) friends of friends (69%) and work colleagues (65%).
Around 55% of parents follow their children on Facebook, while more than half of employers claim not to have decided not to hire someone based on their online profile.
Mr Marder claimed worries over this can lead to a change in offline, everyday behaviour.
He said: "If you're at a party and someone gets a camera out, you might not put your arm around that girl, you might hide the bottle of beer you're drinking, in case the photo ends up on Facebook and your girlfriend or mum sees it."
The academic added that people who use the site are not obliged to accept people to view their page or become friends, but argued it is often seen as a social faux pas to refuse requests.
There are a number of privacy settings available to limit certain people's access to your page.However, the research showed only one-third of users implement them.
Mr Marder claimed they often caused further problems for users.
He said: "There are privacy settings which can be used to stop certain people seeing certain things, but if you reduce your mum to only seeing your profile picture and nothing else, then she's not going to be happy.
"And likewise with your partner or boss – if you limit their access, it can breed distrust.
"Social factors can really inhibit you from implementing privacy settings and they can also be difficult to work out."