From voyeur to voyeur - I completely understand your trepidation. It is hard for some to understand this fetish and many times I find myself in a conundrum or feeling a bit scared to watch (JUST WATCH, ONLY WATCH) because it can come off as creepy or inexplicable to others. Yes, I like group sex, yes I enjoy foursomes (threesomes not so much - someone ALWAYS gets left out, better to stick with even numbers), but more than the actual act itself, I like to sit back and enjoy two or more people getting it on. The good news is, there are a world of exhibitionists out there and they love to be watched and actually get off on it just as much as you (we) get off on watching them watching us watching them get off.
Sex parties are hard. There are archetypes that crowd the place and most people like to do it in darker areas which is the Achilles Heel for voyeurs (see also: dark rooms, saunas, steam rooms, glory holes et al). Some people are shy (read: married) and don't know exactly how things work. Some people may take a visual cue as a come-on, and then you're stuck in that awkward place of trying to be polite about telling them you don't want to have sex with them at a sex party.
Sometimes I'll play the rabbit in that I'll get something started and duck out to the side and encourage them to keep going. It's a chore but worth it.
If you can afford it - ask a professional to help. Perhaps they can bring in an assistant for a nominal fee.
There are other methods as well - inquire within.