It has happened. Im looking for a solution not a dismissal
No you are definitely right. Your worry is a valid one which I can relate too. A little background without getting too much into specifics, I'm pretty sure your average board member is bigger than me. So through my life I needed to adjust myself into handling bigger people. Now first off I have always been a little blind to how big others are, it simply didn't really register with me. But when I got more conscious I noticed that people try to pick on those who are smaller than them. However I always had one big advantage: I have been a good fighter since my very youth, so I knew how to handle situations. Later on I have become a trained fighter and at that point your average meathead is a non-threat. It doesn't get to fighting usually since the psychological intimidation one can portray is enough. Because it is backed with the projection that you are very well prepared! I am not saying you have to fight, but I am saying you need the edge because people disrespect those who they think it an easy target or someone who they can use as a crutch or have to sabotage.
There are multiple dimensions to your question
First off there is this old saying "if you want peace, prepare for war". In other words if you do not want to get into conflict with others you have to become strong as a deterrent. Ofcourse there are multiple dimensions of being strong, like having a crew or lots of influence, but I think the most basic and important strength is what you project with your body. Your body language, your confidence, your intensity even. People pick on to this when they pick their targets. Get into MMA, Muay Thai or BJJ. Understand the nuances and dynamics of fighting and how feeble your average meathead actually is. Lifting weights can make you a bit more intimidating maybe, but it wont really help in a fight. In conjunction with actual combat training yes, otherwise it is just posturing. All this is hard work, but it's the best lifelong solution. Then you become the one who is feared.
Second, I suspect because I have been in conflicts maybe a hundreds of times, probably also because I share your problem, I have developed unconscious nonverbals which project that I am not to be trifled with. A lot of people tend to pick up on this and steer clear with making trouble. This is the result of experience and many psychological victories. Otherwise I try to be a swell guy and make friends and alliances.
How this all has translated into my life:
There will always be an uncalibrated meathead who will try his hand, my experience is that these guys usually are posturing and I escalate the vibe very quickly so they have to quickly decide if they can sustain a conflict yes or no. Almost every time they bow out, since they were looking for easy prey! However there are some who will keep persistent (depiste having a greater strategy) and at one point you have to double down on them.
*I recall this superbig and loud bodybuilder guy who interjected when I was flirting with a girl. I didn't break circle and ignored him. He made the mistake of trying to amog me with touching me and telling me to "relax" at which point I quickly spun around squeezed his shoulder and asked him with a sharp tone if he was trying to lecture me. My intonation was pretty clear, I was prepared to rip him apart. He simply said nothing and walked away with his tail between his legs. That must been one sight for the girl, the super big guy running away scared from the smaller man.
*Two weeks ago I was with this really tall girl who had superlegs and had short pants on so it really showed. Out of the nothing a big tall guy appeared and started scolding me "shouldn't you be working out". I turned around and told him that I feel lazy today and he should mind his own business. But I showed zero concern. He too walked away really stiff and weirdly, making the girl next to me comment "okay" because of how awkward he was. He probably had been watching me with her for 10 or 20 minutes and wondered how a smaller guy can be with a tall babe while he has zero succes.
Recap: If you want peace, prepare for war. But there are no magic bullets. I do NOT believe in just being verbally strong, that is just an easy copout. A man should be able to defend himself.
