Reaching the snapping point?

I'm a nice guy that doesn't go out of my way to bother people. I'm not shy though, sometimes i'm a bit loud and boisterous. I'll be the first person to tell someone what I think, or help someone out when they need it. But sometimes, I wind up in crappy situations, where someone wants to pick a fight with me. I always ignore it or back down, mostly out of fear of going to jail. I can count probably 3 times within the year where I could've drug out with someone but didn't. I know that it's always better to avoid fights unless someone tries to hurt you, but it's kinda ingrained in guys to respond when your called out. I'm a scary looking guy, and some people think i'm crazy. But I don't look for trouble, and fighting always makes me nervous, as it should because it's messy and dangerous. Just to relieve some of the guilt and feeling of cowardice though, I feel like slugging the next person who calls me out, consequences be dammed. What do you guys think?