Holding Others' Boundaries

I've thought about this idea for a while now, and still have not reached a solid conclusion on it. 

What is my responsibility for other people's boundaries?

I can ultimately only be responsible for myself and can ultimately only control myself and my behavior.

And like J and I have talked about before, doesn't it in some way take away another person's autonomy if I say, "Oh sorry I can't do that with you because I know your partner would be upset" ? Isn't that person responsible for their choices and their own relationship agreements and boundaries?

J and I also have discussed the fact that we will negotiate our relationship boundaries and agreements, and that we don't expect another/secondary partner to hold them for us; we are responsible to our primary relationship and any boundaries in place. J wouldn't want, for example, for me to negotiate a boundary that he has with one of his secondary partners. He and I would negotiate whatever boundary, and then he would be responsible for communicating that boundary to his secondary partner. If the boundary was ever broken, I don't think I would be upset with the secondary partner nearly as much as I would be with J. I think I would feel a little hurt by the secondary partner's involvement in breaking a boundary that J and I agreed to, but ultimately, I would feel upset with J and need to rebuild trust and communication with him.

Then again, I don't want to be impacted negatively in the long run. It's possible for instance, for my partner's partner to veto me as a secondary partner for being part of an agreement violations. This is an incentive to knowing what a partner's relationship boundaries are.

Also, sometimes another partner's boundary violation speaks to their own ethics and ability to be an individual of honesty and integrity. Which then impacts my own interest in being romantically involved with them.

It also feels more complicated to me when it comes to emotions versus physical acts. If someone says, making out and oral sex is okay but vaginal intercourse is not, that is pretty cut and dry, and pretty easy to define those boundaries clearly. If someone says, I don't want you to love another person or for another person to express love to you, then what do you do?

So I have yet to reach any kind of firm rule on this. It feels like a grey area to me, and one that is sussed out delicately depending on the situation and context.