It is possible that your online experience is affected by your NAT type.

In general:
NAT 1 can be connected with NAT 1, NAT 2 and NAT 3


NAT 2 can be connected with NAT 1 and NAT 2, but not NAT 3


NAT 3 can be connected with NAT 1, but not NAT 2 or NAT 3

For more information, consult the following article on Xbox One multiplayer solutions:

http://support.xbox.com/en-US/xbox-one/networking/nat-error-solution