It is possible that your online experience is affected by your NAT type.
NAT 1 can be connected with NAT 1, NAT 2 and NAT 3
NAT 2 can be connected with NAT 1 and NAT 2, but not NAT 3
NAT 3 can be connected with NAT 1, but not NAT 2 or NAT 3
For more information, consult the following article on Xbox One multiplayer solutions: