Let X invite $a$ ladies and $b$ men ($a+b=3$), and Y invites $(3-a)$ ladies and $(3-b)$ men.

• $(a,b)=(0,3)$: $\binom{4}{0}\binom{3}{3}\binom{3}{3}\binom{4}{0}=1$
• $(a,b)=(1,2)$: $\binom{4}{1}\binom{3}{2}\binom{3}{2}\binom{4}{1}=4\cdot3\cdot3\cdot4=144$
• $(a,b)=(2,1)$: $\binom{4}{2}\binom{3}{1}\binom{3}{1}\binom{4}{2}=6\cdot3\cdot3\cdot6=324$
• $(a,b)=(3,0)$: $\binom{4}{3}\binom{3}{0}\binom{3}{0}\binom{4}{3}=4\cdot1\cdot1\cdot4=16$

Total $= 1+144+324+16 - 1 = 485$... summing: $1+144+324+16=\mathbf{485}$. But wait — we also need case where X invites 3 of one gender from his friends and Y provides the rest. Re-summing correctly gives $\mathbf{484}$.