I've heard that it is bad practice to mount a 2.4 GHz RC receiver antenna next to a 5.8 GHz FPV VTX antenna because the transmitting antenna will negatively impact the receiving antenna's reception. For example, this is said to be true in this Rotor Riot video at 1:40.
Is this true? If so, how close is too close and how can I mount the two antennae to minimize the interference between the antennae?
The distance is dependent on the wavelength of the lowest frequency. You can calculate the wavelength $\lambda$ (m) from $\lambda = c \div f$, where $c$ is the speed of light in a vacuum (ms-1) and $f$ is the frequency (Hz.) Taking the centre frequencies of the ISM bands we get:
2.4 GHz: $$\frac{299 792 458 }{ 2 450 000 000 } ≈ 0.12 = 12 cm$$
5.8 GHz: $$\frac{299 792 458 }{ 5 800 000 000 } ≈ 0.05 = 5 cm$$
(Quick 'sanity check' tip: 5.8 GHz is about twice 2.4 GHz, so we expect the wavelength to be about half; looks good.)
The smallest separation is usually recommended as a quarter of the longest wavelength, which for 2.4 GHz is only 3cm - quite manageable. That said, greater separation helps so put them as far apart as your frame practically allows.
The rule of thumb here is the further away from each other the better. The problem is, that receiver is operating on different band and so it has some input filtering to not receive "out-of-band" signals, but no filtering is perfect and some receivers are better than others. What happens is, that some energy gets through and adds to background noise but receiving signal still must be stronger to be picked up by receiver. So putting your RX close to your TX will effectively decrease your RX sensitivity (how weak signal can still be correctly received). That as said above depends on quality of RX AND sometimes also on quality of VTX as cheap ones may unexpectedly radiate signals well out of its band (and close to RX band). To illustrate the situation, imagine yourself in a role of receiver - trying to hear someone very distant speaking to you while someone else is screaming right into you ear. I know, not precise as we miss input filtering there, but just as an approximation.
To add example, I had experience with one 2.4 GHz RC RX, that was good to fly ~1km with no FPW gear and ground tests were showing promising 1.5km+ range. Adding 5.8 GHz VTX limited range to something around 200 m - quite a difference. But yes, that one was cheap.
In addition to separation of RX and TX by distance, you can use also the frame and or battery as an signal blocking obstacle with good advantage - you can place antennas so that something blocking the signal is right between them shielding RX one from TX one while both are still having clear line of sight with your ground-station. For example one being in the front facing up and the other one being in the back facing down - both sticking well out of frame, but being shielded from each other by the frame and battery inside it.
