Can circularly and linearly polarized antennae be used together for FPV?

I have a few spare antennae of mixed polarizations I'd like to be able to reuse for my FPV gear.

Is there anything I need to know about incompatible configurations of mismatched polarizations between my transmitting and receiving antennae?

The compatibility of mixed polarity antennae depends greatly. For a little background in RF polarity, this diagram shows the difference between the four main types of antenna polarization available: (mixed/elliptical polarization do exist)

(cit.)

In general, it is best if the TX and RX antennae match in polarization, but some other combinations are possible while minimizing signal strength loss.

Linear → Linear

Linearly polarized antennae are interchangeable, as long as they are oriented in the same direction. Signal reception between linearly polarized antennae oriented with an angle between them of θ will fall off with a polarization loss factor (PLF) of PLF = cos^2 (θ) and reception will be zero when the antennae are perpendicular to one another.

Circular → Circular

As long as the circular polarization is identical (RHCP or LHCP) between the two antennae, no signal loss will be observed due to the polarization of the antennae. Signal strength isn't dependent on the angle between the orientations of the circularly polarized antennae. However, if you aren't using omnidirectional antennae (e.g. mushroom)¹ and instead have a directional antenna (e.g. patch, helical), you must point the directional antenna in the direction of the other antenna to have the most gain. (see "What is antenna gain?"). Within the beam width of the directional ciruclarly polarized antenna, orientation doesn't matter.

As pointed out by @FlashCactus, only circular → circular polarization matches benefit from the multipath interference reduction effect. This is because circularly polarized waves switch polarization when they bounce off of an obstruction (i.e. RHCP → LHCP and vice versa), so duplicate signals taking an odd number of bounces to reach the receiver will be ignored by the antenna. (they will be recieved as the opposite polarization with which they were transmitted)

For more details on how circular polarization works, I (and @FlashCactus) recommend having a look at this question: What is circular polarization in antennas?

¹: Omnidirectional antennae are also not truly omnidirectional; they have a region of low gain along their axis, and will transmit/receive poorly in that direction. It is best to face them sideways towards the other antenna for best reception. The same is true for linear antennae; they receive&transmit best from their sides and worst along the antenna.

Circular → Linear

Because a circularly polarized wave can also be represented by two 90° out-of-phase perpendicular linearly polarized waves, the linear antenna will receive with a signal strength reduction (PLF) of half (or -3dB). Signal strength isn't dependent on the angle between the orientations of the linearly polarized and circularly polarized antenna.

This configuration can be useful if you want to spectate someone who has the opposite transmitting antenna handedness from you (LHCP while you use RHCP, for example), or to scan the radio spectrum that might contain signals of different polarization (RHCP, LHCP or linear) without switching the antenna: A linear antenna will receive both RHCP and LHCP signals equally well.

Linear → Circular

Likewise from before, because the circularly polarized antenna is built to receive both perpendicular components of the circularly polarized wave, it will receive a linearly polarized wave with a signal strength reduction (PLF) of half (-3dB). Signal strength isn't dependent on the angle between the orientations of the linearly polarized and circularly polarized antenna. As with the opposite configuration, RHCP and LHCP antennae will receive linearly polarized signals equally well.

This configuration can be commonly seen in the case of micro-quads like Tinywhoops which only come with linearly polarized transmitter antennae. Here, the orientation-agnostic benefits of receiving a linearly polarized signal with a circularly polarized antenna (and the convenience of not replacing the antennae on your goggles) are often worth the signal strength reduction.

• By the principle of reciprocity, linear -> circular and circular -> linear are the same case. – Phil Frost Apr 30 at 14:54
• Thank you, @PhilFrost. That's a good point. – ifconfig May 1 at 3:04

To add to ifconfig's excellent summation of how the signal strength would be affected by mismatched polarizations, I must point out that in order to benefit from the reduction of multipath interference offered by circular polarized antennae, both the transmitting and receiving antenna need to be circularly polarized.