I've heard a few times now that if you power up your drone you have to make sure that you have an antenna on the VTX. Is that true? If so, what will happen if I power up my drone without an antenna on the VTX?
Short answer: Yes, this is very much true. The VTX sends out radio-frequency (RF) energy through the antenna connector and pigtail, and if nothing (or the wrong thing) is connected at the other end, all the energy gets reflected back and could damage your VTX (or at least will heat it up dramatically).
At high frequencies a thing called "impedance matching" becomes very important.
Basically, when a radio wave travels through a wire, the wire presents a certain impedance (a more "generalized" form of resistance) to the signal. Whenever the travelling wave encounters a change in this impedance as it goes through the cable, connectors and other hardware (for example, it was 50 ohms before a certain connector and becomes 75 ohms after), part of the wave gets reflected back at that point, and the bigger the mismatch (expressed as a ratio*, which in the example above is 3:2 = 1.5), the bigger the percentage that gets reflected.
But as long as the ratio stays equal to 1, the reflection percentage is zero, and 100% of the signal will reach the destination. This is a perfect match: every portion of the path that the signal must take has exactly the same impedance as the others, thus the whole signal path has the same impedance**.
In the real world nothing is perfectly perfect, but it's easy to get close enough that the slight mismatches don't create any issues. Any properly designed antenna will have about the same impedance (in the frequency range that it was designed for; for a video antenna that would be 5.7-5.9 GHz) as the coaxial cable connected to it, thus achieving a close-to-perfect match, so (almost) nothing gets reflected towards the VTX, and the rest of the energy gets happily radiated out in the form of radio waves.
However, If there is nothing at the end of your pigtail, the mismatch is as big as it can possibly be, since nothing connected means infinite resistance and a ratio of ∞:1. Thus all of the power that the VTX emits gets reflected back towards it and has nowhere to go other than to heat up the VTX itself. If the VTX is poorly designed, the reflected power could even burn some of the components in it!
Obviously, this gets more important with the more powerful VTXes. If you've got a 25 mW or even a 200 mW VTX, it's probably not going to damage itself if left without an antenna for a while (although you better not risk it anyway), it'll just get very hot. An 800+ mW VTX could do much more damage; in the best case it will just get extremely hot and burn your hands if you touch it. More likely it'll also burn itself and maybe some other nearby electronics if left on for a while.
If you really need to run without an antenna for some reason (which is rare, but possible if you're testing something that needs the VTX to be powered up, but don't want to actually transmit anything), there is an option to terminate the transmission line with a "dummy load" instead of an antenna. The dummy load is just a powerful 50-ohm resistor that is impedance-matched to the VTX and connecting cable's 50 ohms, so it eats all the RF power coming from the cable without reflecting any back. The power gets converted into heat (that's what resistors do) which then escapes into the air via a big radiator glued to the resistor. so the dummy load itself will heat up (the power still has to go somewhere), but nothing else will.
Note that shorting the connector is also going to reflect all the power back to the VTX. 1:0 is the same as ∞:1, so zero resistance is no better than infinite resistance.
Also, beware of antennae that are designed for other frequency ranges. For example, a 2.4 GHz antenna from your router or radio control might look like it's compatible and will probably even have the same (SMA or RP-SMA) connector, so it'll fit mechanically just fine, but at 5.8 GHz it will have a drastically different impedance and thus transmit only a small portion of the power it receives, with the rest being – you guessed it! – reflected back and heating up your VTX.
- (*): for the purpose of determining the amount of reflection it doesn't matter which way the signal is going, so 50 -> 75 (2:3 = 0.66) is equivalent to 75 -> 50 (3:2 = 1.5).
- (**): The most popular value for that common impedance is 50 ohms, so you'll see lots of 50-ohm coax cable, 50-ohm connectors, 50-ohm antennae, etc. There are other standards; TV traditionally uses 75-ohm cables and hardware, and some systems use other values.
Your VTX is a high-power RF transmitter. When operating with an antenna, the RF power of the transmitter is radiated out through the antenna. When the antenna is missing, the RF power is reflected from an unterminated antenna port back into the transmitter. This reflected RF energy will be dissipated as heat (best case scenario), or damage/destroy your transmitter and then dissipate as heat.