I've heard a few times now that if you power up your drone you have to make sure that you have an antenna on the VTX. Is that true? If so, what will happen if I power up my drone without an antenna on the VTX?
Short answer: Yes, this is very much true. The VTX sends out radio-frequency (RF) energy through the antenna connector and pigtail, and if nothing (or the wrong thing) is connected at the other end, all the energy gets reflected back and could damage your VTX (or just heat it up).
At high frequencies a thing called "impedance matching" becomes very important.
Basically, when a radio wave travels through a wire, the wire presents a certain impedance (a more "generalized" form of resistance) to the signal. Whenever the travelling wave encounters a change in this impedance as it goes through the cable, connectors and other hardware, i.e. it was 50 ohms before a certain connector and becomes 75 ohms after, part of the wave gets reflected back at that point, and the bigger the mismatch, the bigger the percentage that gets reflected.
Any properly designed antenna will have the same impedance (in the frequency range that it was designed for, i.e. 5.8 GHz for a video antenna) as the coaxial cable before it, and dissipate the received radio-frequency energy by radiating it out in the form of radio waves, thus nothing gets reflected towards the VTX. However, If there is nothing at the end of your pigtail, the mismatch is as big as it can possibly be, since nothing connected means infinite resistance. Thus all of the power that the VTX emits gets reflected back towards it and has nowhere to go other than to heat it up. If the VTX is poorly designed, the reflected power could even burn some of the components in it!
Obviously, this gets more important with the more powerful VTXes. If you've got a 25 mW or even a 200 mW VTX, it's probably not going to damage itself if left without an antenna for a while (although you better not risk it anyway). An 800+ mW VTX could do much more damage; in the best case it will just get very very hot.
If you really need to run without an antenna for some reason (which is rare, but possible if you're testing something that needs the VTX to be powered up, but don't want to actually transmit anything), there is an option to terminate the transmission line with a "dummy load" (which is just an powerful resistor that's impedance-matched to the coax and dissipates all the RF power that it gets) instead of an antenna. Note that shorting the connector is also going to reflect all the power back to the VTX; zero resistance is no better than infinite resistance.
Also, beware of antennae that are designed for other frequency ranges. i.e. a 2.4 GHz antenna from your router or radio control might look like it's compatible and will probably even have the same (SMA or RP-SMA) connector, so it'll fit mechanically just fine, but at 5.8 GHz it will have a drastically different impedance and thus transmit only a small portion of the power it receives, with the rest being – you guessed it! – reflected back and heating up your VTX.
Your VTX is a high-power RF transmitter. When operating with an antenna, the RF power of the transmitter is radiated out through the antenna. When the antenna is missing, the RF power is reflected from an unterminated antenna port back into the transmitter. This reflected RF energy will be dissipated as heat (best case scenario), or damage/destroy your transmitter and then dissipate as heat.