Why do most VTXs use 5.8ghz and most transmitters use 2.4ghz?
As is discussed in this Oscar Liang article, FPV hardware exists for all different kinds of frequency bands, stretching from 900 MHz to 5.8 GHz, with 5.8 GHz being the most popular today. People in our hobby have tried nearly everything under the sun. :)
Probably the most well-known tradeoff in RF is that between frequency and penetrating power. In general, higher frequency transmissions will not be able to travel as far and penetrate as many obstacles as lower frequencies.
This would make it sound like lower frequencies would always be better, but this is not the case. Especially in the case of FPV video, available bandwidth and interference issues become more of a concern than pure range.
With increased frequency comes higher bandwidth and reduced latency, which is to say that the 5.8 GHz band accessible to us can support more data throughput at a lower latency than the 2.4 GHz band. This is critical because video transmission requires several orders of magnitude more data to be sent than RC controls and telemetry do because it sends entire pictures instead of a few numbers per transmission.
The Troubles With 2.4 GHz FPV Radios
The need for more bandwidth in the 5.8 GHz range is further compounded by the fact that our analog FPV systems usually aren't frequency-hopping. In the old days of the RC hobby when airplanes were the only option, most RC transmitter/receiver systems had a fixed operating frequency. I have a couple of these transmitters, which actually have a slot where you need to insert a special crystal oscillator to select which frequency the transmitter uses!
This became a problem once people started coming together to fly and the RC hobby became more widespread. All of a sudden, people would begin turning on their transmitters at the flying field and their transmitter would talk over other people's transmitters using the same frequency, blasting them out of the sky and interrupting the controls to their plane. (does this sound familiar? XD) This issue was worsened by the fact that the 2.4 GHz band we are allowed to use doesn't allow for very many non-overlapping channels and because the 2.4 GHz band is used by many other different kinds of devices and appliances which can also step on RC transmitters' signals.
The solution to this problem, which has been implemented in the vast majority of all 2.4 GHz RC systems, is to implement frequency-hopping. This feature allows the RC system to dynamically switch between frequencies on the fly as needed, without any input required from the pilot.
These same issues caused by transmitters stomping on each others' signals are also a massive issue in the 5.8 GHz band, but to a far lesser extent due to the vast array of different non-overlapping channels available for use.
Well, the first thing I must point out is that these are far from the only frequencies used for that (and yes, I'm aware that you said "most" in the question). The reason I'm doing that is that answering the question "why do some people not use those frequencies?" can give us an insight into why the others do.
Let's begin by listing the "at least somewhat popular" frequencies. For video that would be 1.2, 2.4, and 5.8 GHz, and for control/telemetry links 72 MHz (analog), 433 MHz, 900 MHz, and 2.4 GHz.
The first obvious thing about those bands is that most of these are ISM bands. The ISM bands were originally intended as "trash bands" designated for non-telecommunications purposes, where all the industrial equipment such as radio-frequency heaters (think microwave ovens) could emit their RF radiation without interfering with any telecom bands. The most important thing about them for us, though, is that they are globally license-free and mostly unregulated, which basically means you can do about anything on them without requiring any kind of certification for your equipment as long as it only transmits within the band (and sometimes also doesn't exceed a certain maximum power level).
The "unregulated" bit is key, since any bureaucratic process regarding radio frequency use is traditionally very long and rather expensive, and it has to be done in every country for every single model of your product. So the ISM bands had been quickly put to use by mass-produced cheap consumer devices that didn't care too much if they got interfered with: Cordless phones, garage door openers, baby monitors, Bluetooth, NFC, WiFi... RC systems continue this trend.
Second, as you can see, the most popular bands, both for video and control/telemetry, are the highest ones in their corresponding ranges. Why?
To answer that question, as I discussed above, we're going to start with figuring out why they aren't the only ones.
Why do people go lower?
The first and very obvious boon is range. While 2.4 GHz RC systems offer less than one kilometer of usable range, 900 MHz systems like Crossfire easily go several km on low power without even trying, and can reach tens of kilometers if configured properly. Same thing with video: lower-frequency systems are mainly used by long-range pilots seeking those few extra miles of coverage, and readily provide that additional range. Lower frequencies mean longer waves, and longer waves are easier to transmit further.
Also, they permit more transmitter power: both in terms of regulations (maximum allowed power ratings tend to get lower as frequencies get higher) and in terms of technology as well, as lower-frequency radio amplifiers are simpler and cheaper to build than higher-frequency ones, as is lower-frequency radio tech in general.
Simplicity is not to be underestimated. It's one of the reasons why the old analog transmitters used 72 MHz: 72 MHz radio circuits are way simpler to manufacture with through-hole mounting, discrete components, etc etc, while high-frequency designs tend to require SMD components, their performance is heavily influenced by things like the shape of traces on the circuit board and the material of the PCB itself, and the process of designing such devices can be indistinguishable from black magic. 72 MHz systems didn't get much range, though, because... analog. Also, antennae. Also, no diversity. Also, lots of other problems.
Speaking of problems,
The drawbacks of lower frequencies.
There are several.
This is the first and very obvious one. Lower frequencies mean longer waves, and longer waves mean guess what? Longer antennae! Remember the huge extendable antennae of the old RC transmitters? That's 72 MHz for you! A 2.4G video antenna is going to be about 2.5x as large as a 5.8G one, and a 1.2G antenna almost 5x as large! Where a 2.4 GHz receiver antenna can easily stick out from almost any point on a 3" quadcopter and not get in the way, a 900 MHz antenna is longer than the quad itself, and a good 433 MHz antenna would be closer to the size of a 6" quadcopter!
That's a sneaky, but very important factor, especially for video. Any signal that carries data does not use just one frequency. Instead, it occupies a whole range of frequencies adjacent to the main one; for example, a basic AM voice signal transmitted at 10 MHz would actually take up the range of around 9.99 to 10.01 MHz, 10 kHz higher and lower than the main (carrier) frequency. The size of that range is the bandwidth of this signal, 20 kHz. This means that signals of this type must be spaced at least that amount apart to not interfere with each other. Voice is a relatively low-bandwidth type of data. As you push more information per second through a link, the bandwidth will increase, which means you have to space the links further out, which in turn means you can squeeze less of them into a certain frequency range.
Now, the ISM bands (as well as any other radio bands) also have a certain width, the difference between the highest and lowest available frequencies, which you can see in the table above. And as you can see, as the frequency gets higher, the separation also gets bigger. The 900 Mhz band has merely a quarter of the bandwidth available in the 2.4G band, which in turn is 1.5 times smaller than that of the 5.8G band. While control links are affected (it's one of the reasons why you don't see a lot of 433M control systems around), they don't use nearly as much bandwidth as video. Analog TV (and video in general) has a notoriously high bandwidth: 6 _Mega_hertz. You can fit several hundred voice signals in the space occupied by one TV signal! Because of that, analog TV had a huge chunk of the radio spectrum allocated to it: 54 to 890 MHz, for a ratio of highest to lowest frequency of more than 16x. Compare that to the ISM bands, where the top of the band is rarely more than a couple percent away from its bottom. You couldn't fit even one video channel into the whole 433 MHz band! So we use the higher bands out of necessity; the 2.4G band is really the lowest one that can fit a reasonable amount of video signals in it, and the 5.8G band can fit even more.
Why isn't 2.4G the video standard? Well, it would interfere with our 2.4G control links (and they claimed the band before FPV became popular). Also, more bandwidth is better. Also, you love small video antennas, don't you? Besides, using 5.8G is more responsible in general. There are way more different applications in the 2.4G band that you could stomp on with your huge video signal; 5.8 is considerably less popular, it's hard to find anything besides FPV or newer WiFi using it.
Why not 24 GHz or higher? Welll... It's 5x more frequency, and things are getting well into "black magic" territory at above 1GHz already; at 24 GHz you could go crazy designing the gear and possibly just attaching the antenna to your quad. And it would be several times as expensive.
Finally, we come to the last consideration, about 2.4 gigahertz specifically, and why I think it was chosen as The digital RC band. And that, besides the huge bandwidth and conveniently small antennae, is... cheap WiFi chips. Most 2.4G RC systems use one of four 2.4 GHz digital radio microchips: NRF24L01, A7105, CC2500, or CYRF6936. Each of those is a fully-featured 2.4 GHz transceiver in one chip, which you could put on a board, plug an antenna into one end, an Arduino into the other, and voila, you've got an RC controller! Or a receiver. Or WiFi/Bluetooth/whatever dongle. That's literally what the multiprotocol module is, by the way: all 4 of those chips, one microcontroller, and some clever schematics so that they could share an antenna, in one enclosure. Anyway, chips like these make creating digital 2.4GHz radio equipment extremely easy compared to the usual black magic, and thanks to WiFi and other 2.4 GHz tech the chips themselves are really inexpensive. And THAT, I think, is the deciding factor. Because who doesn't love cheap transmitters and receivers at 10$ a pop?
As I understand it, it is a case of available bandwidth, range, and also legislation.
5.8GHz is able to transfer much more information on its carrier wave due to its higher frequency compared to on 2.4GHz.
The reason, therefore, that we use 2.4GHz for a control link and 5.8GHz for FPV is that FPV requires more information; we have to create whole images, dozens of times per second on FPV, but an RC link can be thought of as 16 streams of numbers, which as you can imagine takes up less bandwidth.
The next reason is the range. Lower frequencies have a longer wavelength and therefore a greater range. Take 5G for example, that can go up to 72GHz but requires regular repeaters to get good coverage.
There are also different radio band allocations that play into it. Taking the UK’s OFCOM as an example, they have allocated specific radio bands that they allow for use by hobbyists, so it is much easier to use these bands than get special permission to use other bands (though it is worth noting that in some places you may need a HAM license to operate certain hobbyist radio equipment).