According to electronics-notes.com, the mating cycle rating for the SMA/RP-SMA connector is 500 cycles. However, it is important to note that this rating assumes that the connection and disconnection process happens as smoothly as possible, with the body of the connector not moving at all while the screw connection is tightened sufficiently to not allow the joint to shift while connected. Under nonideal conditions, the actual lifespan of the SMA connectors on FPV goggles is likely to be far shorter.
Under the best of circumstances, the connection would likely last: (assuming one mating cycle per flight session)
- ~19 years @ 2 sessions/month
- ~9.5 years @ 1 session/week
- ~5 years @ 2 sessions/week
- ~2.5 years @ 4 sessions/week
- ~1.4 years @ 7 sessions/week
From this, we can see that the answer likely depends on how often you fly and how long you would like your goggles to last. Although the attenuation (signal strength reduction) of the connector will increase with every mating cycle, if you fly infrequently enough it is possible that you may be ready to upgrade to a new goggle by the time that the SMA connectors begin to become a serious problem.
If you take care to make and break connections cleanly, casual pilots who don't fly very often might not see too much of a problem, but more serious pilots who fly more than once a week definitely shouldn't be removing their antennae on a regular basis.