what kind of effect has on my gear and flights to have set a different number of motor poles that I actually have installed. for example, my motors have 14 poles but I set 12 on BetaFlight or vice-versa.
This is a setting that is needed for RPM sensing and RPM filtering. An ESC can not sense the RPM of your motors directly, but can sense how often the magnets in the motor bell pass by the coils in its stator (this is called eRPM), and will report that number to the flight controller if you've got bidirectional DSHOT enabled. If your motor bell has x magnets (or poles) in it, then they will pass a coil x times per full revolution of the bell. Thus, eRPM = x * RPM.
In other words, to calculate the RPM of a motor, the flight controller will read the eRPM value reported by the corresponding ESC, and divide it by the number of poles. This means that if you set the wrong number of poles, your flight controller will read the wrong RPM from the motors, and the RPM filter will not work properly.
If you don't use RPM filtering, you probably should not care. If you do, it won't work as intended (and may cause your copter to fly worse) unless you put in the correct number.
The rpm filters in betaflightwill be off the correct frequency by about 14%, and the rpm reported in the motors tab will be incorrect also.
When the rpm notch filter centres are not exactly at the correct frequency, they won’t work as well, leading to increased noise in the motor signal, which may cause hot motors and limited thrust at high throttle.
The basic answer is that the RPM notches, and any dynamic idle features engaged will work poorly [typically use an value of a 14 pole motor instead of a 12 pole motor actually present on the craft]. The bracketed statement is an assumption, but the most common error, and one I've made myself... multiple times... and figured out through blackbox logging that I forgot that.
Actual results is that the RPM notches run at too low a frequency, and the RPM Notch Q factor means that it's only attenuating some of the noise, and typically the upper spectral content half of the noise coming off the fastest spinning motor is going to get through notch filters. Depending on lowpass filter setup, and what range/q factor/number of notches the dynamic filter is on, really this will mostly negate the improvements of using RPM notch filters, add phase latency (because a lot of the time the RPM notches will just sit at their Min RPM, and the rest of the time be at lower values), and allow higher frequency noise into the PID controller, which with usable D gains results in extra motor heat. Worst case, the extra noise that has not been filtered out from the notches, and isn't adequately lowpass filtered through the gyro dLPF, any fixed gyro LPF, dTerm dLPF, an any dTerm Fixed LPFs will proceed to be amplified by the PID controller resulting in hot motors, or at worst oscillations that lead to a 'to the moon' condition.
This is worse for dynamic idle applications, because a correctly calculated minRPM (that has not been hardware validated by spinning motors in the motors tab) will result in desyncs due to low RPM... but these won't emerge until pushing the craft hard in negative AoA moves because the DSHOT Idle % will usually override this, but once pushed, the FC will try to allow that motor to spin down to a very low value which will result in a desync.
Practically, the default filters are still set conservatively enough that most [small 12 pole motor craft] can actually fly adequately, but with performance in-line with conservatively set Betaflight 3.2 performance (dynamic notch will try and manage the rest of the motor noise band, but will let noise through, and lean on the lowpass filters to mitigate the higher frequency noise, but at a significant phase latency costs that make propwash response worse).
TL;DR - Flyaways are definitely possible, especially with increased P or D gains, or aggressive filtering applied while this remains incorrect. Most cases, reduced flight performance and limited utility of the RPM notch filter is present.
As always, if you use dynamic idle use the motors tab to check the min RPM, and add a couple hundred RPM to that value (+2 in the config variable) for improved margin for best results. On new builds or after reflash (where this is most likely), use stock filters for maiden flights and check motor temperatures quickly, as those will indicate that something like this is wrong.