There are several things going on here. They can be broken down into three components.
- Torque lever arm
- Angular Velocity and Acceleration
- Moment of Inertia
Torque Lever Arm
Essentially the distance on the axis is directly proportional to the torque lever arm. This means that in order to counteract outside influences on the system, the motor has to exert less force to stabilize the system.

Angular Velocity
Also as a result of the increased distance from the center of rotation, a given velocity of the motor produces a lower rate of rotation due to the increased angular distance needed to travel. The motor has a fixed linear velocity for a given RPMs that is defined by the torque production of the motor, the velocity constant, and the pitch of the prop. Effectively this means that the longer the lever arm, the slower the rotation rate for a given linear velocity of the motor.

Keep in mind angular velocity and linear velocity are directly proportional since motor velocity is a constant defined by the motor and prop combination, not impacted by radius. As a result, angular velocity is inversely proportional to radius (w = v / r).
This is not necessarily the case for acceleration however. One might be tempted to think that because:
Fr = Ia (Force x radius = moment of inertia x angular acceleration)
that the angular acceleration might also be equal because the r and I components of this equation cancel each other out by definition. The moment of inertia increase cancels out the torque increase due to distance in this equation as written. This results in net neutral equation given that r is changing equally on both sides of the equation.
F * r = m*r^2*a
m * dv/dt * r = m * r^2 * dw/dt
m * dv/dt * r = m * r * r * (dv/r)/dt
m * dv/dt * r = m * r * dv/dt
This is an oversimplification however, since we're not dealing with a point moment system defined by this equation. The motors make up only a small portion of the total distributed mass and resulting total moment of inertia. Since the distributed distances on the right do not equal the point force distance on the left side of the equation, as the radius at the point of force increases the rate at which the motors achieve the defined angular velocity speeds up. Essentially their effect against the system-wide moment of inertia is greater. This results in an interesting combination of a more stable system due to both lower sustained velocity at equivalent stick deflection and faster initial rates of change. Because of the very fast rate of change the motors are already capable of producing, the constant angular velocity matters more to control rates for the pilot, while the angular acceleration matters more to the PID controller and resulting stability, so this combination is particularly effective. Going to a longer lever arm on an axis "calms down " that axis.
For more details check out this lesson on angular velocity and this lesson on angular acceleration over at Lumen Learning
One major consideration, particularly in racing and freestyle quads is that the accleration component happens so quickly (in the 100ms range) that it doesn't play a major role in the pilot control of the craft. The acceleration factor primarily impacts tuning and the PID authority. If you look at flight logs of a stretch X quad you'll see the that the stretched axis the motors are running at a higher percentage to achieve the equivalent rotational rate to non-stretched axis (assuming the flight controller rates are set equal on both). If you were to adjust the FC rates so the motor output was equal on both axes, the stretched axis would rotate at a slower rate than the non-stretched axis. The rate systems generally equalize this, but there are still changes in the system that impact the effective resolution on the stretched axis vs the non-stretched. The end result is that the stretched axis feels less jumpy with finer control than the the non-stretched axis. Basically it has greater control authority with more effective steps of motor throttle resolution per degree of rotation.
Moment of Inertia
Moment of inertia also plays a second less obvious role. On most quads, the mass distribution from pitch axis is actually greater than the roll axis. The camera and battery are typically arranged so that their maximum distance from the center of rotation is greater on the pitch axis than the roll axis. This is especially true on freestyle quads with a top mounted battery. This is less true, but still true to a certain extent on race quads with bottom mount batteries. Essentially what this means is that assuming zero outside forces, the pitch axis is already going to react slower to to forces attempting to change the angular velocity than the roll axis, even with an exactly symmetrical frame.
Conclusions
The combination of these factors brings us to the crux of why frames may choose stretch or wide configurations.
Stretch X
Race frame prefer the stretch X configuration. This is increasing the effective control on the pitch axis. Remember because quads don't have vectored thrust (the thrust is always normal to the plane of the motors) They have to tip forward to gain speed.

Effectively this means that very small changes in pitch create large changes in altitude as the vertical component of thrust changes. The larger the forward angle, the larger the horizontal component of thrust, the more altitude is gained/lost per degree of rotation. This makes a more stable and less "twitchy" pitch axis a very attractive proposition for a race quad where fine control of altitude is critical. Also keep in mind roll and yaw are largely inverted when traveling at high speed amd the resulting high forward angles, and having a proportionally higher sustained rotation rate on the roll axis becomes important as well. Tracking in corners will feel more direct, and pitch will feel more stable.
Wide X
Freestyle frames prefer a wider roll axis. Essentially the wider roll axis and resulting increase in effective resolution is compensating for the moment of inertia on the pitch axis, giving finer control on the roll axis and giving a more similar feeling to pitch. You end up with a much more equal type of control on roll and pitch, which is important to freestyle pilots who are more about doing tricks and ballistic maneuvers that rely on symmetrical control on roll and pitch.
Also it's worth noting that a major reason for the wide X configuration on freestyle quads is keeping the props out of the view of the HD footage. I believe the this was the original reason, and the control gains were merely a happy accident.