I read online in many places that you should just tighten the nut until you can't anymore with your hands, and some people recommended using pliers or socket wrenches. My problem is, no matter what tool I use, the nut will not tighten past the point where its top is flush with the shaft (I don't know much about Nyloc but this seems by design). Here are the motors I am using (https://www.amazon.com/iFlight-2750KV-Brushless-Racing-Quadcopter/dp/B07Y9JK2MW). This first result is what I achieve almost every time after tightening as far as the nut will go: But this causes issues because I can tip the rotor in different directions with my hand, and so it seems like it would wobble when turning, and the fit is definitely not secure. (I at first thought this was because the 5mm hole in the rotor was too large for the 4mm shaft, but I was told that the threading in the shaft was 5mm so it would be fine). This second result was achieved after inserting two washers between the nut and rotor, tightening down the nut, untightening and removing the washers, and the tightening the nut once more. I removed the washers because while I had read that some people tighten their rotors with washers, the gap seemed too large for this to be the solution, so I assumed I was doing something incorrectly in the mounting. Where did I go wrong, and how should these rotors be mounted?
I've had the same thing happen when using these same motors and props (and others). The propeller seems to get stuck at a point where it is not flush with the motor. From my experience, the solution is as simple as tightening it further. I use a tool like this that allows me to apply more force than a typical prop-tightening tool.
It should look like this when fastened correctly, with the prop sitting flush with both the motor bell and the nut:
@JacobB's answer that you just need to tighten the nut more is correct, but I just want to explain why it (feels like it) is stuck when it is flush with the top of the motor shaft.
Nyloc nuts, or nylon lock nuts has, as the name implies, a ring of nylon at the top (the white part in the image below). When tightening the nut the nylon has to deform, since it won't fit the threads otherwise, and this is what makes the nut lock in place better than a regular one. Because the nylon doesn't return to its original shape after unscrewing the nut, it will become easier and easier to tighten it the more you use it. Eventually it won't lock any better than a regular nut, and you should think about replacing it to avoid the propellers coming loose mid flight.
Also, to add on how much to tighten: tighten until you can't move the propeller (without moving the motor) with one hand while holding the motor with your other hand. The nut will then be tight enough so that the propeller doesn't spin freely during flight, but it won't be so tight that something risks being damaged.