Thrust is proportional to the change in momentum of the air passing through the prop - i.e. how much the prop speeds it up.
The power required to do this is proportional to the kinetic energy of the air, which is proportional to speed squared. That 'squared' is the problem.
A smaller prop acts on less air, so it has to speed it up more to generate the same thrust. Half as much air moving twice as fast generates the same thrust, but takes twice as much power.
This explains why helicopters can hover relatively economically, propeller planes only if they're powerful aerobatic models, jets only if they're extremely powerful and carrying a minimum of weight, and no one uses rockets unless they have to.