The first thing you should consider is what you have room for. A larger drone will normally need more space to fly and can cause more damage if crashed. I would recommend a small drone if you are just starting out or if you can only fly indoors, in which case you would want a tiny whoop.
The next thing I would think about is cost. A larger drone will be much more expensive for the drone itself, the propellers, the parts (frame, electronics, etc.), and especially batteries. While a larger drone may have slightly longer flight times, one battery for a large drone might cost 10 times as much as a battery for a small drone. So, with smaller drones, you can get more flight time per dollar of battery. (you can buy 4-6 small batteries for the price of one large battery and get a lot more flight time for the same amount of money). So, if you're on a budget, I definitely recommend a small drone. It's also worth mentioning that larger drones usually use batteries with more cells so they can be more dangerous if they catch on fire or puff up.
Next, there is flight performance. Generally, it is true that a smaller drone is more nimble than a large drone, although Larger drones are usually capable of much higher top-speeds. I must say though, after having lots of experience with different drones I much prefer the performance of a large, powerful drone to a small drone because small drones often suffer from lots of battery sag due to the lower battery voltages and so they cannot sustain high throttle as well. Also, a larger drone can carry a GoPro or other camera with ease.
So in the end:
- Get a small drone if: you only have a small area, you only have a little bit of money, you are very worried about damaging something, you want a lot of flight time for less money, or if you just don't want a big drone.
- Get a larger drone if: You have enough space and money, you want high performance/top speed, you want something that can lift a GoPro/camera/payload, or if you just want a larger drone.