I am trying to make a quadrotor UAV (with inverted props) which has 4 circular/semi-circular attachments to help it glide for a longer distance. I have a constraint on the mass, but I don't have any constraint on the area. How am I supposed to size the circular areas? Could anyone help with the formulas or the matlab code or the analysis I must do?

There isn't a simple formula for this, as bigger, slower rotors are more efficient for hovering. However as the rotor gets bigger, the airspeed through it is lower, which limits your top speed.

This means there's a design compromise between top speed and hovering efficiency. You need to predict the proportion of time you spend hovering and how fast you want to go.

Say you have a prop with 4" (100mm) of pitch, spinning at 1000rpm, it would screw through the air 100mm for each revolution, or 100m per minute, or 1.66m/s. That means that when flying at that speed, the prop would be making zero thrust. The air it blows backwards is moving at the same speed as the air around it. Thrust will drop linearly from static to the pitch speed.

If you're thinking of adding wings, the equation for lift is fairly simple: lift force (in Newtons) = wing area (in square meters) * air density (about 1kg/cubic meter) * speed (m/s) squared * 0.5 * lift coefficient (which is about a maximum of 1 for a reasonably well designed wing).

You can look up propeller theory, it's too long to reproduce here, but the actual results depend quite heavily on the aerofoil and shape of the prop blade so to get a good estimate you either need to make some measurements or use someone else's measurements. You can find tables of data and on-line calculators for well known props, showing static thrust and power at various speeds.

For a quadcopter flying sideways (without wings) the thrust is angled, with the vertical component equalling the weight and the horizontal component equally in the drag.

Finally you need to estimate the drag at your target speed. The equation is similar to lift, but the drag coefficient depends on how aerodynamic the aircraft is. You can look up the drag coefficient for various simple shapes but they vary quite widely, with a streamlined strut about 40 times better than a square bar of the same frontal area.

For a full size aircraft, engineers will do all these calculations, and back them up with wind tunnel tests. In the RC world people tend to just copy a popular design, then modify it a little and measure the results. (and when I say 'measure', I mean they often just declare that's it's better without taking any measurements ;-)