A triangle is extremely difficult to do, but you can do a cone very easily (think of a sector of a circle). How, you ask?
First, pick every unit within the radius.
Next, remove any unit from that group such that the angle from the caster to the unit is NOT less than or equal to ((angle from caster to target point)+60) and is NOT greater or equal to ((angle from caster to target point)-60).
And there you have it. That group now holds all units within that cone.
EDIT: Alternatively and probably more efficient:
Pick every unit within the radius such that the angle from the caster to the unit is less than or equal to ((angle from caster to target point)+60) and is greater or equal to ((angle from caster to target point)-60).