The best way is to register the damage and then alter it. However, the altering process is pretty complex for some things (mostly when reducing damage). Since you are new to JASS, I wouldn't recommend using Nes' DDS.
I would recommend something like Bribe's damage engine:
http://www.hiveworkshop.com/forums/...e-v2-2-1-0-a-201016/?prev=d=list&r=20&u=Bribe
Or looking_for_help's damage engine:
http://www.hiveworkshop.com/forums/...a-231846/?prev=d=list&r=20&u=Looking_for_help
They both allow for damage modification. You would multiply the damage result (depending on the critical strike ability. e.g. to deal two times normal damage, you'd multiply the damage by 2). However, I can't go into much detail since I haven't used those systems much. The best thing to do would be to look at the test maps for either one. Then you can make your spell based off of it.
Damage detection is sort of a weird thing, and damage modification can get even weirder. Normally I'd promote trying it out yourself--and you can feel free to--but I think that you may want to at least try to use a DDS (like the ones linked above) since they perform their tasks very well. Good luck!
If you need help with creating the floating text, let me know.