It is probably most important to watch for periodic leaks (make sure your periodic triggers are leak-free--the ones that run every X seconds [including the periodic parts of triggered-spells]).
I recommend that you clean them up in general. It is good practice. It might not make a huge difference, but I don't see a point in ignoring some and fixing a few--it becomes confusing and requires more thinking.
To put it into perspective, you're going to need several thousand or tens of thousands of leaks to degrade performance on modern computers (maybe even hundreds of thousands). This is a lot, but keep in mind that periodic triggers can easily get to these amounts within a matter of seconds.