I know nothing about desyncs, but I seem to remember hearing somewhere that the optimizer turns custom unit data into SLKs, which can cause desyncs. Again, take this with a grain of salt, not sure about this at all.
Either way, it's worth at least testing if the desyncs still happen as frequently without the optimizer.