ah, i found out that this is because the object/material is still being affected by light. where as the default shaders don't (once they are light bakked). Still not sure if it is a bug or me not understand it ;)

It seems I am still getting this. I have spent hours trying to fix it, looking online, but not effect.
I am not sure if I am doing something wrong on my side or if it is a SF bug.

I tried things like:
Deferred vs forward rendering (unity and SF settings)
Gamma or linear space
Different Light map settings like Directional or non-directional.

I am clueless. Help is highly appreciated! Thanks!