Thanks for your answer. I indeed came to the same conclusion in the meantime.
One of the biggest issue I still have is that if you have the same variant in 2 different variants categories, the filter add-on will only show the first one, which is pretty disturbing.
Eg. I have a category "Adult Sizes" with S,M,L,XL and another "Kids Sizes" also with S,M,L,XL... the Kids Size category will never appear... I had to change the sizes to YS,YM,YL,YXL to make it work so that I don't have 2 variants with the same name.