Make the dependency on brick/math optional

[nicolas-grekas]

IIUC, the dep on brick/math is for supporting 32b archs.
These are pretty rare nowadays, so making the dep optional would help clean our vendor folders a bit.
The lib could throw a logic exception that'd suggest adding the dep when it's need and missing?

WDYT?