Right now, HVPs are computed individually and assembled, but this was only useful for the sparse case