GPU array preservation and parallelism in wrong mode

[gdalle]

Currently:

• Wrong-mode pushforward or pullback runs the map on the CPU and copies back to the GPU

DifferentiationInterface.jl/DifferentiationInterface/src/first_order/pullback.jl

Lines 354 to 370 in 7e80fef

	function _value_and_pullback_via_pushforward(
	f::F,
	pushforward_prep::PushforwardPrep,
	backend::AbstractADType,
	x::AbstractArray{<:Real},
	ty::NTuple{B},
	contexts::Vararg{Context, C},
	) where {F, B, C}
	y = f(x, map(unwrap, contexts)...)
	tx = map(CartesianIndices(x)) do j
	a = only(pushforward(f, pushforward_prep, backend, x, (basis(x, j),), contexts...))
	map(ty) do dy
	dot(a, dy)
	end
	end
	return y, arroftup_to_tupofarr(tx)
	end

• If I try to parallelize it, I need to duplicate the prep object to avoid race conditions, which is not possible

See also:

• fix: respect array type in wrong-mode pushforward/pullback #620
• GPU scalar indexing when transpose(::CuArray) #970