[AutoDiff] Deprecate Differentiable.AllDifferentiableVariables.

[dan-zheng]

Deprecate the AllDifferentiableVariables associated type of the
Differentiable protocol.

AllDifferentiableVariables is not essential for differentiable
programming and was added as a workaround to enable key-path-based
machine learning optimizers: let parameters and gradients have the same type
(AllDifferentiableVariables == TangentVector) to enable joint key-path
iteration.

It is possible to implement key-path-based machine learning optimizers
via other means (do key-path-based operations on TangentVector, then
call Differentiable.move(along:) to perform update), so
AllDifferentiableVariables is no longer necessary.

---

Simplified Differentiable protocol:

/// A type that mathematically represents a differentiable manifold whose
/// tangent spaces are finite-dimensional.
public protocol Differentiable {
  /// A type representing a differentiable value’s derivatives.
  /// Mathematically, this is equivalent to the tangent bundle of the
  /// differentiable manifold represented by the differentiable type.
  associatedtype TangentVector: Differentiable & AdditiveArithmetic
    where TangentVector.TangentVector == TangentVector

  /// Moves `self` along the given direction. In Riemannian geometry,
  /// this is equivalent to exponential map, which moves `self` on the
  /// geodesic surface along the given tangent vector.
  mutating func move(along direction: TangentVector)
}

Resolves TF-707.

[rxwei]

In a separate patch, Array.DifferentiableView should be replaced with a struct TangentVector as described in the design overview.

[dan-zheng]

@swift-ci Please clean test tensorflow

[dan-zheng]

CI weirdly fails with:

/Users/swiftci/jenkins/workspace/swift-PR-TensorFlow-macOS/tensorflow-swift-apis/Sources/TensorFlow/Loss.swift:290:12: error: the compiler is unable to type-check this expression in reasonable time; try breaking up the expression into distinct sub-expressions
    return reduction(maxLogitsWithZero - logits * labels + log(1 + exp(-abs(logits))))
           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

But I'm not able to reproduce this error locally. I'll break up the expression into two subexpressions nonetheless and try again.

[dan-zheng]

@swift-ci Please clean test tensorflow

[dan-zheng]

Oops, I missed failing AutoDiff tests.

Failing Tests (7):
    Swift(linux-x86_64) :: AutoDiff/leakchecking.swift
    Swift(linux-x86_64) :: AutoDiff/vtable_sil.swift
    Swift(linux-x86_64) :: AutoDiff/generics.swift
    Swift(linux-x86_64) :: AutoDiff/derived_differentiable.swift
    Swift(linux-x86_64) :: AutoDiff/control_flow.swift
    Swift(linux-x86_64) :: AutoDiff/e2e_differentiable_property.swift
    Swift(linux-x86_64) :: AutoDiff/derived_conformances.swift

Will fix tomorrow.

[dan-zheng]

@swift-ci Please test tensorflow

[dan-zheng]

Note: tensorflow/swift-models and https://github.com./fastai/fastai_dev/tree/master/swift likely need to be updated.

[rxwei]

@BradLarson @sgugger