DiffSharp: Differentiable Functional Programming

DiffSharp is a tensor library with advanced support for differentiable programming. It is designed for use in machine learning, probabilistic programming, optimization and other domains.

DiffSharp provides advanced automatic differentiation capabilities for tensor code, making it possible to use derivative-taking operations, including gradients, Hessians, Jacobians, directional derivatives, and matrix-free Hessian- and Jacobian-vector products, as higher-order function compositions. This goes far beyond the standard reverse-mode gradients of traditional tensor libraries such as TensorFlow and PyTorch, allowing the use of nested forward and reverse differentiation up to any level, meaning that you can compute higher-order derivatives efficiently or differentiate functions that are internally making use of differentiation. Please see API Overview for a list of available operations.

DiffSharp 1.0 is implemented in F# and uses PyTorch C++ tensors (without the derivative computation graph) as the default raw-tensor backend. It is tested on Linux and Windows. DiffSharp is developed by Atılım Güneş Baydin, Don Syme and other contributors, having started as a project supervised by Barak Pearlmutter and Jeffrey Siskind. Please join us!

The library and documentation are undergoing development.

Current features and roadmap

The primary features of DiffSharp 1.0 are:

  • A tensor programming model for F#.
  • A reference backend for correctness testing.
  • PyTorch backend for CUDA support and highly optimized native tensor operations.

  • Nested differentiation for tensors, supporting forward and reverse AD, or any combination thereof, up to any level.
  • Matrix-free Jacobian- and Hessian-vector products.
  • Common optimizers and model elements including convolutions.
  • Probability distributions.

Quick usage example

Below is a sample of using DiffSharp. You can access this sample as a script or a .NET Interactive Jupyter Notebook (open in Binder).

open DiffSharp

// A 1D tensor
let t1 = dsharp.tensor [ 0.0 .. 0.2 .. 1.0 ]

// A 2x2 tensor
let t2 = dsharp.tensor [ [ 0; 1 ]; [ 2; 2 ] ]

// Define a scalar-to-scalar function
let f (x: Tensor) = sin (sqrt x)

f (dsharp.tensor 1.2)

// Get its derivative
let df = dsharp.diff f

df (dsharp.tensor 1.2)

// Now define a vector-to-scalar function
let g (x: Tensor) = exp (x.[0] * x.[1]) + x.[2]

g (dsharp.tensor [ 0.0; 0.3; 0.1 ])

// Now compute the gradient of g
let gg = dsharp.grad g 

gg (dsharp.tensor [ 0.0; 0.3; 0.1 ])

// Compute the hessian of g
let hg = dsharp.hessian g

hg (dsharp.tensor [ 0.0; 0.3; 0.1 ])

More information

To learn more about DiffSharp, use the navigation links to the left.

If you are using DiffSharp, please raise any issues you might have on GitHub. We also have a Gitter chat room. If you would like to cite this library, please use the following information:

Baydin, A.G., Pearlmutter, B.A., Radul, A.A. and Siskind, J.M., 2017. Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Research, 18(1), pp.5595-5637. (link)

namespace DiffSharp
val t1 : Tensor
type dsharp =
  static member abs : input:Tensor -> Tensor
  static member acos : input:Tensor -> Tensor
  static member add : a:Tensor * b:Tensor -> Tensor
  static member arange : endVal:int * ?startVal:int * ?step:int * ?dtype:Dtype * ?device:Device * ?backend:Backend -> Tensor
  static member arange : endVal:float * ?startVal:float * ?step:float * ?dtype:Dtype * ?device:Device * ?backend:Backend -> Tensor
  static member arangeLike : input:Tensor * endVal:int * ?startVal:int * ?step:int * ?dtype:Dtype * ?device:Device * ?backend:Backend -> Tensor
  static member arangeLike : input:Tensor * endVal:float * ?startVal:float * ?step:float * ?dtype:Dtype * ?device:Device * ?backend:Backend -> Tensor
  static member argmax : input:Tensor -> int []
  static member argmin : input:Tensor -> int []
  static member asin : input:Tensor -> Tensor
  ...
static member dsharp.tensor : value:obj * ?dtype:Dtype * ?device:Device * ?backend:Backend -> Tensor
val t2 : Tensor
val f : x:Tensor -> Tensor
val x : Tensor
type Tensor =
  private | Tensor0 of primalRaw: RawTensor
          | TensorF of primal: Tensor * derivative: Tensor * nestingTag: uint32
          | TensorR of primal: Tensor * derivative: Tensor ref * parentOp: TensorOp * fanout: uint32 ref * nestingTag: uint32
    interface IConvertible
    interface IComparable
    override Equals : other:obj -> bool
    override GetHashCode : unit -> int
    member private GetSlice : bounds:int [,] -> Tensor
    override ToString : unit -> string
    member abs : unit -> Tensor
    member acos : unit -> Tensor
    member add : b:scalar -> Tensor
    member add : b:Tensor -> Tensor
    ...
val sin : value:'T -> 'T (requires member Sin)
val sqrt : value:'T -> 'U (requires member Sqrt)
val df : (Tensor -> Tensor)
static member dsharp.diff : f:(Tensor -> Tensor) -> x:Tensor -> Tensor
val g : x:Tensor -> Tensor
val exp : value:'T -> 'T (requires member Exp)
val gg : (Tensor -> Tensor)
static member dsharp.grad : f:(Tensor -> Tensor) -> x:Tensor -> Tensor
val hg : (Tensor -> Tensor)
static member dsharp.hessian : f:(Tensor -> Tensor) -> x:Tensor -> Tensor

© Copyright 2020, DiffSharp Contributors.