DiffSharp


Please note: this is an introductory example and therefore the code is kept very simple. More advanced cases, including recurrent and convolutional networks, will be released as part of the Hype library built on top of DiffSharp.

Neural Networks

Artificial neural networks are computational models inspired by biological nervous systems, capable of approximating functions that depend on a large number of inputs. A network is defined by a connectivity structure and a set of weights between interconnected processing units ("neurons"). Neural networks "learn" a given task by tuning the set of weights under an optimization procedure.

Let's create a feedforward neural network with DiffSharp and implement the backpropagation algorithm for training it. As mentioned before, backpropagation is just a special case of reverse mode AD.

We start by defining our neural network structure.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
open DiffSharp.AD.Float64
open DiffSharp.Util

// A layer of neurons
type Layer =
    {mutable W:DM  // Weight matrix
     mutable b:DV  // Bias vector
     a:DV->DV}     // Activation function

// A feedforward network of several layers
type Network =
    {layers:Layer[]} // The layers forming this network

The network will consist of several layers of neurons. Each neuron works by taking an input vector \(\mathbf{x}\) and calculating the activation (output)

\[ a = \sigma \left(\sum_{i} w_i x_i + b\right) \; ,\]

where \(w_i\) are synapse weights associated with each input, \(b\) is a bias, and \(\sigma\) is an activation function representing the rate of action potential firing in the neuron.

Chart

A conventional choice for the activation function had been the sigmoid \(\sigma (z) = 1 / (1 + e^{-z})\) for a long period because of its simple derivative and gain control properties. Recently the hyperbolic tangent \(\tanh\) and the rectified linear unit \(\textrm{ReLU}(z) = \max(0, z)\) have been more popular choices due to their convergence and performance characteristics.

Now let's write the network evaluation code and a function for creating a given network configuration and initializing the weights and biases with small random values. In practice, proper weight initialization has been demonstrated to have an important effect on training convergence and it would depend on the network structure and the type of activation functions used.

Network evaluation is implemented using linear algebra, where we have a weight matrix \(\mathbf{W}^l\) holding the weights of all neurons in layer \(l\). The elements of this matrix \(w_{ij}\) represent the weight between the \(j\)-th neuron in layer \(l - 1\) and the \(i\)-th neuron in layer \(l\). We also have a bias vector \(\mathbf{b}^l\) holding the biases of all neurons in layer \(l\) (one bias per neuron). Thus evaluating the network is just a matter of computing the activation

\[ \mathbf{a}^l = \mathbf{W}^l \mathbf{a}^{l - 1} + \mathbf{b}^l \; ,\]

for each layer and passing the activation vector (output) of each layer as the input to the next layer, until we get the network output as the output vector of the last layer.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
let runLayer (x:DV) (l:Layer) =
    l.W * x + l.b |> l.a

let runNetwork (x:DV) (n:Network) =
    Array.fold runLayer x n.layers


let rnd = System.Random()

// Initialize a fully connected feedforward neural network
// Weights and biases between -0.5 and 0.5
// l : number of inputs, followed by the number of neurons in each subsequent layer
let createNetwork (l:int[]) =
    {layers = Array.init (l.Length - 1) (fun i ->
        {W = DM.init l.[i + 1] l.[i] (fun _ _ -> -0.5 + rnd.NextDouble())
         b = DV.init l.[i + 1] (fun _ -> -0.5 + rnd.NextDouble())
         a = sigmoid})}

This gives us an easily scalable feedforward network architecture capable of expressing any number of inputs, outputs, and hidden layers. The network is fully connected, meaning that each neuron in a layer receives the output of all the neurons in the previous layer.

For example,

1: 
let net1 = createNetwork [|3; 4; 2|]

would give us the following network with 3 input nodes, a hidden layer with 4 neurons, and an output layer with 2 neurons:

Chart

We can also have more than one hidden layer.

For training networks, we will make use of reverse mode AD for propagating a loss \(Q\) at the output backwards through the network weights. This will give us the partial derivative of the loss at the output with respect to each weight \(w_i\) and bias \(b_i\) in the network, which we will then use in an update rule

\[ \begin{eqnarray*} \Delta w_i &=& -\eta \frac{\partial Q}{\partial w_i} \; ,\\ \Delta b_i &=& -\eta \frac{\partial Q}{\partial b_i} \; ,\\ \end{eqnarray*}\]

where \(\eta\) is the learning rate.

We will use the quadratic loss

\[ Q = \sum_{i=1}^{d} \Vert \mathbf{y}_i - \mathbf{a}(\mathbf{x}_i) \Vert^{2} \; ,\]

where there are \(d\) cases in the training set, \(\mathbf{y}_i\) is the \(i\)-th training target and \(\mathbf{a}(\mathbf{x}_i)\) is the output vector of the last layer when the \(i\)-th training input \(\mathbf{x}_i\) is supplied to the first layer.

Please see the Nested AD page for a better understanding of the low-level usage of adjoints and their backwards propagation.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
// The backpropagation algorithm
// n: network to be trained
// eta: learning rate
// epochs: number of training epochs
// x: training input vectors
// y: training target vectors
let backprop (n:Network) eta epochs (x:DV[]) (y:DV[]) =
    let i = DiffSharp.Util.GlobalTagger.Next
    seq {for j in 0 .. epochs do
            for l in n.layers do
                l.W <- l.W |> makeReverse i
                l.b <- l.b |> makeReverse i

            let L = Array.map2 (fun x y -> DV.l2normSq (y - runNetwork x n)) x y |> Array.sum
            L |> reverseProp (D 1.) // Propagate adjoint value 1 backward

            for l in n.layers do
                l.W <- primal (l.W.P - eta * l.W.A)
                l.b <- primal (l.b.P - eta * l.b.A)

            printfn "Iteration %i, loss %f" j (float L)
            yield float L}

Using reverse mode AD here has two big advantages: (1) it makes the backpropagation code succinct and straightforward to write and maintain; and (2) it allows us to freely code our neural network without the burden of coding derivatives or modifying the backpropagation code accordingly.

We can now test the algorithm by training some networks.

It is known that linearly separable rules such as logical disjunction can be learned by a single neuron.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
open FSharp.Charting

let ORx = [|toDV [0.; 0.]
            toDV [0.; 1.]
            toDV [1.; 0.]
            toDV [1.; 1.]|]
let ORy = [|toDV [0.]
            toDV [1.]
            toDV [1.]
            toDV [1.]|]

// 2 inputs, one layer with one neuron
let net2 = createNetwork [|2; 1|]

// Train
let train2 = backprop net2 0.9 1000 ORx ORy

// Plot the error during training
Chart.Line train2
1: 
2: 
val net2 : Network = {layers = [|{W = DM [[0.230677625; 0.1414874814]];
                                  b = DV [|0.4233988253|];}|];}
Chart

Linearly inseparable problems such as exclusive or require one or more hidden layers to learn.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
let XORx = [|toDV [0.; 0.]
             toDV [0.; 1.]
             toDV [1.; 0.]
             toDV [1.; 1.]|]
let XORy = [|toDV [0.]
             toDV [1.]
             toDV [1.]
             toDV [0.]|]

// 2 inputs, 3 neurons in a hidden layer, 1 neuron in the output layer
let net3 = createNetwork [|2; 3; 1|]

// Train
let train3 = backprop net3 0.9 1000 XORx XORy

// Plot the error during training
Chart.Line train3
1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
val net3 : Network =
  {layers =
    [|{W = DM [[-0.04536837132; -0.3447727025]
               [-0.07626016418; 0.06522091877]
               [0.2581558948; 0.1597980939]];
       b = DV [|-0.3051199176; 0.2980325892; 0.4621827649|];};
      {W = DM [[-0.347911722; 0.2696812725; 0.2704776571]];
       b = DV [|-0.1477482923|];}|];}
Chart

Some Performance Tricks

For higher training performance, it is better to propagate matrices, instead of vectors, through the network. In other words, instead of treating the traning data as a set of input vectors \(\mathbf{x}_i \in \mathbb{R}^n\) and target vectors \(\mathbf{y}_i \in \mathbb{R}^m\), \(i = 1 \dots d\), we can have an input matrix \(\mathbf{X} \in \mathbb{R}^{n\times d}\) and a target matrix \(\mathbf{Y} \in \mathbb{R}^{m\times d}\), where \(d\) is the number of examples in the training set. In this scheme, vectors \(\mathbf{x}_i\) form the columns of the input matrix \(\mathbf{X}\), and propagating \(\mathbf{X}\) through the layers computes the network's output for all \(\mathbf{x}_i\) simultaneously. In DiffSharp, as in other linear algebra libraries, computing matrix-matrix multiplications are a lot more efficient than computing a series of matrix-vector multiplications.

Let's modify the network evaluation code to propagate matrices.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
// A layer of neurons
type Layer' =
    {mutable W:DM  // Weight matrix
     mutable b:DV  // Bias vector
     a:DM->DM}     // Activation function

// A feedforward network of neuron layers
type Network' =
    {layers:Layer'[]} // The layers forming this network

let runLayer' (x:DM) (l:Layer') =
    l.W * x + (DM.createCols x.Cols l.b) |> l.a

let runNetwork' (x:DM) (n:Network') =
    Array.fold runLayer' x n.layers

The backpropagation code given previously computed the loss over the whole set of training cases at each iteration and used simple gradient descent to iteratively decrease this loss. When the training set is large, this leads to training time bottlenecks.

In practice, backpropagation is combined with stochastic gradient descent (SGD), which makes the duration of each training iteration independent from the training set size (also see the SGD example). Furthermore, instead of using one random case at a time to compute the loss, SGD is used with "minibatches" of more than one case (a small number compared to the full training set size). Minibatches allow us to exploit the matrix-matrix multiplication trick for performance and also have the added benefit of smoothing the SGD estimation of the true gradient.

Backpropagation combined with SGD and minibatches is the de facto standard for training neural networks.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
24: 
25: 
26: 
27: 
28: 
29: 
30: 
31: 
32: 
33: 
// Backpropagation with SGD and minibatches
// n: network
// eta: learning rate
// epochs: number of training epochs
// mbsize: minibatch size
// loss: loss function
// x: training input matrix
// y: training target matrix
let backprop' (n:Network') (eta:float) epochs mbsize loss (x:DM) (y:DM) =
    let i = DiffSharp.Util.GlobalTagger.Next
    let mutable b = 0
    let batches = x.Cols / mbsize
    let mutable j = 0
    while j < epochs do
        b <- 0
        while b < batches do
            let mbX = x.[*, (b * mbsize)..((b + 1) * mbsize - 1)]
            let mbY = y.[*, (b * mbsize)..((b + 1) * mbsize - 1)]

            for l in n.layers do
                l.W <- l.W |> makeReverse i
                l.b <- l.b |> makeReverse i

            let L:D = loss (runNetwork' mbX n) mbY
            L |> reverseProp (D 1.)

            for l in n.layers do
                l.W <- primal (l.W.P - eta * l.W.A)
                l.b <- primal (l.b.P - eta * l.b.A)

            printfn "Epoch %i, minibatch %i, loss %f" j b (float L)
            b <- b + 1
        j <- j + 1

The Obligatory MNIST Example

The MNIST database of handwritten digits is commonly used for demonstrating neural network training. The database contains 60,000 training images and 10,000 testing images. More information on the database and downloadable files can be found here.

The following code reads the standard MNIST files into matrices.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
15: 
16: 
17: 
18: 
19: 
20: 
21: 
22: 
23: 
open System.IO

type MNIST =
    static member Load(filename, items) =
        let d = new BinaryReader(File.Open(filename, FileMode.Open, FileAccess.Read, FileShare.Read))
        let magicnumber = d.ReadInt32() |> System.Net.IPAddress.NetworkToHostOrder
        match magicnumber with
        | 2049 -> // Labels
            let maxitems = d.ReadInt32() |> System.Net.IPAddress.NetworkToHostOrder
            d.ReadBytes(min items maxitems)
            |> Array.map float |> DV
            |> DM.ofDV 1
        | 2051 -> // Images
            let maxitems = d.ReadInt32() |> System.Net.IPAddress.NetworkToHostOrder
            let rows = d.ReadInt32() |> System.Net.IPAddress.NetworkToHostOrder
            let cols = d.ReadInt32() |> System.Net.IPAddress.NetworkToHostOrder
            let n = min items maxitems
            d.ReadBytes(n * rows * cols)
            |> Array.map float |> DV
            |> DM.ofDV n
            |> DM.transpose
        | _ -> failwith "Given file is not in the MNIST format."
    static member Load(filename) = MNIST.Load(filename, System.Int32.MaxValue)

For a quick demonstration, let's start by loading 10,000 training images and their class labels.

1: 
2: 
let mnistTrainX = MNIST.Load("C:/datasets/MNIST/train-images.idx3-ubyte", 10000)
let mnistTrainY = MNIST.Load("C:/datasets/MNIST/train-labels.idx1-ubyte", 10000)

The first matrix, 784x10,000, contains one raster image of 784 pixels (28x28) in each column and the second matrix, 1x10,000, contains the class labels (0, 1, 2, 3, 4, 5, 6, 7, 8, 9) of each of these images.

For this classification task, we will have a neural network with 784 input nodes, one for each pixel in the input image; a hidden layer of 300 nodes; and an output layer of 10 nodes, representing the scores for each possible class. When an input image is propagated through the network, the output node with the highest score will be the class predicted for that image.

There are several ways of training such a network. Here we use \(\tanh\) activations in the hidden layer and \(\textrm{softmax}\) activations in the output layer. The softmax function

\[ \sigma(\mathbf{z})_{j} = \frac{e^{z_j}}{\sum_{k=1}^{K} e^{z_k}} \; ,\]

for a vector \(\mathbf{z}\) of length \(K\), transforms real-valued scores \(z_k\) into a vector \(\mathbf{\sigma}\) of components \(\sigma_j\) between zero and one, where \(\textrm{sum}(\mathbf{\sigma}) = 1\). Thus, the resulting vector \(\sigma\) is interpretable as normalized class probabilities.

1: 
2: 
3: 
4: 
5: 
6: 
7: 
8: 
9: 
let l0 = {W = DM.init 300 784 (fun _ _ -> -0.075 + 0.15 * rnd.NextDouble())
          b = DV.zeroCreate 300
          a = tanh}

let l1 = {W = DM.init 10 300 (fun _ _ -> -0.075 + 0.15 * rnd.NextDouble())
          b = DV.zeroCreate 10
          a = DM.mapCols softmax}

let nn = {layers = [|l0; l1|]}

We train this softmax classifier with a cross-entropy loss of the form

\[ Q(\mathbf{p}, \mathbf{q}) = -\sum_i \mathbf{p}(\mathbf{x}_i) \log \mathbf{q}(\mathbf{x}_i) \; ,\]

where \(\mathbf{q}\) is the vector of predicted class probabilities (the output of the network) and \(\mathbf{p}\) is the "true" distribution (e.g., \(\mathbf{p} = (0,0,1,0,0,0,0,0,0,0)\) for \(\mathbf{y}_i = 2\)).

1: 
2: 
3: 
4: 
5: 
let crossEntropy (x:DM) (y:DM) =
    -(x |> DM.toCols |> Seq.mapi (fun i v -> 
        (DV.standardBasis v.Length (int (float y.[0, i]))) * log v) |> Seq.sum) / x.Cols

backprop' nn 0.01 10 500 crossEntropy mnistTrainX mnistTrainY
1: 
2: 
3: 
4: 
5: 
...
Epoch 7, minibatch 17, loss 0.612930
Epoch 7, minibatch 18, loss 0.523338
Epoch 7, minibatch 19, loss 0.487236
...

Let's test the trained network on a few digits from the test set.

 1: 
 2: 
 3: 
 4: 
 5: 
 6: 
 7: 
 8: 
 9: 
10: 
11: 
12: 
13: 
14: 
let mnistTestX = MNIST.Load("C:/datasets/MNIST/t10k-images.idx3-ubyte", 5)
let mnistTestY = MNIST.Load("C:/datasets/MNIST/t10k-labels.idx1-ubyte", 5)

// Run the test set through the network
let testY = runNetwork' mnistTestX nn |> primal

// Compute the cross-entropy loss for the test set
let testLoss = crossEntropy testY mnistTestY

// Predicted classes
let testPredict = testY |> DM.toCols |> Seq.map DV.maxIndex |> Seq.toArray

// Correct classes
let testTrue = mnistTestY
1: 
2: 
3: 
val testLoss : D = D 0.3540015679
val testPredict : int [] = [|7; 2; 1; 0; 4|]
val testTrue : DM = DM [[7.0; 2.0; 1.0; 0.0; 4.0]]
1: 
2: 
3: 
for i = 0 to testY.Cols - 1 do
    printfn "Predicted label: %i" testPredict.[i]
    printfn "Image:\n %s" ((mnistTestX.[*,i] |> DM.ofDV 28).Visualize())
  1: 
  2: 
  3: 
  4: 
  5: 
  6: 
  7: 
  8: 
  9: 
 10: 
 11: 
 12: 
 13: 
 14: 
 15: 
 16: 
 17: 
 18: 
 19: 
 20: 
 21: 
 22: 
 23: 
 24: 
 25: 
 26: 
 27: 
 28: 
 29: 
 30: 
 31: 
 32: 
 33: 
 34: 
 35: 
 36: 
 37: 
 38: 
 39: 
 40: 
 41: 
 42: 
 43: 
 44: 
 45: 
 46: 
 47: 
 48: 
 49: 
 50: 
 51: 
 52: 
 53: 
 54: 
 55: 
 56: 
 57: 
 58: 
 59: 
 60: 
 61: 
 62: 
 63: 
 64: 
 65: 
 66: 
 67: 
 68: 
 69: 
 70: 
 71: 
 72: 
 73: 
 74: 
 75: 
 76: 
 77: 
 78: 
 79: 
 80: 
 81: 
 82: 
 83: 
 84: 
 85: 
 86: 
 87: 
 88: 
 89: 
 90: 
 91: 
 92: 
 93: 
 94: 
 95: 
 96: 
 97: 
 98: 
 99: 
100: 
101: 
102: 
103: 
104: 
105: 
106: 
107: 
108: 
109: 
110: 
111: 
112: 
113: 
114: 
115: 
116: 
117: 
118: 
119: 
120: 
121: 
122: 
123: 
124: 
125: 
126: 
127: 
128: 
129: 
130: 
131: 
132: 
133: 
134: 
135: 
136: 
137: 
138: 
139: 
140: 
141: 
142: 
143: 
144: 
145: 
146: 
147: 
148: 
149: 
150: 
151: 
152: 
153: 
154: 
155: 
156: 
Predicted label: 7
Image:
 DM : 28 x 28







      ■■■■■■                
      ■■■■■■■■■■■■■■■■      
      ■■■■■■■■■■■■■■■■      
            ■ ■■■■ ■■■      
                  ■■■       
                  ■■■       
                 ■■■■       
                ■■■■        
                ■■■         
                ■■■         
               ■■■          
              ■■■■          
              ■■■           
             ■■■■           
            ■■■■            
           ■■■■             
           ■■■■             
          ■■■■■             
          ■■■■■             
          ■■■               


Predicted label: 2
Image:
 DM : 28 x 28



          ■■■■■■■           
         ■■■■■■■■           
        ■■■■■■■■■■          
       ■■■■    ■■■          
       ■■■    ■■■■          
              ■■■■          
             ■■■■           
            ■■■■■           
            ■■■■            
           ■■■■             
           ■■■              
          ■■■■              
         ■■■■               
         ■■■■               
        ■■■■                
        ■■■                 
        ■■■■         ■■■■■  
        ■■■■■■■■■■■■■■■■■■■ 
        ■■■■■■■■■■■■■■■■■■■ 
         ■■■■■■■■■■■■       






Predicted label: 1
Image:
 DM : 28 x 28




                ■■■         
                ■■■         
                ■■          
               ■■■          
               ■■■          
               ■■           
              ■■■           
              ■■■           
              ■■■           
             ■■■            
             ■■■            
             ■■■            
            ■■■             
            ■■■             
            ■■■             
            ■■■             
           ■■■■             
           ■■■              
           ■■■              
           ■■               





Predicted label: 0
Image:
 DM : 28 x 28




             ■■■            
             ■■■■           
            ■■■■■           
          ■■■■■■■■■         
          ■■■■■■■■■■        
         ■■■■■■■■■■■        
        ■■■■■■■■■■■■■       
        ■■■■■■    ■■■■      
        ■■■■      ■■■■      
        ■■■        ■■■      
        ■■         ■■■■     
       ■■■        ■■■■      
       ■■■       ■■■■■      
       ■■■      ■■■■■■      
       ■■■     ■■■■■■       
       ■■■■■■■■■■■■■        
       ■■■■■■■■■■■■■        
        ■■■■■■■■■■          
        ■■■■■■■■■           
          ■■■■■■            





Predicted label: 4
Image:
 DM : 28 x 28





          ■■       ■■       
          ■■       ■■       
          ■■       ■■       
         ■■■       ■■       
        ■■■        ■■       
        ■■■        ■■       
       ■■■        ■■■       
       ■■■       ■■■■       
       ■■        ■■■        
       ■■        ■■■        
       ■■        ■■■        
       ■■■■■■■■■■■■■        
       ■■■■■■■■■■■■         
         ■■■■■  ■■■         
                ■■■■        
                ■■■         
                ■■■         
                ■■■         
                ■■■         
                ■■          
namespace DiffSharp
namespace DiffSharp.AD
module Float64

from DiffSharp.AD
module Util

from DiffSharp
type Layer =
  {mutable W: DM;
   mutable b: DV;
   a: DV -> DV;}

Full name: Examples-neuralnetworks.Layer
Layer.W: DM
Multiple items
union case DM.DM: float [,] -> DM

--------------------
module DM

from DiffSharp.AD.Float64

--------------------
type DM =
  | DM of float [,]
  | DMF of DM * DM * uint32
  | DMR of DM * DM ref * TraceOp * uint32 ref * uint32
  member Copy : unit -> DM
  member GetCols : unit -> seq<DV>
  member GetForward : t:DM * i:uint32 -> DM
  member GetReverse : i:uint32 -> DM
  member GetRows : unit -> seq<DV>
  member GetSlice : rowStart:int option * rowFinish:int option * col:int -> DV
  member GetSlice : row:int * colStart:int option * colFinish:int option -> DV
  member GetSlice : rowStart:int option * rowFinish:int option * colStart:int option * colFinish:int option -> DM
  member ToMathematicaString : unit -> string
  member ToMatlabString : unit -> string
  override ToString : unit -> string
  member Visualize : unit -> string
  member A : DM
  member Cols : int
  member F : uint32
  member Item : i:int * j:int -> D with get
  member Length : int
  member P : DM
  member PD : DM
  member Rows : int
  member T : DM
  member A : DM with set
  member F : uint32 with set
  static member Abs : a:DM -> DM
  static member Acos : a:DM -> DM
  static member AddDiagonal : a:DM * b:DV -> DM
  static member AddItem : a:DM * i:int * j:int * b:D -> DM
  static member AddSubMatrix : a:DM * i:int * j:int * b:DM -> DM
  static member Asin : a:DM -> DM
  static member Atan : a:DM -> DM
  static member Atan2 : a:int * b:DM -> DM
  static member Atan2 : a:DM * b:int -> DM
  static member Atan2 : a:float * b:DM -> DM
  static member Atan2 : a:DM * b:float -> DM
  static member Atan2 : a:D * b:DM -> DM
  static member Atan2 : a:DM * b:D -> DM
  static member Atan2 : a:DM * b:DM -> DM
  static member Ceiling : a:DM -> DM
  static member Cos : a:DM -> DM
  static member Cosh : a:DM -> DM
  static member Det : a:DM -> D
  static member Diagonal : a:DM -> DV
  static member Exp : a:DM -> DM
  static member Floor : a:DM -> DM
  static member Inverse : a:DM -> DM
  static member Log : a:DM -> DM
  static member Log10 : a:DM -> DM
  static member Max : a:DM -> D
  static member Max : a:D * b:DM -> DM
  static member Max : a:DM * b:D -> DM
  static member Max : a:DM * b:DM -> DM
  static member MaxIndex : a:DM -> int * int
  static member Mean : a:DM -> D
  static member Min : a:DM -> D
  static member Min : a:D * b:DM -> DM
  static member Min : a:DM * b:D -> DM
  static member Min : a:DM * b:DM -> DM
  static member MinIndex : a:DM -> int * int
  static member Normalize : a:DM -> DM
  static member OfArray : m:int * a:D [] -> DM
  static member OfArray2D : a:D [,] -> DM
  static member OfCols : n:int * a:DV -> DM
  static member OfRows : s:seq<DV> -> DM
  static member OfRows : m:int * a:DV -> DM
  static member Op_DM_D : a:DM * ff:(float [,] -> float) * fd:(DM -> D) * df:(D * DM * DM -> D) * r:(DM -> TraceOp) -> D
  static member Op_DM_DM : a:DM * ff:(float [,] -> float [,]) * fd:(DM -> DM) * df:(DM * DM * DM -> DM) * r:(DM -> TraceOp) -> DM
  static member Op_DM_DM_DM : a:DM * b:DM * ff:(float [,] * float [,] -> float [,]) * fd:(DM * DM -> DM) * df_da:(DM * DM * DM -> DM) * df_db:(DM * DM * DM -> DM) * df_dab:(DM * DM * DM * DM * DM -> DM) * r_d_d:(DM * DM -> TraceOp) * r_d_c:(DM * DM -> TraceOp) * r_c_d:(DM * DM -> TraceOp) -> DM
  static member Op_DM_DV : a:DM * ff:(float [,] -> float []) * fd:(DM -> DV) * df:(DV * DM * DM -> DV) * r:(DM -> TraceOp) -> DV
  static member Op_DM_DV_DM : a:DM * b:DV * ff:(float [,] * float [] -> float [,]) * fd:(DM * DV -> DM) * df_da:(DM * DM * DM -> DM) * df_db:(DM * DV * DV -> DM) * df_dab:(DM * DM * DM * DV * DV -> DM) * r_d_d:(DM * DV -> TraceOp) * r_d_c:(DM * DV -> TraceOp) * r_c_d:(DM * DV -> TraceOp) -> DM
  static member Op_DM_DV_DV : a:DM * b:DV * ff:(float [,] * float [] -> float []) * fd:(DM * DV -> DV) * df_da:(DV * DM * DM -> DV) * df_db:(DV * DV * DV -> DV) * df_dab:(DV * DM * DM * DV * DV -> DV) * r_d_d:(DM * DV -> TraceOp) * r_d_c:(DM * DV -> TraceOp) * r_c_d:(DM * DV -> TraceOp) -> DV
  static member Op_DM_D_DM : a:DM * b:D * ff:(float [,] * float -> float [,]) * fd:(DM * D -> DM) * df_da:(DM * DM * DM -> DM) * df_db:(DM * D * D -> DM) * df_dab:(DM * DM * DM * D * D -> DM) * r_d_d:(DM * D -> TraceOp) * r_d_c:(DM * D -> TraceOp) * r_c_d:(DM * D -> TraceOp) -> DM
  static member Op_DV_DM_DM : a:DV * b:DM * ff:(float [] * float [,] -> float [,]) * fd:(DV * DM -> DM) * df_da:(DM * DV * DV -> DM) * df_db:(DM * DM * DM -> DM) * df_dab:(DM * DV * DV * DM * DM -> DM) * r_d_d:(DV * DM -> TraceOp) * r_d_c:(DV * DM -> TraceOp) * r_c_d:(DV * DM -> TraceOp) -> DM
  static member Op_DV_DM_DV : a:DV * b:DM * ff:(float [] * float [,] -> float []) * fd:(DV * DM -> DV) * df_da:(DV * DV * DV -> DV) * df_db:(DV * DM * DM -> DV) * df_dab:(DV * DV * DV * DM * DM -> DV) * r_d_d:(DV * DM -> TraceOp) * r_d_c:(DV * DM -> TraceOp) * r_c_d:(DV * DM -> TraceOp) -> DV
  static member Op_D_DM_DM : a:D * b:DM * ff:(float * float [,] -> float [,]) * fd:(D * DM -> DM) * df_da:(DM * D * D -> DM) * df_db:(DM * DM * DM -> DM) * df_dab:(DM * D * D * DM * DM -> DM) * r_d_d:(D * DM -> TraceOp) * r_d_c:(D * DM -> TraceOp) * r_c_d:(D * DM -> TraceOp) -> DM
  static member Pow : a:int * b:DM -> DM
  static member Pow : a:DM * b:int -> DM
  static member Pow : a:float * b:DM -> DM
  static member Pow : a:DM * b:float -> DM
  static member Pow : a:D * b:DM -> DM
  static member Pow : a:DM * b:D -> DM
  static member Pow : a:DM * b:DM -> DM
  static member ReLU : a:DM -> DM
  static member ReshapeToDV : a:DM -> DV
  static member Round : a:DM -> DM
  static member Sigmoid : a:DM -> DM
  static member Sign : a:DM -> DM
  static member Sin : a:DM -> DM
  static member Sinh : a:DM -> DM
  static member SoftPlus : a:DM -> DM
  static member SoftSign : a:DM -> DM
  static member Solve : a:DM * b:DV -> DV
  static member SolveSymmetric : a:DM * b:DV -> DV
  static member Sqrt : a:DM -> DM
  static member StandardDev : a:DM -> D
  static member Standardize : a:DM -> DM
  static member Sum : a:DM -> D
  static member Tan : a:DM -> DM
  static member Tanh : a:DM -> DM
  static member Trace : a:DM -> D
  static member Transpose : a:DM -> DM
  static member Variance : a:DM -> D
  static member ZeroMN : m:int -> n:int -> DM
  static member Zero : DM
  static member ( + ) : a:int * b:DM -> DM
  static member ( + ) : a:DM * b:int -> DM
  static member ( + ) : a:float * b:DM -> DM
  static member ( + ) : a:DM * b:float -> DM
  static member ( + ) : a:DM * b:DV -> DM
  static member ( + ) : a:DV * b:DM -> DM
  static member ( + ) : a:D * b:DM -> DM
  static member ( + ) : a:DM * b:D -> DM
  static member ( + ) : a:DM * b:DM -> DM
  static member ( / ) : a:int * b:DM -> DM
  static member ( / ) : a:DM * b:int -> DM
  static member ( / ) : a:float * b:DM -> DM
  static member ( / ) : a:DM * b:float -> DM
  static member ( / ) : a:D * b:DM -> DM
  static member ( / ) : a:DM * b:D -> DM
  static member ( ./ ) : a:DM * b:DM -> DM
  static member ( .* ) : a:DM * b:DM -> DM
  static member op_Explicit : d:float [,] -> DM
  static member op_Explicit : d:DM -> float [,]
  static member ( * ) : a:int * b:DM -> DM
  static member ( * ) : a:DM * b:int -> DM
  static member ( * ) : a:float * b:DM -> DM
  static member ( * ) : a:DM * b:float -> DM
  static member ( * ) : a:D * b:DM -> DM
  static member ( * ) : a:DM * b:D -> DM
  static member ( * ) : a:DV * b:DM -> DV
  static member ( * ) : a:DM * b:DV -> DV
  static member ( * ) : a:DM * b:DM -> DM
  static member ( - ) : a:int * b:DM -> DM
  static member ( - ) : a:DM * b:int -> DM
  static member ( - ) : a:float * b:DM -> DM
  static member ( - ) : a:DM * b:float -> DM
  static member ( - ) : a:D * b:DM -> DM
  static member ( - ) : a:DM * b:D -> DM
  static member ( - ) : a:DM * b:DM -> DM
  static member ( ~- ) : a:DM -> DM

Full name: DiffSharp.AD.Float64.DM
Layer.b: DV
Multiple items
union case DV.DV: float [] -> DV

--------------------
module DV

from DiffSharp.AD.Float64

--------------------
type DV =
  | DV of float []
  | DVF of DV * DV * uint32
  | DVR of DV * DV ref * TraceOp * uint32 ref * uint32
  member Copy : unit -> DV
  member GetForward : t:DV * i:uint32 -> DV
  member GetReverse : i:uint32 -> DV
  member GetSlice : lower:int option * upper:int option -> DV
  member ToArray : unit -> D []
  member ToColDM : unit -> DM
  member ToMathematicaString : unit -> string
  member ToMatlabString : unit -> string
  member ToRowDM : unit -> DM
  override ToString : unit -> string
  member Visualize : unit -> string
  member A : DV
  member F : uint32
  member Item : i:int -> D with get
  member Length : int
  member P : DV
  member PD : DV
  member T : DV
  member A : DV with set
  member F : uint32 with set
  static member Abs : a:DV -> DV
  static member Acos : a:DV -> DV
  static member AddItem : a:DV * i:int * b:D -> DV
  static member AddSubVector : a:DV * i:int * b:DV -> DV
  static member Append : a:DV * b:DV -> DV
  static member Asin : a:DV -> DV
  static member Atan : a:DV -> DV
  static member Atan2 : a:int * b:DV -> DV
  static member Atan2 : a:DV * b:int -> DV
  static member Atan2 : a:float * b:DV -> DV
  static member Atan2 : a:DV * b:float -> DV
  static member Atan2 : a:D * b:DV -> DV
  static member Atan2 : a:DV * b:D -> DV
  static member Atan2 : a:DV * b:DV -> DV
  static member Ceiling : a:DV -> DV
  static member Cos : a:DV -> DV
  static member Cosh : a:DV -> DV
  static member Exp : a:DV -> DV
  static member Floor : a:DV -> DV
  static member L1Norm : a:DV -> D
  static member L2Norm : a:DV -> D
  static member L2NormSq : a:DV -> D
  static member Log : a:DV -> DV
  static member Log10 : a:DV -> DV
  static member LogSumExp : a:DV -> D
  static member Max : a:DV -> D
  static member Max : a:D * b:DV -> DV
  static member Max : a:DV * b:D -> DV
  static member Max : a:DV * b:DV -> DV
  static member MaxIndex : a:DV -> int
  static member Mean : a:DV -> D
  static member Min : a:DV -> D
  static member Min : a:D * b:DV -> DV
  static member Min : a:DV * b:D -> DV
  static member Min : a:DV * b:DV -> DV
  static member MinIndex : a:DV -> int
  static member Normalize : a:DV -> DV
  static member OfArray : a:D [] -> DV
  static member Op_DV_D : a:DV * ff:(float [] -> float) * fd:(DV -> D) * df:(D * DV * DV -> D) * r:(DV -> TraceOp) -> D
  static member Op_DV_DM : a:DV * ff:(float [] -> float [,]) * fd:(DV -> DM) * df:(DM * DV * DV -> DM) * r:(DV -> TraceOp) -> DM
  static member Op_DV_DV : a:DV * ff:(float [] -> float []) * fd:(DV -> DV) * df:(DV * DV * DV -> DV) * r:(DV -> TraceOp) -> DV
  static member Op_DV_DV_D : a:DV * b:DV * ff:(float [] * float [] -> float) * fd:(DV * DV -> D) * df_da:(D * DV * DV -> D) * df_db:(D * DV * DV -> D) * df_dab:(D * DV * DV * DV * DV -> D) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> D
  static member Op_DV_DV_DM : a:DV * b:DV * ff:(float [] * float [] -> float [,]) * fd:(DV * DV -> DM) * df_da:(DM * DV * DV -> DM) * df_db:(DM * DV * DV -> DM) * df_dab:(DM * DV * DV * DV * DV -> DM) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> DM
  static member Op_DV_DV_DV : a:DV * b:DV * ff:(float [] * float [] -> float []) * fd:(DV * DV -> DV) * df_da:(DV * DV * DV -> DV) * df_db:(DV * DV * DV -> DV) * df_dab:(DV * DV * DV * DV * DV -> DV) * r_d_d:(DV * DV -> TraceOp) * r_d_c:(DV * DV -> TraceOp) * r_c_d:(DV * DV -> TraceOp) -> DV
  static member Op_DV_D_DV : a:DV * b:D * ff:(float [] * float -> float []) * fd:(DV * D -> DV) * df_da:(DV * DV * DV -> DV) * df_db:(DV * D * D -> DV) * df_dab:(DV * DV * DV * D * D -> DV) * r_d_d:(DV * D -> TraceOp) * r_d_c:(DV * D -> TraceOp) * r_c_d:(DV * D -> TraceOp) -> DV
  static member Op_D_DV_DV : a:D * b:DV * ff:(float * float [] -> float []) * fd:(D * DV -> DV) * df_da:(DV * D * D -> DV) * df_db:(DV * DV * DV -> DV) * df_dab:(DV * D * D * DV * DV -> DV) * r_d_d:(D * DV -> TraceOp) * r_d_c:(D * DV -> TraceOp) * r_c_d:(D * DV -> TraceOp) -> DV
  static member Pow : a:int * b:DV -> DV
  static member Pow : a:DV * b:int -> DV
  static member Pow : a:float * b:DV -> DV
  static member Pow : a:DV * b:float -> DV
  static member Pow : a:D * b:DV -> DV
  static member Pow : a:DV * b:D -> DV
  static member Pow : a:DV * b:DV -> DV
  static member ReLU : a:DV -> DV
  static member ReshapeToDM : m:int * a:DV -> DM
  static member Round : a:DV -> DV
  static member Sigmoid : a:DV -> DV
  static member Sign : a:DV -> DV
  static member Sin : a:DV -> DV
  static member Sinh : a:DV -> DV
  static member SoftMax : a:DV -> DV
  static member SoftPlus : a:DV -> DV
  static member SoftSign : a:DV -> DV
  static member Split : d:DV * n:seq<int> -> seq<DV>
  static member Sqrt : a:DV -> DV
  static member StandardDev : a:DV -> D
  static member Standardize : a:DV -> DV
  static member Sum : a:DV -> D
  static member Tan : a:DV -> DV
  static member Tanh : a:DV -> DV
  static member Variance : a:DV -> D
  static member ZeroN : n:int -> DV
  static member Zero : DV
  static member ( + ) : a:int * b:DV -> DV
  static member ( + ) : a:DV * b:int -> DV
  static member ( + ) : a:float * b:DV -> DV
  static member ( + ) : a:DV * b:float -> DV
  static member ( + ) : a:D * b:DV -> DV
  static member ( + ) : a:DV * b:D -> DV
  static member ( + ) : a:DV * b:DV -> DV
  static member ( &* ) : a:DV * b:DV -> DM
  static member ( / ) : a:int * b:DV -> DV
  static member ( / ) : a:DV * b:int -> DV
  static member ( / ) : a:float * b:DV -> DV
  static member ( / ) : a:DV * b:float -> DV
  static member ( / ) : a:D * b:DV -> DV
  static member ( / ) : a:DV * b:D -> DV
  static member ( ./ ) : a:DV * b:DV -> DV
  static member ( .* ) : a:DV * b:DV -> DV
  static member op_Explicit : d:float [] -> DV
  static member op_Explicit : d:DV -> float []
  static member ( * ) : a:int * b:DV -> DV
  static member ( * ) : a:DV * b:int -> DV
  static member ( * ) : a:float * b:DV -> DV
  static member ( * ) : a:DV * b:float -> DV
  static member ( * ) : a:D * b:DV -> DV
  static member ( * ) : a:DV * b:D -> DV
  static member ( * ) : a:DV * b:DV -> D
  static member ( - ) : a:int * b:DV -> DV
  static member ( - ) : a:DV * b:int -> DV
  static member ( - ) : a:float * b:DV -> DV
  static member ( - ) : a:DV * b:float -> DV
  static member ( - ) : a:D * b:DV -> DV
  static member ( - ) : a:DV * b:D -> DV
  static member ( - ) : a:DV * b:DV -> DV
  static member ( ~- ) : a:DV -> DV

Full name: DiffSharp.AD.Float64.DV
Layer.a: DV -> DV
type Network =
  {layers: Layer [];}

Full name: Examples-neuralnetworks.Network
Network.layers: Layer []
val runLayer : x:DV -> l:Layer -> DV

Full name: Examples-neuralnetworks.runLayer
val x : DV
val l : Layer
val runNetwork : x:DV -> n:Network -> DV

Full name: Examples-neuralnetworks.runNetwork
val n : Network
Multiple items
module Array

from DiffSharp.Util

--------------------
module Array

from Microsoft.FSharp.Collections
val fold : folder:('State -> 'T -> 'State) -> state:'State -> array:'T [] -> 'State

Full name: Microsoft.FSharp.Collections.Array.fold
val rnd : System.Random

Full name: Examples-neuralnetworks.rnd
namespace System
Multiple items
type Random =
  new : unit -> Random + 1 overload
  member Next : unit -> int + 2 overloads
  member NextBytes : buffer:byte[] -> unit
  member NextDouble : unit -> float

Full name: System.Random

--------------------
System.Random() : unit
System.Random(Seed: int) : unit
val createNetwork : l:int [] -> Network

Full name: Examples-neuralnetworks.createNetwork
val l : int []
Multiple items
val int : value:'T -> int (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.int

--------------------
type int = int32

Full name: Microsoft.FSharp.Core.int

--------------------
type int<'Measure> = int

Full name: Microsoft.FSharp.Core.int<_>
val init : count:int -> initializer:(int -> 'T) -> 'T []

Full name: Microsoft.FSharp.Collections.Array.init
property System.Array.Length: int
val i : int
val init : m:int -> n:int -> f:(int -> int -> 'a) -> DM

Full name: DiffSharp.AD.Float64.DM.init
System.Random.NextDouble() : float
val init : n:int -> f:(int -> 'a) -> DV

Full name: DiffSharp.AD.Float64.DV.init
val sigmoid : x:'a -> 'a (requires member Sigmoid)

Full name: DiffSharp.Util.sigmoid
val net1 : Network

Full name: Examples-neuralnetworks.net1
val backprop : n:Network -> eta:float -> epochs:int -> x:DV [] -> y:DV [] -> seq<float>

Full name: Examples-neuralnetworks.backprop
val eta : float
val epochs : int
val x : DV []
val y : DV []
val i : uint32
Multiple items
type GlobalTagger =
  new : unit -> GlobalTagger
  static member Next : uint32
  static member Reset : unit

Full name: DiffSharp.Util.GlobalTagger

--------------------
new : unit -> GlobalTagger
property GlobalTagger.Next: uint32
Multiple items
val seq : sequence:seq<'T> -> seq<'T>

Full name: Microsoft.FSharp.Core.Operators.seq

--------------------
type seq<'T> = System.Collections.Generic.IEnumerable<'T>

Full name: Microsoft.FSharp.Collections.seq<_>
val j : int
val makeReverse : i:uint32 -> p:'a -> 'a (requires member GetReverse)

Full name: DiffSharp.AD.Float64.DOps.makeReverse
val L : D
val map2 : mapping:('T1 -> 'T2 -> 'U) -> array1:'T1 [] -> array2:'T2 [] -> 'U []

Full name: Microsoft.FSharp.Collections.Array.map2
val y : DV
val l2normSq : v:DV -> D

Full name: DiffSharp.AD.Float64.DV.l2normSq
val sum : array:'T [] -> 'T (requires member ( + ) and member get_Zero)

Full name: Microsoft.FSharp.Collections.Array.sum
val reverseProp : v:obj -> d:obj -> unit

Full name: DiffSharp.AD.Float64.DOps.reverseProp
union case D.D: float -> D
val primal : d:'a -> 'a (requires member get_P)

Full name: DiffSharp.AD.Float64.DOps.primal
property DM.P: DM
property DM.A: DM
property DV.P: DV
property DV.A: DV
val printfn : format:Printf.TextWriterFormat<'T> -> 'T

Full name: Microsoft.FSharp.Core.ExtraTopLevelOperators.printfn
Multiple items
val float : value:'T -> float (requires member op_Explicit)

Full name: Microsoft.FSharp.Core.Operators.float

--------------------
type float = System.Double

Full name: Microsoft.FSharp.Core.float

--------------------
type float<'Measure> = float

Full name: Microsoft.FSharp.Core.float<_>
namespace FSharp
namespace FSharp.Charting
val ORx : DV []

Full name: Examples-neuralnetworks.ORx
val toDV : v:seq<'a> -> DV (requires member op_Explicit)

Full name: DiffSharp.AD.Float64.DOps.toDV
val ORy : DV []

Full name: Examples-neuralnetworks.ORy
val net2 : Network

Full name: Examples-neuralnetworks.net2
val train2 : seq<float>

Full name: Examples-neuralnetworks.train2
type Chart =
  static member Area : data:seq<#value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> GenericChart
  static member Area : data:seq<#key * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> GenericChart
  static member Bar : data:seq<#value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> GenericChart
  static member Bar : data:seq<#key * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> GenericChart
  static member BoxPlotFromData : data:seq<#key * #seq<'a2>> * ?Name:string * ?Title:string * ?Color:Color * ?XTitle:string * ?YTitle:string * ?Percentile:int * ?ShowAverage:bool * ?ShowMedian:bool * ?ShowUnusualValues:bool * ?WhiskerPercentile:int -> GenericChart (requires 'a2 :> value)
  static member BoxPlotFromStatistics : data:seq<#key * #value * #value * #value * #value * #value * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string * ?Percentile:int * ?ShowAverage:bool * ?ShowMedian:bool * ?ShowUnusualValues:bool * ?WhiskerPercentile:int -> GenericChart
  static member Bubble : data:seq<#value * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string * ?BubbleMaxSize:int * ?BubbleMinSize:int * ?BubbleScaleMax:float * ?BubbleScaleMin:float * ?UseSizeForLabel:bool -> GenericChart
  static member Bubble : data:seq<#key * #value * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string * ?BubbleMaxSize:int * ?BubbleMinSize:int * ?BubbleScaleMax:float * ?BubbleScaleMin:float * ?UseSizeForLabel:bool -> GenericChart
  static member Candlestick : data:seq<#value * #value * #value * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> CandlestickChart
  static member Candlestick : data:seq<#key * #value * #value * #value * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:Color * ?XTitle:string * ?YTitle:string -> CandlestickChart
  ...

Full name: FSharp.Charting.Chart
static member Chart.Line : data:seq<#value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:System.Drawing.Color * ?XTitle:string * ?YTitle:string -> ChartTypes.GenericChart
static member Chart.Line : data:seq<#key * #value> * ?Name:string * ?Title:string * ?Labels:#seq<string> * ?Color:System.Drawing.Color * ?XTitle:string * ?YTitle:string -> ChartTypes.GenericChart
val XORx : DV []

Full name: Examples-neuralnetworks.XORx
val XORy : DV []

Full name: Examples-neuralnetworks.XORy
val net3 : Network

Full name: Examples-neuralnetworks.net3
val train3 : seq<float>

Full name: Examples-neuralnetworks.train3
type Layer' =
  {mutable W: DM;
   mutable b: DV;
   a: DM -> DM;}

Full name: Examples-neuralnetworks.Layer'
Layer'.W: DM
Layer'.b: DV
Layer'.a: DM -> DM
type Network' =
  {layers: Layer' [];}

Full name: Examples-neuralnetworks.Network'
Network'.layers: Layer' []
val runLayer' : x:DM -> l:Layer' -> DM

Full name: Examples-neuralnetworks.runLayer'
val x : DM
val l : Layer'
val createCols : n:int -> v:DV -> DM

Full name: DiffSharp.AD.Float64.DM.createCols
property DM.Cols: int
val runNetwork' : x:DM -> n:Network' -> DM

Full name: Examples-neuralnetworks.runNetwork'
val n : Network'
val backprop' : n:Network' -> eta:float -> epochs:int -> mbsize:int -> loss:(DM -> DM -> D) -> x:DM -> y:DM -> unit

Full name: Examples-neuralnetworks.backprop'
val mbsize : int
val loss : (DM -> DM -> D)
val y : DM
val mutable b : int
val batches : int
val mutable j : int
val mbX : DM
val mbY : DM
type D =
  | D of float
  | DF of D * D * uint32
  | DR of D * D ref * TraceOp * uint32 ref * uint32
  interface IComparable
  member Copy : unit -> D
  override Equals : other:obj -> bool
  member GetForward : t:D * i:uint32 -> D
  override GetHashCode : unit -> int
  member GetReverse : i:uint32 -> D
  override ToString : unit -> string
  member A : D
  member F : uint32
  member P : D
  member PD : D
  member T : D
  member A : D with set
  member F : uint32 with set
  static member Abs : a:D -> D
  static member Acos : a:D -> D
  static member Asin : a:D -> D
  static member Atan : a:D -> D
  static member Atan2 : a:int * b:D -> D
  static member Atan2 : a:D * b:int -> D
  static member Atan2 : a:float * b:D -> D
  static member Atan2 : a:D * b:float -> D
  static member Atan2 : a:D * b:D -> D
  static member Ceiling : a:D -> D
  static member Cos : a:D -> D
  static member Cosh : a:D -> D
  static member Exp : a:D -> D
  static member Floor : a:D -> D
  static member Log : a:D -> D
  static member Log10 : a:D -> D
  static member LogSumExp : a:D -> D
  static member Max : a:D * b:D -> D
  static member Min : a:D * b:D -> D
  static member Op_D_D : a:D * ff:(float -> float) * fd:(D -> D) * df:(D * D * D -> D) * r:(D -> TraceOp) -> D
  static member Op_D_D_D : a:D * b:D * ff:(float * float -> float) * fd:(D * D -> D) * df_da:(D * D * D -> D) * df_db:(D * D * D -> D) * df_dab:(D * D * D * D * D -> D) * r_d_d:(D * D -> TraceOp) * r_d_c:(D * D -> TraceOp) * r_c_d:(D * D -> TraceOp) -> D
  static member Pow : a:int * b:D -> D
  static member Pow : a:D * b:int -> D
  static member Pow : a:float * b:D -> D
  static member Pow : a:D * b:float -> D
  static member Pow : a:D * b:D -> D
  static member ReLU : a:D -> D
  static member Round : a:D -> D
  static member Sigmoid : a:D -> D
  static member Sign : a:D -> D
  static member Sin : a:D -> D
  static member Sinh : a:D -> D
  static member SoftPlus : a:D -> D
  static member SoftSign : a:D -> D
  static member Sqrt : a:D -> D
  static member Tan : a:D -> D
  static member Tanh : a:D -> D
  static member One : D
  static member Zero : D
  static member ( + ) : a:int * b:D -> D
  static member ( + ) : a:D * b:int -> D
  static member ( + ) : a:float * b:D -> D
  static member ( + ) : a:D * b:float -> D
  static member ( + ) : a:D * b:D -> D
  static member ( / ) : a:int * b:D -> D
  static member ( / ) : a:D * b:int -> D
  static member ( / ) : a:float * b:D -> D
  static member ( / ) : a:D * b:float -> D
  static member ( / ) : a:D * b:D -> D
  static member op_Explicit : d:D -> float
  static member ( * ) : a:int * b:D -> D
  static member ( * ) : a:D * b:int -> D
  static member ( * ) : a:float * b:D -> D
  static member ( * ) : a:D * b:float -> D
  static member ( * ) : a:D * b:D -> D
  static member ( - ) : a:int * b:D -> D
  static member ( - ) : a:D * b:int -> D
  static member ( - ) : a:float * b:D -> D
  static member ( - ) : a:D * b:float -> D
  static member ( - ) : a:D * b:D -> D
  static member ( ~- ) : a:D -> D

Full name: DiffSharp.AD.Float64.D
namespace System.IO
type MNIST =
  static member Load : filename:string -> DM
  static member Load : filename:string * items:int -> DM

Full name: Examples-neuralnetworks.MNIST
static member MNIST.Load : filename:string * items:int -> DM

Full name: Examples-neuralnetworks.MNIST.Load
val filename : string
val items : int
val d : BinaryReader
Multiple items
type BinaryReader =
  new : input:Stream -> BinaryReader + 1 overload
  member BaseStream : Stream
  member Close : unit -> unit
  member Dispose : unit -> unit
  member PeekChar : unit -> int
  member Read : unit -> int + 2 overloads
  member ReadBoolean : unit -> bool
  member ReadByte : unit -> byte
  member ReadBytes : count:int -> byte[]
  member ReadChar : unit -> char
  ...

Full name: System.IO.BinaryReader

--------------------
BinaryReader(input: Stream) : unit
BinaryReader(input: Stream, encoding: System.Text.Encoding) : unit
type File =
  static member AppendAllLines : path:string * contents:IEnumerable<string> -> unit + 1 overload
  static member AppendAllText : path:string * contents:string -> unit + 1 overload
  static member AppendText : path:string -> StreamWriter
  static member Copy : sourceFileName:string * destFileName:string -> unit + 1 overload
  static member Create : path:string -> FileStream + 3 overloads
  static member CreateText : path:string -> StreamWriter
  static member Decrypt : path:string -> unit
  static member Delete : path:string -> unit
  static member Encrypt : path:string -> unit
  static member Exists : path:string -> bool
  ...

Full name: System.IO.File
File.Open(path: string, mode: FileMode) : FileStream
File.Open(path: string, mode: FileMode, access: FileAccess) : FileStream
File.Open(path: string, mode: FileMode, access: FileAccess, share: FileShare) : FileStream
type FileMode =
  | CreateNew = 1
  | Create = 2
  | Open = 3
  | OpenOrCreate = 4
  | Truncate = 5
  | Append = 6

Full name: System.IO.FileMode
field FileMode.Open = 3
type FileAccess =
  | Read = 1
  | Write = 2
  | ReadWrite = 3

Full name: System.IO.FileAccess
field FileAccess.Read = 1
type FileShare =
  | None = 0
  | Read = 1
  | Write = 2
  | ReadWrite = 3
  | Delete = 4
  | Inheritable = 16

Full name: System.IO.FileShare
field FileShare.Read = 1
val magicnumber : int
BinaryReader.ReadInt32() : int
namespace System.Net
Multiple items
type IPAddress =
  new : newAddress:int64 -> IPAddress + 2 overloads
  member Address : int64 with get, set
  member AddressFamily : AddressFamily
  member Equals : comparand:obj -> bool
  member GetAddressBytes : unit -> byte[]
  member GetHashCode : unit -> int
  member IsIPv6LinkLocal : bool
  member IsIPv6Multicast : bool
  member IsIPv6SiteLocal : bool
  member IsIPv6Teredo : bool
  ...

Full name: System.Net.IPAddress

--------------------
System.Net.IPAddress(newAddress: int64) : unit
System.Net.IPAddress(address: byte []) : unit
System.Net.IPAddress(address: byte [], scopeid: int64) : unit
System.Net.IPAddress.NetworkToHostOrder(network: int16) : int16
System.Net.IPAddress.NetworkToHostOrder(network: int) : int
System.Net.IPAddress.NetworkToHostOrder(network: int64) : int64
val maxitems : int
BinaryReader.ReadBytes(count: int) : byte []
val min : e1:'T -> e2:'T -> 'T (requires comparison)

Full name: Microsoft.FSharp.Core.Operators.min
val map : mapping:('T -> 'U) -> array:'T [] -> 'U []

Full name: Microsoft.FSharp.Collections.Array.map
val ofDV : m:int -> v:DV -> DM

Full name: DiffSharp.AD.Float64.DM.ofDV
val rows : int
val cols : int
val n : int
val transpose : m:DM -> DM

Full name: DiffSharp.AD.Float64.DM.transpose
val failwith : message:string -> 'T

Full name: Microsoft.FSharp.Core.Operators.failwith
static member MNIST.Load : filename:string -> DM

Full name: Examples-neuralnetworks.MNIST.Load
static member MNIST.Load : filename:string -> DM
static member MNIST.Load : filename:string * items:int -> DM
type Int32 =
  struct
    member CompareTo : value:obj -> int + 1 overload
    member Equals : obj:obj -> bool + 1 overload
    member GetHashCode : unit -> int
    member GetTypeCode : unit -> TypeCode
    member ToString : unit -> string + 3 overloads
    static val MaxValue : int
    static val MinValue : int
    static member Parse : s:string -> int + 3 overloads
    static member TryParse : s:string * result:int -> bool + 1 overload
  end

Full name: System.Int32
field int.MaxValue = 2147483647
val mnistTrainX : DM

Full name: Examples-neuralnetworks.mnistTrainX
val mnistTrainY : DM

Full name: Examples-neuralnetworks.mnistTrainY
val l0 : Layer'

Full name: Examples-neuralnetworks.l0
val zeroCreate : n:int -> DV

Full name: DiffSharp.AD.Float64.DV.zeroCreate
val tanh : value:'T -> 'T (requires member Tanh)

Full name: Microsoft.FSharp.Core.Operators.tanh
val l1 : Layer'

Full name: Examples-neuralnetworks.l1
val mapCols : f:(DV -> DV) -> m:DM -> DM

Full name: DiffSharp.AD.Float64.DM.mapCols
val softmax : x:'a -> 'a (requires member SoftMax)

Full name: DiffSharp.Util.softmax
val nn : Network'

Full name: Examples-neuralnetworks.nn
val crossEntropy : x:DM -> y:DM -> D

Full name: Examples-neuralnetworks.crossEntropy
val toCols : m:DM -> seq<DV>

Full name: DiffSharp.AD.Float64.DM.toCols
module Seq

from Microsoft.FSharp.Collections
val mapi : mapping:(int -> 'T -> 'U) -> source:seq<'T> -> seq<'U>

Full name: Microsoft.FSharp.Collections.Seq.mapi
val v : DV
val standardBasis : n:int -> i:int -> DV

Full name: DiffSharp.AD.Float64.DV.standardBasis
property DV.Length: int
val log : value:'T -> 'T (requires member Log)

Full name: Microsoft.FSharp.Core.Operators.log
val sum : source:seq<'T> -> 'T (requires member ( + ) and member get_Zero)

Full name: Microsoft.FSharp.Collections.Seq.sum
val mnistTestX : DM

Full name: Examples-neuralnetworks.mnistTestX
val mnistTestY : DM

Full name: Examples-neuralnetworks.mnistTestY
val testY : DM

Full name: Examples-neuralnetworks.testY
val testLoss : D

Full name: Examples-neuralnetworks.testLoss
val testPredict : int []

Full name: Examples-neuralnetworks.testPredict
val map : mapping:('T -> 'U) -> source:seq<'T> -> seq<'U>

Full name: Microsoft.FSharp.Collections.Seq.map
val maxIndex : v:DV -> int

Full name: DiffSharp.AD.Float64.DV.maxIndex
val toArray : source:seq<'T> -> 'T []

Full name: Microsoft.FSharp.Collections.Seq.toArray
val testTrue : DM

Full name: Examples-neuralnetworks.testTrue