modeci_mdf.functions.onnx.gru

modeci_mdf.functions.onnx.gru(*args, **kwargs)

Computes an one-layer GRU. This operator is usually supported via some custom implementation such as CuDNN.

Notations:

  • X - input tensor

  • z - update gate

  • r - reset gate

  • h - hidden gate

  • t - time step (t-1 means previous time step)

  • W[zrh] - W parameter weight matrix for update, reset, and hidden gates

  • R[zrh] - R recurrence weight matrix for update, reset, and hidden gates

  • Wb[zrh] - W bias vectors for update, reset, and hidden gates

  • Rb[zrh] - R bias vectors for update, reset, and hidden gates

  • WB[zrh] - W parameter weight matrix for backward update, reset, and hidden gates

  • RB[zrh] - R recurrence weight matrix for backward update, reset, and hidden gates

  • WBb[zrh] - W bias vectors for backward update, reset, and hidden gates

  • RBb[zrh] - R bias vectors for backward update, reset, and hidden gates

  • H - Hidden state

  • num_directions - 2 if direction == bidirectional else 1

Activation functions:

  • Relu(x) - max(0, x)

  • Tanh(x) - (1 - e^{-2x})/(1 + e^{-2x})

  • Sigmoid(x) - 1/(1 + e^{-x})

Note

Below are optional

  • Affine(x) - alpha * x + beta

  • LeakyRelu(x) - x if x >= 0 else alpha * x

  • ThresholdedRelu(x) - x if x >= alpha else 0

  • ScaledTanh(x) - alpha * Tanh(beta * x)

  • HardSigmoid(x) - min(max(alpha * x + beta, 0), 1)

  • Elu(x) - x if x >= 0 else alpha * (e^x - 1)

  • Softsign(x) - x/(1 + |x|)

  • Softplus(x) - log(1 + e^x)

Equations (Default: f=Sigmoid, g=Tanh):

  • zt = f(Xt*(Wz^T) + Ht-1*(Rz^T) + Wbz + Rbz)

  • rt = f(Xt*(Wr^T) + Ht-1*(Rr^T) + Wbr + Rbr)

  • ht = g(Xt*(Wh^T) + (rt (.) Ht-1)*(Rh^T) + Rbh + Wbh) # default, when linear_before_reset = 0

  • ht = g(Xt*(Wh^T) + (rt (.) (Ht-1*(Rh^T) + Rbh)) + Wbh) # when linear_before_reset != 0

  • Ht = (1 - zt) (.) ht + zt (.) Ht-1

This operator has optional inputs/outputs. See [the doc](IR.md) for more details about the representation of optional arguments. An empty string may be used in the place of an actual argument’s name to indicate a missing argument. Trailing optional arguments (those not followed by an argument that is present) may also be simply omitted.