cntk.layers.typing module

The CNTK typing module contains basic CNTK type meta-classes for update_signature() and type signatures for the CNTK Function decorator.

The type of a CNTK Variable is defined by five properties: shape, dynamic_axes, is_sparse, dtype, and needs_gradient. Some API functions accept these variables as independent arguments, e.g. input. The typing module provides a Pythonic way to represent the variable type properties as a single data object.

Python type syntax can be used to create such a record for the three main properties, shape, dynamic_axes, and is_sparse, using Tensor, SparseTensor, ParameterTensor, Sequence, and SequenceOver.

Note: This new type system may undergo changes. Please give us feedback on github or stackoverflow

Example

>>> # Tensor[...] denotes a data variable (with implied batch dimension)
>>> from cntk.layers.typing import *
>>> tp = Tensor[13,42]
>>> tp.shape
(13, 42)
>>> tp.is_sparse
False
>>> [str(axis.name) for axis in tp.dynamic_axes]
['defaultBatchAxis']

>>> # SparseTensor[...] is a sparse Tensor
>>> tp = SparseTensor[9000]
>>> tp.is_sparse
True

>>> # if the first argument is np.float32 or np.float64 or np.float16, then this sets the dtype
>>> tp = Tensor[np.float32,13,42]
>>> print(tp.dtype == np.float32)
True

This record can be directly passed to update_signature().

Example

>>> from cntk.layers import *
>>> f = Dense(500)
>>> f.update_signature(Tensor[13,42])
>>> f.shape
(500,)

>>> # This is just the same as saying
>>> f = Dense(500)
>>> _ = f.replace_placeholders({f.arguments[0]: C.input_variable(shape=(13,42), dynamic_axes=[Axis.default_batch_axis()])})
>>> f.shape
(500,)

Scalars can be just specified as float or np.float32 or np.float64 or np.float16.

Example

>>> @Function
... def f(x):
...    return x+1
>>> f.update_signature(np.float64)
>>> print(f.dtype == np.float64)
True

To specify types with a dynamic axis, use Sequence[].

Example

>>> tp = Sequence[SparseTensor[9000]]
>>> [str(axis.name) for axis in tp.dynamic_axes]
['defaultBatchAxis', 'defaultDynamicAxis']

This will refer to the default dynamic axis. If your model uses multiple dynamic axes, such as a sequence-to-sequence model, you use SequenceOver[] to define your own sequence type for each.

Example

>>> InputSequence = SequenceOver[Axis('input')]
>>> tp = InputSequence[SparseTensor[9000]]
>>> [str(axis.name) for axis in tp.dynamic_axes]
['defaultBatchAxis', 'input']

The typing syntax can be used to directly define CNTK functions with their input types. This is often done for the criterion function.

Example

>>> from cntk import debugging, cross_entropy_with_softmax
>>> model = Sequential([Embedding(300), Fold(GRU(128)), Dense(10)])
>>> print(model)
Composite(keep: Sequence[tensor]) -> Sequence[tensor]
>>> inputAxis = Axis('inputAxis')
>>> @Function
... @Signature(input=SequenceOver[inputAxis][Tensor[128]], label=Tensor[10])
... def criterion(input, label):
...     output = model(input)
...     return cross_entropy_with_softmax(output, label)
>>> print(criterion)
Composite(input: SequenceOver[inputAxis][Tensor[128]], label: Tensor[10]) -> Tensor[1]

The following lists a few common errors with CNTK type objects:

Example

>>> # types are abstract, they cannot be instantiated directly
>>> from cntk.layers.typing import Tensor
>>> try:
...     inp = Tensor[32]()   # attempt to create an instance of type Tensor[32]
... except TypeError as e:
...     print('ERROR: ' + str(e))
ERROR: abstract type Tensor[32] cannot be instantiated; use 'input_variable(**Tensor[32])' instead

>>> # types are not inputs
>>> try:
...     inp = Tensor[32]
...     y = sigmoid(inp)
... except ValueError as e:
...     print('ERROR: ' + str(e))
ERROR: Input is a type object (Tensor[32]). Did you mean to pass 'input_variable(**Tensor[32])'?

>>> # nested sequences are currently not supported
>>> try:
...     t = Sequence[Sequence[Tensor[32]]]
... except TypeError as e:
...     print('ERROR: ' + str(e))
ERROR: sequences over sequences are currently not supported

>>> # a function with specified type gets passed a differently-shaped input
>>> @Function
... @Signature(x=Tensor[13])
... def f(x):
...    return sigmoid(x)
>>> try:
...     x = C.input_variable((42,))
...     y = f(x)
... except TypeError as e:
...     print('ERROR: ' + str(e))
ERROR: argument x's type Tensor[13] is incompatible with the type Tensor[42] of the passed Variable

Using Python type syntax, besides being more concise and easier to memorize, has the added benefit of beign able to more easily talk about types of CNTK objects, very similar to how one would talk about the types of Python objects (e.g. List[Tuple[int,float]]). This is particularly beneficial for the functional-programming style of the Layers library, where functions are also reasoned about by their types. In functional programming, it has been observed that getting the types of functions right is a critical step towards correct code.

Note that the type syntax does not allow to specify the special-purpose type property needs_gradient, nor to dtype which instead should be specified as a global setting. If these properties are needed on a type object, please use construct an input using input_var() and get its type property.

class ParameterTensor

Bases: object

Meta class to denote a parameter tensor (no batch axis). Use with dimensions, e.g. ParameterTensor[512,256].

class Sequence

Bases: object

Meta-meta class to denote a sequence of data tensors. Example: Sequence[Tensor[13,42]]

class SequenceOver

Bases: object

Meta-meta-meta class to denote a sequence of data tensors over a custom axis. Example: userAxis = Axis(); SequenceOver[userAxis][Tensor[13,42]]

class SequenceOverMeta

Bases: type

Signature(*args, **kwargs)

@Signature is a decorator to implement the function-argument annotations in Python-2.7, as needed by the @Function decorator. This is only needed when you have not yet migrated to Python 3.x.

Note: Although this is aimed at enabling @Function syntax with type annotations in Python 2.7, @Signature is independent of CNTK and can be used for any argument annotation.

Parameters:

• *args – types of arguments of the function that this decorator is applied to, in the same order.
• **kwargs – types of arguments with optional names, e.g. x=Tensor[42]. Use this second form for longer argument lists.

Example:

# Python 3:
@Function
def f(x: Tensor[42]):
    return sigmoid(x)

# Python 2.7:
@Function
@Signature(Tensor[42])
def f(x):
    return sigmoid(x)

# note that this:
@Function
@Signature(x:int)
def sqr(x):
    return x*x
# is identical to:
def sqr(x):
    return x*x
sqr.__annotations__ = {'x': int}

class SparseTensor

Bases: object

Meta class to denote a sparse data tensor (with batch axis). Use with dimensions, e.g. SparseTensor[129].

class Tensor

Bases: object

Meta class to denote a data tensor (with batch axis). Use with dimensions, e.g. Tensor[13,42].