cntk.debugging.debug module

In order to debug a graph one simply needs to wrap the root node as follows:

# ... setting up the model in z
from cntk.debugging import debug_model
z = debug_model(z)

Then, when z is evaluated or trained (i.e. when either forward() or backward() is called, you will see the following command-line interface:

=================================== forward  ===================================
Parameter node with uid='Parameter28' shape=[](2,)
[CNTK forward] >>> help
    Commands:
    n - execute the next node
    n <number> - execute the next <number> nodes

    u f - execute until forward pass (like 'n' when already in forward pass)
    u b - execute until backward pass (like 'n' when already in backward pass)
    u name - execute until a node with that name is hit
    u <lambda> - execute until the lambda expression is True. Examples:
                 Until a Times node is hit:
                     lambda arg, node: node.op_name == 'Times'
                 Until a node is hit that has 3 dimensions:
                     lambda arg, node: len(node.shape) == 3
                 Until the variance of the input exceeds 1 (np = numpy):
                     lambda arg, node: np.var(arg) > 1

    c - execute until end
    p - print input (forward) or root gradients (backward)
    d - drop into a pdb shell
    q - quit

[CNTK backward] >>> n

Times node with uid='Times29' shape=[*,*](2,)
[CNTK forward] >>> n
=================================== backward ===================================
Times node with uid='Times29' shape=[*,*](2,)
[CNTK backward] >>> p
State: None
Root gradients:
[[[-0.79412955  0.79412955]]
 [[-0.79412955  0.79412955]]
 [[ 0.20587046 -0.20587045]]
 [[ 0.20587046 -0.20587045]]
 [[ 0.20587046 -0.20587045]]
 [[ 0.20587046 -0.20587045]]
 [[-0.79412955  0.79412955]]
 [[ 0.20587046 -0.20587045]]
 [[ 0.20587039 -0.20587039]]
 [[-0.79412961  0.79412961]]]
At every stop the following information is given:
  • Forward or backward pass
  • Node type (e.g. ‘Times’)
  • Name if given, otherwise it is omitted
  • uid, which is a unique reference within the graph
  • shape having the format [dynamic axis](static axes). E.g. [*,*](2,) means that the node’s output has two dynamic axes (batch and sequence) and one static axis (2 dimensions)
debug_model(model, in_stream=sys.stdin, out_stream=sys.stdout, exit_func=sys.exit)[source]

Returns a cloned model that has debug nodes inserted everywhere. When the graph is evaluated or trained, those nodes will allow to inspect the graph.

Parameters:
  • model (root node) – root node until which the nodes are to be debugged
  • in_stream (object behaving like sys.stdin, default stdin) – readline() will be called on it to obtain user input
  • out_stream (object behaving like sys.stdout, default stdout) – write() and flush() will be called on it to output debug info to the user
  • exit_func (callable, default sys.exit) – callable that takes an exit code and is called, when the user exits the debugging process
Returns:

a clone of the model that has debugging enabled

save_as_legacy_model(root_op, filename)[source]

Save the network of root_op in filename. For debugging purposes only, very likely to be deprecated in the future.

Parameters:
  • root_op (Function) – op of the graph to save
  • filename (str) – filename to store the model in.
set_checked_mode(enable)[source]
Checked mode enables additional runtime verification such as:
  • Tracking NaN occurrences in sequence gaps.
  • Function graph verification after binding of free static axes to actual values at runtime

Enabling checked mode incurs additional runtime costs and is meant to be used as a debugging aid.

Parameters:enable (bool) – whether to enable checked mode (with performance impact)
set_computation_network_trace_level(level)[source]

Set trace level to the computation network. Currently supported values:

0
turn off trace
1
output nodes’ dimensions and some other static info
1000
output each node’s abs sum of elements in its value matrix for every forward/backward
1000000
output each node’s full matrix for every forward/backward
Parameters:level (int) – trace level
set_node_timing(enable)[source]

Node-timing records per-node average execution time per-minibatch. Enabling checked mode incurs a little runtime costs and is meant to be used as a debugging aid.

Parameters:enable (bool) – whether to enable per-node timing