cntk.debugging.debug module¶
In order to debug a graph one simply needs to wrap the root node as follows:
# ... setting up the model in z
from cntk.debugging import debug_model
z = debug_model(z)
Then, when z
is evaluated or trained (i.e. when either
forward()
or
backward()
is called, you will see the
following command-line interface:
=================================== forward ===================================
Parameter node with uid='Parameter28' shape=[](2,)
[CNTK forward] >>> help
Commands:
n - execute the next node
n <number> - execute the next <number> nodes
u f - execute until forward pass (like 'n' when already in forward pass)
u b - execute until backward pass (like 'n' when already in backward pass)
u name - execute until a node with that name is hit
u <lambda> - execute until the lambda expression is True. Examples:
Until a Times node is hit:
lambda arg, node: node.op_name == 'Times'
Until a node is hit that has 3 dimensions:
lambda arg, node: len(node.shape) == 3
Until the variance of the input exceeds 1 (np = numpy):
lambda arg, node: np.var(arg) > 1
c - execute until end
p - print input (forward) or root gradients (backward)
d - drop into a pdb shell
q - quit
[CNTK backward] >>> n
Times node with uid='Times29' shape=[*,*](2,)
[CNTK forward] >>> n
=================================== backward ===================================
Times node with uid='Times29' shape=[*,*](2,)
[CNTK backward] >>> p
State: None
Root gradients:
[[[-0.79412955 0.79412955]]
[[-0.79412955 0.79412955]]
[[ 0.20587046 -0.20587045]]
[[ 0.20587046 -0.20587045]]
[[ 0.20587046 -0.20587045]]
[[ 0.20587046 -0.20587045]]
[[-0.79412955 0.79412955]]
[[ 0.20587046 -0.20587045]]
[[ 0.20587039 -0.20587039]]
[[-0.79412961 0.79412961]]]
- At every stop the following information is given:
- Forward or backward pass
- Node type (e.g. ‘Times’)
- Name if given, otherwise it is omitted
- uid, which is a unique reference within the graph
- shape having the format [dynamic axis](static axes). E.g.
[*,*](2,)
means that the node’s output has two dynamic axes (batch and sequence) and one static axis (2 dimensions)
-
debug_model
(model, in_stream=sys.stdin, out_stream=sys.stdout, exit_func=sys.exit)[source]¶ Returns a cloned model that has debug nodes inserted everywhere. When the graph is evaluated or trained, those nodes will allow to inspect the graph.
Parameters: - model (root node) – root node until which the nodes are to be debugged
- in_stream (object behaving like sys.stdin, default stdin) – readline() will be called on it to obtain user input
- out_stream (object behaving like sys.stdout, default stdout) – write() and flush() will be called on it to output debug info to the user
- exit_func (callable, default sys.exit) – callable that takes an exit code and is called, when the user exits the debugging process
Returns: a clone of the model that has debugging enabled
-
save_as_legacy_model
(root_op, filename)[source]¶ Save the network of
root_op
infilename
. For debugging purposes only, very likely to be deprecated in the future.Parameters: - root_op (
Function
) – op of the graph to save - filename (str) – filename to store the model in.
- root_op (
-
set_checked_mode
(enable)[source]¶ - Checked mode enables additional runtime verification such as:
- Tracking NaN occurrences in sequence gaps.
- Function graph verification after binding of free static axes to actual values at runtime
Enabling checked mode incurs additional runtime costs and is meant to be used as a debugging aid.
Parameters: enable (bool) – whether to enable checked mode (with performance impact)
-
set_computation_network_trace_level
(level)[source]¶ Set trace level to the computation network. Currently supported values:
- 0
- turn off trace
- 1
- output nodes’ dimensions and some other static info
- 1000
- output each node’s abs sum of elements in its value matrix for every forward/backward
- 1000000
- output each node’s full matrix for every forward/backward
Parameters: level (int) – trace level