CNTK 202: Language Understanding with Recurrent Networks

This tutorial shows how to implement a recurrent network to process text, for the Air Travel Information Services (ATIS) task of slot tagging (tag individual words to their respective classes, where the classes are provided as labels in the training data set).

There are 2 parts to this tutorial: - Part 1: We will tag each word in a sequence to their corresponding label - Part 2: We will classify a sequence to its corresponding intent.

We will start with a straight-forward (linear) embedding of the words followed by a recurrent LSTM to label each word in a sequence to the corresponding class. We will show how to classify each word token in a sequence to the corresponding class. This will then be extended to include neighboring words and run bidirectionally.

We will take the last state of the sequence and train a model that classifies the entire sequence to the corresponding class label (in this case the intent associated with the sequence).

The techniques you will practice are: * model description by composing layer blocks, a convenient way to compose networks/models without requiring the need to write formulas, * creating your own layer block * variables with different sequence lengths in the same network * training the network

We assume that you are familiar with basics of deep learning, and these specific concepts: * recurrent networks (Wikipedia page) * text embedding (Wikipedia page)

Prerequisites

We assume that you have already installed CNTK. This tutorial requires CNTK V2. We strongly recommend to run this tutorial on a machine with a capable CUDA-compatible GPU. Deep learning without GPUs is not fun.

Data download

In this tutorial, we are going to use a (lightly preprocessed) version of the ATIS dataset. You can download the data automatically by running the cells below or by executing the manual instructions.

Fallback manual instructions Download the ATIS training and test files and put them at the same folder as this notebook. If you want to see how the model is predicting on new sentences you will also need the vocabulary files for queries and slots

from __future__ import print_function # Use a function definition from future version (say 3.x from 2.7 interpreter)
import requests
import os

def download(url, filename):
    """ utility function to download a file """
    response = requests.get(url, stream=True)
    with open(filename, "wb") as handle:
        for data in response.iter_content():
            handle.write(data)

locations = ['Tutorials/SLUHandsOn', 'Examples/LanguageUnderstanding/ATIS/BrainScript']

data = {
  'train': { 'file': 'atis.train.ctf', 'location': 0 },
  'test': { 'file': 'atis.test.ctf', 'location': 0 },
  'query': { 'file': 'query.wl', 'location': 1 },
  'slots': { 'file': 'slots.wl', 'location': 1 },
  'intent': { 'file': 'intent.wl', 'location': 1 }
}

for item in data.values():
    location = locations[item['location']]
    path = os.path.join('..', location, item['file'])
    if os.path.exists(path):
        print("Reusing locally cached:", item['file'])
        # Update path
        item['file'] = path
    elif os.path.exists(item['file']):
        print("Reusing locally cached:", item['file'])
    else:
        print("Starting download:", item['file'])
        url = "https://github.com/Microsoft/CNTK/blob/release/2.6/%s/%s?raw=true"%(location, item['file'])
        download(url, item['file'])
        print("Download completed")

Reusing locally cached: query.wl
Reusing locally cached: intent.wl
Reusing locally cached: atis.test.ctf
Reusing locally cached: atis.train.ctf
Reusing locally cached: slots.wl

Importing libraries: CNTK, math and numpy

CNTK’s Python module contains several submodules like io, learner, and layers. We also use NumPy in some cases since the results returned by CNTK work like NumPy arrays.

import math
import numpy as np

import cntk as C
import cntk.tests.test_utils
cntk.tests.test_utils.set_device_from_pytest_env() # (only needed for our build system)
C.cntk_py.set_fixed_random_seed(1) # fix a random seed for CNTK components

Task overview: Slot tagging

The task we want to approach in this tutorial is slot tagging. We use the ATIS corpus. ATIS contains human-computer queries from the domain of Air Travel Information Services, and our task will be to annotate (tag) each word of a query whether it belongs to a specific item of information (slot), and which one.

The data in your working folder has already been converted into the “CNTK Text Format.” Let us look at an example from the test-set file atis.test.ctf:

19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O
19  |S0 770:1 |# show                         |S2 128:1 |# O
19  |S0 429:1 |# flights                      |S2 128:1 |# O
19  |S0 444:1 |# from                         |S2 128:1 |# O
19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name
19  |S0 851:1 |# to                           |S2 128:1 |# O
19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name
19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name
19  |S0 654:1 |# on                           |S2 128:1 |# O
19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name
19  |S0 179:1 |# EOS                          |S2 128:1 |# O

This file has 7 columns:

• a sequence id (19). There are 11 entries with this sequence id. This means that sequence 19 consists of 11 tokens;
• column S0, which contains numeric word indices; the input data is encoded in one-hot vectors. There are 943 words in the vocabulary, so each word is a 943 element vector of all 0 with a 1 at a vector index chosen to represent that word. For example the word “from” is represented with a 1 at index 444 and zero everywhere else in the vector. The word “monday” is represented with a 1 at index 601 and zero everywhere else in the vector.
• a comment column denoted by #, to allow a human reader to know what the numeric word index stands for; Comment columns are ignored by the system. BOS and EOS are special words to denote beginning and end of sentence, respectively;
• column S1 is an intent label, which we will use in the second part of the tutorial;
• another comment column that shows the human-readable label of the numeric intent index;
• column S2 is the slot label, represented as a numeric index; and
• another comment column that shows the human-readable label of the numeric label index.

The task of the neural network is to look at the query (column S0) and predict the slot label (column S2). As you can see, each word in the input gets assigned either an empty label O or a slot label that begins with B- for the first word, and with I- for any additional consecutive word that belongs to the same slot.

Model Creation

The model we will use is a recurrent model consisting of an embedding layer, a recurrent LSTM cell, and a dense layer to compute the posterior probabilities:

slot label   "O"        "O"        "O"        "O"  "B-fromloc.city_name"
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +-------+  +-------+  +-------+  +-------+  +-------+
          | Dense |  | Dense |  | Dense |  | Dense |  | Dense |  ...
          +-------+  +-------+  +-------+  +-------+  +-------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +------+   +------+   +------+   +------+   +------+
     0 -->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->...
          +------+   +------+   +------+   +------+   +------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +-------+  +-------+  +-------+  +-------+  +-------+
          | Embed |  | Embed |  | Embed |  | Embed |  | Embed |  ...
          +-------+  +-------+  +-------+  +-------+  +-------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
w      ------>+--------->+--------->+--------->+--------->+------...
             BOS      "show"    "flights"    "from"   "burbank"

Or, as a CNTK network description. Please have a quick look and match it with the description above: (descriptions of these functions can be found at: the layers reference)

# number of words in vocab, slot labels, and intent labels
vocab_size = 943 ; num_labels = 129 ; num_intents = 26

# model dimensions
input_dim  = vocab_size
label_dim  = num_labels
emb_dim    = 150
hidden_dim = 300

# Create the containers for input feature (x) and the label (y)
x = C.sequence.input_variable(vocab_size)
y = C.sequence.input_variable(num_labels)

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels, name='classify')
        ])

Now we are ready to create a model and inspect it.

The model attributes are fully accessible from Python. The first layer named embed is an Embedding layer. Here we use the CNTK default, which is linear embedding. It is a simple matrix with dimension (input word encoding x output projected dimension). You can access its parameter E (where the embeddings are stored) like any other attribute of a Python object. Its shape contains a -1 which indicates that this parameter (with input dimension) is not fully specified yet, while the output dimension is set to emb_dim ( = 150 in this tutorial).

Additionally, we also inspect the value of the bias vector in the Dense layer named classify. The Dense layer is a fundamental compositional unit of a Multi-Layer Perceptron (as introduced in CNTK 103C tutorial). The Dense layer has both weight and bias parameters, one each per Dense layer. Bias terms are by default initialized to 0 (but there is a way to change that if you need). As you create the model, one should name the layer component and then access the parameters as shown here.

Suggested task: What should be the expected dimension of the weight matrix from the layer named classify? Try printing the weight matrix of the classify layer? Does it match with your expected size?

# peek
z = create_model()
print(z.embed.E.shape)
print(z.classify.b.value)

(-1, 150)
[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.
  0.  0.  0.]

In our case we have input as one-hot encoded vector of length 943 and the output dimension emb_dim is set to 150. In the code below we pass the input variable x to our model z. This binds the model with input data of known shape. In this case, the input shape will be the size of the input vocabulary. With this modification, the parameter returned by the embed layer is completely specified (943, 150). Note: You can initialize the Embedding matrix with pre-computed vectors using Word2Vec or GloVe.

# Pass an input and check the dimension
z = create_model()
print(z(x).embed.E.shape)

(943, 150)

To train and test a model in CNTK, we need to create a model and specify how to read data and perform training and testing.

In order to train we need to specify:

• how to read the data
• the model function, its inputs, and outputs
• hyper-parameters for the learner such as the learning rate

Data Reading

We already looked at the data. But how do you generate this format? For reading text, this tutorial uses the CNTKTextFormatReader. It expects the input data to be in a specific format, as described here.

For this tutorial, we created the corpora by two steps: * convert the raw data into a plain text file that contains of TAB-separated columns of space-separated text. For example:

BOS show flights from burbank to st. louis on monday EOS (TAB) flight (TAB) O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-depart_date.day_name O

This is meant to be compatible with the output of the paste command. * convert it to CNTK Text Format (CTF) with the following command:

python [CNTK root]/Scripts/txt2ctf.py --map query.wl intent.wl slots.wl --annotated True --input atis.test.txt --output atis.test.ctf where the three .wl files give the vocabulary as plain text files, one word per line.

In these CTF files, our columns are labeled S0, S1, and S2. These are connected to the actual network inputs by the corresponding lines in the reader definition:

def create_reader(path, is_training):
    return C.io.MinibatchSource(C.io.CTFDeserializer(path, C.io.StreamDefs(
         query         = C.io.StreamDef(field='S0', shape=vocab_size,  is_sparse=True),
         intent        = C.io.StreamDef(field='S1', shape=num_intents, is_sparse=True),
         slot_labels   = C.io.StreamDef(field='S2', shape=num_labels,  is_sparse=True)
     )), randomize=is_training, max_sweeps = C.io.INFINITELY_REPEAT if is_training else 1)

# peek
reader = create_reader(data['train']['file'], is_training=True)
reader.streams.keys()

dict_keys(['query', 'slot_labels', 'intent'])

Training

We also must define the training criterion (loss function), and also an error metric to track. In most tutorials, we know the input dimensions and the corresponding labels. We directly create the loss and the error functions. In this tutorial we will do the same. However, we take a brief detour and learn about placeholders. This concept would be useful for Task 3.

Learning note: Introduction to placeholder: Remember that the code we have been writing is not actually executing any heavy computation it is just specifying the function we want to compute on data during training/testing. And in the same way that it is convenient to have names for arguments when you write a regular function in a programming language, it is convenient to have placeholders that refer to arguments (or local computations that need to be reused). Eventually, some other code will replace these placeholders with other known quantities in the same way that in a programming language the function will be called with concrete values bound to its arguments.

Specifically, the input variables you have created above x = C.sequence.input_variable(vocab_size) holds data pre-defined by vocab_size. In the case where such instantiations are challenging or not possible, using placeholder is a logical choice. Having the placeholder only allows you to defer the specification of the argument at a later time when you may have the data.

Here is an example below that illustrates the use of placeholder.

def create_criterion_function(model):
    labels = C.placeholder(name='labels')
    ce   = C.cross_entropy_with_softmax(model, labels)
    errs = C.classification_error      (model, labels)
    return C.combine ([ce, errs]) # (features, labels) -> (loss, metric)

criterion = create_criterion_function(create_model())
criterion.replace_placeholders({criterion.placeholders[0]: C.sequence.input_variable(num_labels)})

Composite(Combine): Input('Input2300', [#, *], [129]), Placeholder('labels', [???], [???]) -> Output('Block2270_Output_0', [#, *], [1]), Output('Block2290_Output_0', [#, *], [])

While the cell above works well when one has input parameters defined at network creation, it compromises readability. Hence we prefer creating functions as shown below

def create_criterion_function_preferred(model, labels):
    ce   = C.cross_entropy_with_softmax(model, labels)
    errs = C.classification_error      (model, labels)
    return ce, errs # (model, labels) -> (loss, error metric)

def train(reader, model_func, max_epochs=10, task='slot_tagging'):

    # Instantiate the model function; x is the input (feature) variable
    model = model_func(x)

    # Instantiate the loss and error function
    loss, label_error = create_criterion_function_preferred(model, y)

    # training config
    epoch_size = 18000        # 18000 samples is half the dataset size
    minibatch_size = 70

    # LR schedule over epochs
    # In CNTK, an epoch is how often we get out of the minibatch loop to
    # do other stuff (e.g. checkpointing, adjust learning rate, etc.)
    lr_per_sample = [3e-4]*4+[1.5e-4]
    lr_per_minibatch = [lr * minibatch_size for lr in lr_per_sample]
    lr_schedule = C.learning_parameter_schedule(lr_per_minibatch, epoch_size=epoch_size)

    # Momentum schedule
    momentums = C.momentum_schedule(0.9048374180359595, minibatch_size=minibatch_size)

    # We use a the Adam optimizer which is known to work well on this dataset
    # Feel free to try other optimizers from
    # https://www.cntk.ai/pythondocs/cntk.learner.html#module-cntk.learner
    learner = C.adam(parameters=model.parameters,
                     lr=lr_schedule,
                     momentum=momentums,
                     gradient_clipping_threshold_per_sample=15,
                     gradient_clipping_with_truncation=True)

    # Setup the progress updater
    progress_printer = C.logging.ProgressPrinter(tag='Training', num_epochs=max_epochs)

    # Uncomment below for more detailed logging
    #progress_printer = ProgressPrinter(freq=100, first=10, tag='Training', num_epochs=max_epochs)

    # Instantiate the trainer
    trainer = C.Trainer(model, (loss, label_error), learner, progress_printer)

    # process minibatches and perform model training
    C.logging.log_number_of_parameters(model)

    # Assign the data fields to be read from the input
    if task == 'slot_tagging':
        data_map={x: reader.streams.query, y: reader.streams.slot_labels}
    else:
        data_map={x: reader.streams.query, y: reader.streams.intent}

    t = 0
    for epoch in range(max_epochs):         # loop over epochs
        epoch_end = (epoch+1) * epoch_size
        while t < epoch_end:                # loop over minibatches on the epoch
            data = reader.next_minibatch(minibatch_size, input_map= data_map)  # fetch minibatch
            trainer.train_minibatch(data)               # update model with it
            t += data[y].num_samples                    # samples so far
        trainer.summarize_training_progress()

Run the trainer

You can find the complete recipe below.

def do_train():
    global z
    z = create_model()
    reader = create_reader(data['train']['file'], is_training=True)
    train(reader, z)
do_train()

Training 721479 parameters in 6 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 10]: [Training] loss = 1.740198 * 18010, metric = 28.02% * 18010 6.466s (2785.3 samples/s);
Finished Epoch[2 of 10]: [Training] loss = 0.665177 * 18051, metric = 14.30% * 18051 5.238s (3446.2 samples/s);
Finished Epoch[3 of 10]: [Training] loss = 0.526256 * 17941, metric = 11.34% * 17941 5.198s (3451.5 samples/s);
Finished Epoch[4 of 10]: [Training] loss = 0.395405 * 18059, metric = 8.22% * 18059 5.329s (3388.8 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 10]: [Training] loss = 0.293512 * 17957, metric = 6.20% * 17957 5.106s (3516.8 samples/s);
Finished Epoch[6 of 10]: [Training] loss = 0.264932 * 18021, metric = 5.73% * 18021 5.335s (3377.9 samples/s);
Finished Epoch[7 of 10]: [Training] loss = 0.217258 * 17980, metric = 4.69% * 17980 5.248s (3426.1 samples/s);
Finished Epoch[8 of 10]: [Training] loss = 0.209614 * 18025, metric = 4.55% * 18025 5.139s (3507.5 samples/s);
Finished Epoch[9 of 10]: [Training] loss = 0.165851 * 17956, metric = 3.84% * 17956 5.636s (3185.9 samples/s);
Finished Epoch[10 of 10]: [Training] loss = 0.157653 * 18039, metric = 3.41% * 18039 5.646s (3195.0 samples/s);

This shows how learning proceeds over epochs (passes through the data). For example, after four epochs, the loss, which is the cross-entropy criterion, has reduced significantly as measured on the ~18000 samples of this epoch, and the same with the error rate on those same 18000 training samples.

The epoch size is the number of samples–counted as word tokens, not sentences–to process between model checkpoints.

Once the training has completed (a little less than 2 minutes on a Titan-X or a Surface Book), you will see an output like this

Finished Epoch[10 of 10]: [Training] loss = 0.157653 * 18039, metric = 3.41% * 18039

which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.

On a CPU-only machine, it can be 4 or more times slower. You can try setting

emb_dim    = 50
hidden_dim = 100

to reduce the time it takes to run on a CPU, but the model will not fit as well as when the hidden and embedding dimension are larger.

Evaluating the model

Like the train() function, we also define a function to measure accuracy on a test set by computing the error over multiple minibatches of test data. For evaluating on a small sample read from a file, you can set a minibatch size reflecting the sample size and run the test_minibatch on that instance of data. To see how to evaluate a single sequence, we provide an instance later in the tutorial.

def evaluate(reader, model_func, task='slot_tagging'):

    # Instantiate the model function; x is the input (feature) variable
    model = model_func(x)

    # Create the loss and error functions
    loss, label_error = create_criterion_function_preferred(model, y)

    # process minibatches and perform evaluation
    progress_printer = C.logging.ProgressPrinter(tag='Evaluation', num_epochs=0)

    # Assign the data fields to be read from the input
    if task == 'slot_tagging':
        data_map={x: reader.streams.query, y: reader.streams.slot_labels}
    else:
        data_map={x: reader.streams.query, y: reader.streams.intent}

    while True:
        minibatch_size = 500
        data = reader.next_minibatch(minibatch_size, input_map= data_map)  # fetch minibatch
        if not data:                                 # until we hit the end
            break

        evaluator = C.eval.Evaluator(loss, progress_printer)
        evaluator.test_minibatch(data)

    evaluator.summarize_test_progress()

Now we can measure the model accuracy by going through all the examples in the test set and using the C.eval.Evaluator method.

def do_test():
    reader = create_reader(data['test']['file'], is_training=False)
    evaluate(reader, z)
do_test()
z.classify.b.value

Finished Evaluation [1]: Minibatch[1-23]: metric = 0.34% * 10984;

array([ -1.86572317e-02,  -8.51036515e-03,   1.38878925e-02,
        -1.95176266e-02,  -2.78977025e-03,  -1.23168388e-02,
        -6.16775267e-03,  -1.48158008e-02,  -5.82036236e-03,
        -2.99133137e-02,  -1.39552690e-02,  -2.11144108e-02,
        -1.13499342e-02,  -1.25011550e-02,  -8.19404377e-04,
        -5.26463473e-03,  -2.67275460e-02,  -1.80706571e-04,
        -3.97865893e-03,  -2.99989916e-02,  -1.00385472e-02,
        -6.81575621e-03,  -2.65348833e-02,  -2.01367699e-02,
        -2.63106022e-02,   4.22888156e-03,   5.74267423e-03,
        -1.56373512e-02,   7.71288527e-04,  -2.11508083e-03,
        -9.64712165e-03,  -1.98591035e-02,  -1.32136559e-02,
         7.97899254e-03,  -1.76088810e-02,   9.19441786e-03,
         1.30802142e-02,  -3.85359419e-03,   1.86733739e-03,
        -5.96518070e-03,  -3.07163727e-02,  -3.04672867e-03,
        -3.46868881e-04,  -1.29565294e-03,  -4.47260169e-03,
        -1.29292896e-02,  -1.05356863e-02,  -9.16024856e-03,
         6.08767197e-03,  -3.75504000e-03,  -2.08706614e-02,
        -6.74075307e-03,  -1.62283499e-02,  -1.54837407e-02,
        -4.45737224e-03,  -2.18946021e-02,  -7.09120464e-03,
        -2.59322841e-02,  -7.19473930e-03,  -2.38050371e-02,
        -2.12035086e-02,  -1.92295481e-02,  -1.78258196e-02,
        -2.89904419e-02,  -2.11317427e-02,  -1.59252994e-02,
         1.15247713e-02,  -6.23733690e-03,  -2.34362725e-02,
        -2.94410121e-02,  -2.90733539e-02,  -2.31353957e-02,
        -2.56022997e-02,  -2.99183521e-02,  -3.48845944e-02,
        -2.37278938e-02,  -1.23830158e-02,  -8.28807056e-03,
        -7.39323627e-03,  -2.71228235e-02,  -1.66217834e-02,
        -2.01343931e-02,  -7.25648087e-03,  -1.39272353e-02,
        -6.12456305e-03,  -1.73326526e-02,  -2.00424399e-02,
        -6.42115856e-03,  -1.77380182e-02,   2.44801558e-05,
        -2.94576529e-02,   3.32167302e-03,  -2.08815038e-02,
        -1.13182077e-02,  -1.59333460e-02,  -1.49212936e-02,
         5.97879477e-03,  -1.84684750e-02,  -2.37341877e-02,
        -3.12264990e-02,   5.03906514e-03,  -3.30699719e-02,
        -2.31159870e-02,  -9.83368699e-03,  -2.43863855e-02,
        -1.25425290e-02,  -2.47525666e-02,  -9.63981543e-03,
        -1.55018885e-02,  -9.93501674e-03,  -1.19379470e-02,
        -5.87523589e-03,  -1.70155372e-02,  -2.29082517e-02,
        -1.84413474e-02,  -1.43948747e-02,  -1.95573717e-02,
        -1.57539565e-02,  -1.90414693e-02,  -9.15751979e-03,
        -2.89104711e-02,  -1.02876564e-02,  -2.83453409e-02,
        -1.30684981e-02,  -5.12228906e-03,  -1.68853626e-02,
        -1.10401753e-02,  -7.05094775e-03,   1.51731307e-02], dtype=float32)

The following block of code illustrates how to evaluate a single sequence. Additionally we show how one can pass in the information using NumPy arrays.

# load dictionaries
query_wl = [line.rstrip('\n') for line in open(data['query']['file'])]
slots_wl = [line.rstrip('\n') for line in open(data['slots']['file'])]
query_dict = {query_wl[i]:i for i in range(len(query_wl))}
slots_dict = {slots_wl[i]:i for i in range(len(slots_wl))}

# let's run a sequence through
seq = 'BOS flights from new york to seattle EOS'
w = [query_dict[w] for w in seq.split()] # convert to word indices
print(w)
onehot = np.zeros([len(w),len(query_dict)], np.float32)
for t in range(len(w)):
    onehot[t,w[t]] = 1

#x = C.sequence.input_variable(vocab_size)
pred = z(x).eval({x:[onehot]})[0]
print(pred.shape)
best = np.argmax(pred,axis=1)
print(best)
list(zip(seq.split(),[slots_wl[s] for s in best]))

[178, 429, 444, 619, 937, 851, 752, 179]
(8, 129)
[128 128 128  48 110 128  78 128]

[('BOS', 'O'),
 ('flights', 'O'),
 ('from', 'O'),
 ('new', 'B-fromloc.city_name'),
 ('york', 'I-fromloc.city_name'),
 ('to', 'O'),
 ('seattle', 'B-toloc.city_name'),
 ('EOS', 'O')]

Modifying the Model

In the following, you will be given tasks to practice modifying CNTK configurations. The solutions are given at the end of this document... but please try without!

A Word About ```Sequential()`` <https://www.cntk.ai/pythondocs/layerref.html#sequential>`__

Before jumping to the tasks, let’s have a look again at the model we just ran. The model is described in what we call function-composition style.

Sequential([
    Embedding(emb_dim),
    Recurrence(LSTM(hidden_dim), go_backwards=False),
    Dense(num_labels)
])

You may be familiar with the “sequential” notation from other neural-network toolkits. If not, `Sequential() <https://www.cntk.ai/pythondocs/layerref.html#sequential>`__ is a powerful operation that, in a nutshell, allows to compactly express a very common situation in neural networks where an input is processed by propagating it through a progression of layers. Sequential() takes an list of functions as its argument, and returns a new function that invokes these functions in order, each time passing the output of one to the next. For example,

FGH = Sequential ([F,G,H])
y = FGH (x)

means the same as

y = H(G(F(x)))

This is known as “function composition”, and is especially convenient for expressing neural networks, which often have this form:

     +-------+   +-------+   +-------+
x -->|   F   |-->|   G   |-->|   H   |--> y
     +-------+   +-------+   +-------+

Coming back to our model at hand, the Sequential expression simply says that our model has this form:

     +-----------+   +----------------+   +------------+
x -->| Embedding |-->| Recurrent LSTM |-->| DenseLayer |--> y
     +-----------+   +----------------+   +------------+

Task 1: Add Batch Normalization

We now want to add new layers to the model, specifically batch normalization.

Batch normalization is a popular technique for speeding up convergence. It is often used for image-processing setups. But could it work for recurrent models, too?

Note: training with Batch Normalization is currently only supported on GPU.

So your task will be to insert batch-normalization layers before and after the recurrent LSTM layer. If you have completed the hands-on labs on image processing, you may remember that the batch-normalization layer has this form:

BatchNormalization()

So please go ahead and modify the configuration and see what happens.

If everything went right, you will notice improved convergence speed (loss and metric) compared to the previous configuration.

# Your task: Add batch normalization
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

# Enable these when done:
#do_train()
#do_test()

Task 2: Add a Lookahead

Our recurrent model suffers from a structural deficit: Since the recurrence runs from left to right, the decision for a slot label has no information about upcoming words. The model is a bit lopsided. Your task will be to modify the model such that the input to the recurrence consists not only of the current word, but also of the next one (lookahead).

Your solution should be in function-composition style. Hence, you will need to write a Python function that does the following:

• takes no input arguments
• creates a placeholder (sequence) variable
• computes the “next value” in this sequence using the sequence.future_value() operation and
• concatenates the current and the next value into a vector of twice the embedding dimension using splice()

and then insert this function into Sequential()‘s list right after the embedding layer.

# Your task: Add lookahead
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

# Enable these when done:
#do_train()
#do_test()

Task 3: Bidirectional Recurrent Model

Aha, knowledge of future words help. So instead of a one-word lookahead, why not look ahead until all the way to the end of the sentence, through a backward recurrence? Let us create a bidirectional model!

Your task is to implement a new layer that performs both a forward and a backward recursion over the data, and concatenates the output vectors.

Note, however, that this differs from the previous task in that the bidirectional layer contains learnable model parameters. In function-composition style, the pattern to implement a layer with model parameters is to write a factory function that creates a function object.

A function object, also known as *functor*, is an object that is both a function and an object. Which means nothing else that it contains data yet still can be invoked as if it was a function.

For example, Dense(outDim) is a factory function that returns a function object that contains a weight matrix W, a bias b, and another function to compute input @ W + b. (This is using Python 3.5 notation for matrix multiplication. In Numpy syntax it is input.dot(W) + b). E.g. saying Dense(1024) will create this function object, which can then be used like any other function, also immediately: Dense(1024)(x).

Let’s look at an example for further clarity: Let us implement a new layer that combines a linear layer with a subsequent batch normalization. To allow function composition, the layer needs to be realized as a factory function, which could look like this:

def DenseLayerWithBN(dim):
    F = Dense(dim)
    G = BatchNormalization()
    x = placeholder()
    apply_x = G(F(x))
    return apply_x

Invoking this factory function will create F, G, x, and apply_x. In this example, F and G are function objects themselves, and apply_x is the function to be applied to the data. Thus, e.g. calling DenseLayerWithBN(1024) will create an object containing a linear-layer function object called F, a batch-normalization function object G, and apply_x which is the function that implements the actual operation of this layer using F and G. It will then return apply_x. To the outside, apply_x looks and behaves like a function. Under the hood, however, apply_x retains access to its specific instances of F and G.

Now back to our task at hand. You will now need to create a factory function, very much like the example above. You shall create a factory function that creates two recurrent layer instances (one forward, one backward), and then defines an apply_x function which applies both layer instances to the same x and concatenate the two results.

Alright, give it a try! To know how to realize a backward recursion in CNTK, please take a hint from how the forward recursion is done. Please also do the following: * remove the one-word lookahead you added in the previous task, which we aim to replace; and * make sure each LSTM is using hidden_dim//2 outputs to keep the total number of model parameters limited.

# Your task: Add bidirectional recurrence
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

# Enable these when done:
#do_train()
#do_test()

The bidirectional model has 40% less parameters than the lookahead one. However, if you go back and look closely you may find that the lookahead one trained about 30% faster. This is because the lookahead model has both less horizontal dependencies (one instead of two recurrences) and larger matrix products, and can thus achieve higher parallelism.

Solution 1: Adding Batch Normalization

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            #C.layers.BatchNormalization(), #Remove this comment if running on GPU
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            #C.layers.BatchNormalization(), #Remove this comment if running on GPU
            C.layers.Dense(num_labels)
        ])

do_train()
do_test()

Training 721479 parameters in 6 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 10]: [Training] loss = 1.740198 * 18010, metric = 28.02% * 18010 4.223s (4264.7 samples/s);
Finished Epoch[2 of 10]: [Training] loss = 0.665177 * 18051, metric = 14.30% * 18051 3.917s (4608.4 samples/s);
Finished Epoch[3 of 10]: [Training] loss = 0.526256 * 17941, metric = 11.34% * 17941 3.898s (4602.6 samples/s);
Finished Epoch[4 of 10]: [Training] loss = 0.395405 * 18059, metric = 8.22% * 18059 4.061s (4446.9 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 10]: [Training] loss = 0.293512 * 17957, metric = 6.20% * 17957 3.996s (4493.7 samples/s);
Finished Epoch[6 of 10]: [Training] loss = 0.264932 * 18021, metric = 5.73% * 18021 3.931s (4584.3 samples/s);
Finished Epoch[7 of 10]: [Training] loss = 0.217258 * 17980, metric = 4.69% * 17980 3.941s (4562.3 samples/s);
Finished Epoch[8 of 10]: [Training] loss = 0.209614 * 18025, metric = 4.55% * 18025 4.105s (4391.0 samples/s);
Finished Epoch[9 of 10]: [Training] loss = 0.165851 * 17956, metric = 3.84% * 17956 3.963s (4530.9 samples/s);
Finished Epoch[10 of 10]: [Training] loss = 0.157653 * 18039, metric = 3.41% * 18039 4.051s (4453.0 samples/s);
Finished Evaluation [1]: Minibatch[1-23]: metric = 0.34% * 10984;

Solution 2: Add a Lookahead

def OneWordLookahead():
    x = C.placeholder()
    apply_x = C.splice(x, C.sequence.future_value(x))
    return apply_x

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            OneWordLookahead(),
            C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_labels)
        ])

do_train()
do_test()

Training 901479 parameters in 6 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 10]: [Training] loss = 1.618925 * 18010, metric = 26.40% * 18010 4.567s (3943.5 samples/s);
Finished Epoch[2 of 10]: [Training] loss = 0.572762 * 18051, metric = 12.46% * 18051 5.560s (3246.6 samples/s);
Finished Epoch[3 of 10]: [Training] loss = 0.420728 * 17941, metric = 8.57% * 17941 5.254s (3414.7 samples/s);
Finished Epoch[4 of 10]: [Training] loss = 0.297996 * 18059, metric = 6.28% * 18059 5.697s (3169.9 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 10]: [Training] loss = 0.224015 * 17957, metric = 4.81% * 17957 5.612s (3199.8 samples/s);
Finished Epoch[6 of 10]: [Training] loss = 0.207126 * 18021, metric = 4.61% * 18021 5.487s (3284.3 samples/s);
Finished Epoch[7 of 10]: [Training] loss = 0.170268 * 17980, metric = 3.69% * 17980 5.538s (3246.7 samples/s);
Finished Epoch[8 of 10]: [Training] loss = 0.164910 * 18025, metric = 3.65% * 18025 5.467s (3297.1 samples/s);
Finished Epoch[9 of 10]: [Training] loss = 0.126314 * 17956, metric = 2.92% * 17956 4.340s (4137.3 samples/s);
Finished Epoch[10 of 10]: [Training] loss = 0.122896 * 18039, metric = 2.67% * 18039 4.218s (4276.7 samples/s);
Finished Evaluation [1]: Minibatch[1-23]: metric = 0.38% * 10984;

Solution 3: Bidirectional Recurrent Model

def BiRecurrence(fwd, bwd):
    F = C.layers.Recurrence(fwd)
    G = C.layers.Recurrence(bwd, go_backwards=True)
    x = C.placeholder()
    apply_x = C.splice(F(x), G(x))
    return apply_x

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim),
            BiRecurrence(C.layers.LSTM(hidden_dim//2),
                                  C.layers.LSTM(hidden_dim//2)),
            C.layers.Dense(num_labels)
        ])

do_train()
do_test()

Training 541479 parameters in 9 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 10]: [Training] loss = 1.886776 * 18010, metric = 30.06% * 18010 7.619s (2363.8 samples/s);
Finished Epoch[2 of 10]: [Training] loss = 0.683211 * 18051, metric = 14.83% * 18051 7.325s (2464.3 samples/s);
Finished Epoch[3 of 10]: [Training] loss = 0.521379 * 17941, metric = 11.42% * 17941 7.265s (2469.5 samples/s);
Finished Epoch[4 of 10]: [Training] loss = 0.394698 * 18059, metric = 8.11% * 18059 7.567s (2386.5 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 10]: [Training] loss = 0.288926 * 17957, metric = 6.06% * 17957 7.386s (2431.2 samples/s);
Finished Epoch[6 of 10]: [Training] loss = 0.267000 * 18021, metric = 5.73% * 18021 7.401s (2434.9 samples/s);
Finished Epoch[7 of 10]: [Training] loss = 0.215379 * 17980, metric = 4.69% * 17980 7.269s (2473.5 samples/s);
Finished Epoch[8 of 10]: [Training] loss = 0.206970 * 18025, metric = 4.37% * 18025 7.333s (2458.1 samples/s);
Finished Epoch[9 of 10]: [Training] loss = 0.160564 * 17956, metric = 3.39% * 17956 7.196s (2495.3 samples/s);
Finished Epoch[10 of 10]: [Training] loss = 0.154584 * 18039, metric = 3.20% * 18039 7.337s (2458.6 samples/s);
Finished Evaluation [1]: Minibatch[1-23]: metric = 0.38% * 10984;

Task overview: Sequence classification

We will reuse the same data for this task. We revisit a sample again:

19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O
19  |S0 770:1 |# show                         |S2 128:1 |# O
19  |S0 429:1 |# flights                      |S2 128:1 |# O
19  |S0 444:1 |# from                         |S2 128:1 |# O
19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name
19  |S0 851:1 |# to                           |S2 128:1 |# O
19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name
19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name
19  |S0 654:1 |# on                           |S2 128:1 |# O
19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name
19  |S0 179:1 |# EOS                          |S2 128:1 |# O

The task of the neural network is to look at the query (column S0) and predict the intent of the sequence (column S1). We will ignore the slot-tags (column S2) this time.

Model Creation

The model we will use is a recurrent model consisting of an embedding layer, a recurrent LSTM cell, and a dense layer to compute the posterior probabilities. Though very similar to the slot tagging model in this case we look only at the embedding from the last layer:

intent                                                "flight"
                                                          ^
                                                          |
                                                      +-------+
                                                      | Dense |   ...
                                                      +-------+
                                                          ^
                                                          |
          +------+   +------+   +------+   +------+   +------+
     0 -->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->...
          +------+   +------+   +------+   +------+   +------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +-------+  +-------+  +-------+  +-------+  +-------+
          | Embed |  | Embed |  | Embed |  | Embed |  | Embed |  ...
          +-------+  +-------+  +-------+  +-------+  +-------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
w      ------>+--------->+--------->+--------->+--------->+------...
             BOS      "show"    "flights"    "from"   "burbank"

Or, as a CNTK network description. Please have a quick look and match it with the description above: (descriptions of these functions can be found at: the layers reference)

Points to note:

• The first difference between this model with the previous one is with regards to the specification of the label y. Since there is only one label per sequence, we use C.input_variable.
• The second difference is the use of Stabilizer. We stabilize the embedded output. The stabilizer adds an additional scalar parameter to the learning that can help our network converge more quickly during training.
• The third difference is the use of a layer function called Fold. As shown in the model above we want the model to have LSTM recurrence except the final one, we set up an LSTM recurrence. The final recurrence will be a Fold operation where we pick the hidden state from the last LSTM block and use it for classification of the entire sequence.

# number of words in vocab, slot labels, and intent labels
vocab_size = 943 ; num_intents = 26

# model dimensions
emb_dim    = 150
hidden_dim = 300

# Create the containers for input feature (x) and the label (y)
x = C.sequence.input_variable(vocab_size)
y = C.input_variable(num_intents)

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Stabilizer(),
            C.layers.Fold(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_intents, name='classify')
        ])

We create the criterion function with the new model and correspondingly update the placeholder.

criterion = create_criterion_function(create_model())
criterion.replace_placeholders({criterion.placeholders[0]: C.sequence.input_variable(num_intents)})

Composite(Combine): Input('Input14623', [#, *], [26]), Placeholder('labels', [???], [???]) -> Output('Block14593_Output_0', [#], [1]), Output('Block14613_Output_0', [#], [])

The same train code can be used except for the fact that in this case we provide the intent tags as the labels.

def do_train():
    global z
    z = create_model()
    reader = create_reader(data['train']['file'], is_training=True)
    train(reader, z, 5, 'intent')
do_train()

Training 690477 parameters in 7 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 5]: [Training] loss = 0.585308 * 18004, metric = 14.42% * 18004 79.421s (226.7 samples/s);
Finished Epoch[2 of 5]: [Training] loss = 0.153843 * 17998, metric = 3.92% * 17998 74.394s (241.9 samples/s);
Finished Epoch[3 of 5]: [Training] loss = 0.081964 * 18000, metric = 2.17% * 18000 59.060s (304.8 samples/s);
Finished Epoch[4 of 5]: [Training] loss = 0.069163 * 18000, metric = 1.92% * 18000 58.621s (307.1 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 5]: [Training] loss = 0.018180 * 17998, metric = 0.47% * 17998 58.345s (308.5 samples/s);

Now we can measure the model accuracy by going through all the examples in the test set and using the C.eval.Evaluator method.

def do_test():
    reader = create_reader(data['test']['file'], is_training=False)
    evaluate(reader, z, 'intent')
do_test()
z.classify.b.value

Finished Evaluation [1]: Minibatch[1-23]: metric = 0.00% * 893;

array([ 0.51183742, -0.29180217, -0.41856512, -0.28958377, -0.72160971,
       -0.24883319,  0.02916328, -0.0852626 ,  0.19014142, -0.41936243,
       -0.23639332, -0.29029393, -0.74258387, -0.12296562,  0.34665295,
       -0.46388549, -0.73981428, -0.2296015 , -0.78151304, -0.24418215,
       -0.52702737,  0.10101102, -0.27433836, -0.24181543, -0.39551306,
        0.30569023], dtype=float32)

The following block of code illustrates how to evaluate a single sequence. Additionally, we show how one can pass in the information using NumPy arrays.

# load dictionaries
query_wl = [line.rstrip('\n') for line in open(data['query']['file'])]
intent_wl = [line.rstrip('\n') for line in open(data['intent']['file'])]
query_dict = {query_wl[i]:i for i in range(len(query_wl))}
intent_dict = {intent_wl[i]:i for i in range(len(intent_wl))}

# let's run a sequence through
seq = 'BOS flights from new york to seattle EOS'
w = [query_dict[w] for w in seq.split()] # convert to word indices
onehot = np.zeros([len(w),len(query_dict)], np.float32)
for t in range(len(w)):
    onehot[t,w[t]] = 1

pred = z(x).eval({x:[onehot]})[0]
best = np.argmax(pred)
print(best)
print(seq, ":", intent_wl[best])

14
BOS flights from new york to seattle EOS : flight

Task 4: Use all hidden states for sequence classification

In the last model, we looked at the output of the last LSTM block. There is another way to model, where we aggregate the output from all the LSTM blocks and use the aggregated output to the final Dense layer.

So your task will be to replace the C.layers.Fold with C.layers.Recurrence layer function. This is the explicit way of setting up recurrence. You will aggregate all the intermediate outputs from LSTM blocks using C.sequence.reduce_sum. Note: this is different from last model where we looked only at the output of the last LSTM block.

So please go ahead and modify the configuration and see what happens.

If everything went right, you will notice improved accuracy (metric). The solution is presented right after, but we suggest you refrain from looking at the solution.

# Replace the line with Fold operation
def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Stabilizer(),
            C.layers.Fold(C.layers.LSTM(hidden_dim), go_backwards=False),
            C.layers.Dense(num_intents, name='classify')
        ])
# Enable these when done:
#do_train()
#do_test()

Aggregating all the intermediate states improves the accuracy for the same number of iterations without any significant increase in the computation time.

Solution 4: Use all hidden states for sequence classification

def create_model():
    with C.layers.default_options(initial_state=0.1):
        return C.layers.Sequential([
            C.layers.Embedding(emb_dim, name='embed'),
            C.layers.Stabilizer(),
            C.sequence.reduce_sum(C.layers.Recurrence(C.layers.LSTM(hidden_dim), go_backwards=False)),
            C.layers.Dense(num_intents, name='classify')
        ])

do_train()
do_test()

Training 690477 parameters in 7 parameter tensors.
Learning rate per minibatch: 0.020999999999999998
Finished Epoch[1 of 5]: [Training] loss = 0.193482 * 18004, metric = 4.07% * 18004 56.904s (316.4 samples/s);
Finished Epoch[2 of 5]: [Training] loss = 0.028983 * 17998, metric = 0.64% * 17998 73.820s (243.8 samples/s);
Finished Epoch[3 of 5]: [Training] loss = 0.059269 * 18000, metric = 1.24% * 18000 71.494s (251.8 samples/s);
Finished Epoch[4 of 5]: [Training] loss = 0.005993 * 18000, metric = 0.18% * 18000 61.856s (291.0 samples/s);
Learning rate per minibatch: 0.010499999999999999
Finished Epoch[5 of 5]: [Training] loss = 0.000344 * 17998, metric = 0.01% * 17998 57.114s (315.1 samples/s);
Finished Evaluation [1]: Minibatch[1-23]: metric = 0.00% * 893;