In this assignment, you will be implementing and training various neural network models
for three different tasks, and analysing the results.
You are to submit two Python files kuzu.py and encoder.py, as well as a written report hw1.pdf
(in pdf format).
Provided Files
Copy the archive hw1.zip into your own filespace and unzip it. This should create a directory
hw1, subdirectory net, and ten Python files kuzu.py, encoder.py, kuzu_main.py, encoder_main.py,
encoder_model.py, seq_train.py, seq_models.py, seq_plot.py, reber.py and anbn.py.
Your task is to complete the skeleton files kuzu.py and encoder.py and submit them, along
with your report.
Part 1: Japanese Character Recognition
For Part 1 of the assignment you will be implementing networks to recognize handwritten
Hiragana symbols. The dataset to be used is Kuzushiji-MNIST or KMNIST for short. The
paper describing the dataset is available here. It is worth reading, but in short: significant
changes occurred to the language when Japan reformed their education system in 1868,
and the majority of Japanese today cannot read texts published over 150 years ago. This
paper presents a dataset of handwritten, labeled examples of this old-style script
(Kuzushiji). Along with this dataset, however, they also provide a much simpler one,
containing 10 Hiragana characters with 7000 samples per class. This is the dataset we will
be using.
the image, followed by log sof
Text from 1772 (left) compared to 1900 showing the standardization of written
Japanese.
1. [1 mark] Implement a model NetLin which computes a linear function of the pixels in
tmax. Run the code by typing:
2023/6/28 17:37
python3 kuzu_main.py --net lin
Copy the final accuracy and confusion matrix into your report. The final accuracy
should be around 70%. Note that the rows of the confusion matrix indicate the target
character, while the columns indicate the one chosen by the network. (0="o", 1="ki",
2="su", 3="tsu", 4="na", 5="ha", 6="ma", 7="ya", 8="re", 9="wo"). More examples of
each character can be found here.
2. [1 mark] Implement a fully connected 2-layer network NetFull (i.e. one hidden layer,
plus the output layer), using tanh at the hidden nodes and log softmax at the output
node. Run the code by typing:
python3 kuzu_main.py --net full
Try different values (multiples of 10) for the number of hidden nodes and try to
determine a value that achieves high accuracy (at least 84%) on the test set. Copy the
final accuracy and confusion matrix into your report, and include a calculation of the
total number of independent parameters in the network.
3. [2 marks] Implement a convolutional network called NetConv, with two convolutional
layers plus one fully connected layer, all using relu activation function, followed by the
output layer, using log softmax. You are free to choose for yourself the number and
size of the filters, metaparameter values (learning rate and momentum), and whether
to use max pooling or a fully convolutional architecture. Run the code by typing:
python3 kuzu_main.py --net conv
Your network should consistently achieve at least 93% accuracy on the test set after
10 training epochs. Copy the final accuracy and confusion matrix into your report, and
include a calculation of the total number of independent parameters in the network.
4. [4 marks] Briefly discuss the following points:
a. the relative accuracy of the three models,
b. the number of independent parameters in each of the three models,
c. the confusion matrix for each model: which characters are most likely to be
mistaken for which other characters, and why?
Part 2: Encoder Networks
In Part 2 you will be editing the file encoder.py to create a dataset which, when run in
combination with encoder_main.py, produces the following image (which is intended to be a
stylized map of mainland China).
2023/6/28 17:37
You should first run the code by typing
python3 encoder_main.py --target star16
Note that target is determined by the tensor star16 in encoder.py, which has 16 rows and 8
columns, indicating that there are 16 inputs and 8 outputs. The inputs use a one-hot
encoding and are generated in the form of an identity matrix using torch.eye()
1. [2 marks] Create by hand a dataset in the form of a tensor called ch34 in the file
encoder.py which, when run with the following command, will produce an image
essentially the same as the one shown above (but possibly rotated or reflected).
python3 encoder_main.py --target ch34
The pattern of dots and lines must be topologically identical. But, it is fine for it to be
rotated or reflected, compared to the image above. Note in particular the five "anchor
points" in the corners and on the edge of the figure.
Your tensor should have 34 rows and 23 columns. Include the final image in your
report, and include the tensor ch34 in your file encoder.py
Part 3: Hidden Unit Dynamics for Recurrent Networks
In Part 3 you will be investigating the hidden unit dynamics of recurrent networks trained
on language prediction tasks, using the supplied code seq_train.py and seq_plot.py.
2023/6/28 17:37
1. [2 marks] Train a Simple Recurrent Network (SRN) on the Reber Grammar prediction
task by typing
python3 seq_train.py --lang reber
This SRN has 7 inputs, 2 hidden units and 7 outputs. The trained networks are stored
every 10000 epochs, in the net subdirectory. After the training finishes, plot the hidden
unit activations at epoch 50000 by typing
python3 seq_plot.py --lang reber --epoch 50
The dots should be arranged in discernable clusters by color. If they are not, run the
code again until the training is successful. The hidden unit activations are printed
according to their "state", using the colormap "jet":
Based on this colormap, annotate your figure (either electronically, or with a pen on a
printout) by drawing a circle around the cluster of points corresponding to each state
in the state machine, and drawing arrows between the states, with each arrow labeled
with its corresponding symbol. Include the annotated figure in your report.
2. [1 mark] Train an SRN on the a
nb
n
language prediction task by typing
python3 seq_train.py --lang anbn
The a
nb
n
language is a concatenation of a random number of A's followed by an
equal number of B's. The SRN has 2 inputs, 2 hidden units and 2 outputs.
Look at the predicted probabilities of A and B as the training progresses. The first B in
each sequence and all A's after the first A are not deterministic and can only be
predicted in a probabilistic sense. But, if the training is successful, all other symbols
should be correctly predicted. In particular, the network should predict the last B in
each sequence as well as the subsequent A. The error should be consistently below
0.01. If the network appears to have learned the task successfully, you can stop it at
any time using ⟨cntrl⟩-c. If it appears to be stuck in a local minimum, you can stop it
and run the code again until it is successful.
After the training finishes, plot the hidden unit activations by typing
python3 seq_plot.py --lang anbn --epoch 100
Include the resulting figure in your report. The states are again printed according to
the colormap "jet". Note, however, that these "states" are not unique but are instead
2023/6/28 17:37
used to count either the number of A's we have seen or the number of B's we are still
expecting to see.
3. [1 mark] Briefly explain how the a
nb
n
prediction task is achieved by the network,
based on the figure you generated in Question 2. Specifically, you should describe
how the hidden unit activations change as the string is processed, and how it is able
to correctly predict the last B in each sequence as well as the following A.
4. [1 mark] Train an SRN on the a
nb
n
c
n
language prediction task by typing
python3 seq_train.py --lang anbncn
The SRN now has 3 inputs, 3 hidden units and 3 outputs. Again, the "state" is used to
count up the A's and count down the B's and C's. Continue training (re-starting, if
necessary) for 200k epochs, or until the network is able to reliably predict all the C's as
well as the subsequent A, and the error is consistently in the range of 0.01 or 0.02.
After the training finishes, plot the hidden unit activations by typing
python3 seq_plot.py --lang anbncn --epoch 200
Rotate the figure in 3 dimensions to get one or more good view(s) of the points in
hidden unit space.
5. [1 mark] Briefly explain how the a
nb
nc
n
prediction task is achieved by the network,
based on the figure you generated in Question 4. Specifically, you should describe
how the hidden unit activations change as the string is processed, and how it is able
to correctly predict the last B in each sequence as well as all of the C's and the
following A.
6. [4 marks] This question is intended to be more challenging. Train an LSTM network to
predict the Embedded Reber Grammar, by typing
python3 seq_train.py --lang reber --embed True --model lstm --hid 4
You can adjust the number of hidden nodes if you wish. Once the training is
successful, try to analyse the behavior of the LSTM and explain how the task is
accomplished (this might involve modifying the code so that it returns and prints out
the context units as well as the hidden units).
Submission
You should submit by typing
give cs9444 hw1 kuzu.py encoder.py hw1.pdf
You can submit as many times as you like - later submissions will overwrite earlier ones.