Supervised Sequence Labelling with Recurrent Neural Networks

1. Introduction

    1.1 Sequence Labelling

1) Definition: Encompass all tasks where sequences of data are transcribed with sequences of discrete labels;

2) Characteristic: Both the inputs and labels form strongly correlated sequences;

3) A further complication: In many cases the alignment between inputs and labels is unkown.

    1.2 RNN

1) Advantage

(1) They are flexible in their use of context information(Because they can learn what to store and what to ignore);

(2) They accept many types and representation of data;

(3) They can recognise sequential patterns in the presence of sequential distortions.

2) Disadvantage

(1) It is difficult to get them to store information for long periods of time;

(2) It can only access contextual information in one direction.

3) Application

曾经，在一些输入数据与标注信息不能对齐的领域，如：语音识别中，RNN只能作为一种辅助角色。因为标准的训练方法需要输入数据能有效地分离。

传统的方法是使用HMM＋RNN，但是这种方法并不能发挥RNN的全部潜能，而且非常讨厌的是，这是一种Discriminative和Generative训练相融合的方法。CTC的出现解决了这个问题。

      1.3 Structure of the Book

1) Chapters 2~4: Background material is prensented;

2) Chapters 5~6: Experiment;

3) Chapters 7~8: New methods.

2. Supervised Sequence Labelling

3. Neural Network