CTC does not always induce a proper probability distribution

This note discusses why connectionist temporal classification (CTC) does not always induce a proper probability distribution.¹ The result is perhaps not surprising to many, but it is never formally written down. At the high level, the probability distribution is not proper when repeating labels appear in the label sequence. I will briefly review the definition of CTC, formally define the problem, and discuss the sufficient conditions for the induced distribution to be proper.

Suppose we have a sequence of $T$ frames $x=(x_1,\dots ,x_T)$ and a sequence of $K$ labels $y=(y_1,\dots ,y_K)$ for $y_k\in L$ for $k=1,\dots ,K$ and for some label set $L$. Let $B:(L\cup \{\varnothing \}{)}^{\ast }\to L^{\ast }$ be a function that collapses repeating labels and removes blanks ($\varnothing$). CTC defines the probability distribution $$\begin{array}{r}p(y|x)=\sum _{z\in B^{-1}(y)}p(z|x)\end{array}$$ where $B^{-1}$ is the pre-image of $B$. In words, if $z\in B^{-1}(y)$, then $y=B(z)$. Each $z_t$ is assumed to be independent of $z_{t^{\prime }}$ for $t^{\prime }\ne t$, so we have $$\begin{array}{r}p(z|x)=\prod _{t=1}^{T}p(z_t|x).\end{array}$$ The sequence $z$ is often called an alignment or a path.

Every time we refer to a function as a probability distribution, we should always ask ourselves whether the distribution is well defined. For a function $p$ to be a proper probability distribution, it should satisfy $p(\omega )\ge 0$ for all $\omega$ is the domain and $\sum _{\omega }p(\omega )=1$. Here is the central question of this note: Given the definition of CTC above, is it true that $\sum _{y}p(y|x)=1$?

Sufficient conditions for properness

Before looking at $p(y|x)$, we first show that $p(z|x)$ is proper. Since $p(z_t|x)$'s are almost always the result of a softmax, we can assume $p(z_t|x)$ is proper. It is also not hard to see that $$\begin{array}{rl}\sum _{z}p(z|x)& =\sum _{z_1}\dots \sum _{z_T}p(z|x)=\sum _{z_1}\dots \sum _{z_T}\prod _{t=1}^{T}p(z_t|x)\\ & =\prod _{t=1}^T\sum _{z_t}p(z_t|x)=\prod _{t=1}^{T}1=1,\end{array}$$ so $p(z|x)$ is proper as long as $p(z_t|x)$'s are.

Given that $p(z|x)$ is proper and that $p(y|x)$ is a sum based on $p(z|x)$, $p(y|x)$ should be proper if the sum is carefully handled. We will use the following fact about probability distributions. Let $\mathrm{\Omega }$ be the event space of distribution $p$. If $A_1,\dots ,A_n$ are disjoint subsets of $\mathrm{\Omega }$ and $\bigcup _{i=1}^n{A}_i=\mathrm{\Omega }$, then $$\begin{array}{r}\sum _{i=1}^{n}p(A_i)=p(\bigcup _{i=1}^n{A}_i)=p(\mathrm{\Omega })=1.\end{array}$$ The key condition here is disjointness ($A_i\cap A_j=\mathrm{\varnothing }$ for any $i$ and $j$) and coverage ($\bigcup _{i=1}^n{A}_i=\mathrm{\Omega }$, or for any $\omega \in \mathrm{\Omega }$, there exists $i\in \{1,\dots ,n\}$ such that $\omega \in A_i$).

To apply the above fact, let $\mathrm{\Omega }$ be the domain of $p(z|x)$, i.e., $(L\cup \{\varnothing \}{)}^T$. The function $p(y|x)$ sums over $B^{-1}(y)$, a subset of $\mathrm{\Omega }$. If the subsets are disjoint and they cover everything in $\mathrm{\Omega }$, then $\sum _{y}p(y|x)=1$ and $p(y|x)$ is proper. More formally, disjointness requires $B^{-1}(y)\cap B^{-1}(y^{\prime })=\mathrm{\varnothing }$ for any $y$ and $y^{\prime }$, and coverage requires that there is a $y$ for every $z$ such that $z\in B^{-1}(y)$. These form the sufficient condition for $p(y|x)$ to be proper.

In the case of CTC, it is trivially true that any alignment $z$ there is a label sequence $y$ such that $y=B(z)$—we simply apply $B$ on $z$ to get a $y$. The deciding condition is whether $B^{-1}(y)$ and $B^{-1}(y^{\prime })$ are disjoint for any $y$ and $y^{\prime }$. What happens when there are $y$ and $y^{\prime }$ such that $B^{-1}(y)\cap B^{-1}(y^{\prime })\ne \mathrm{\varnothing }$? If we train to maximize $p(y|x)$, the probability mass of $p(y^{\prime }|x)$ would also increase.

Implementation of $B^{-1}$

To compute the marginalization efficiently, care needs to be taken when implementing $B^{-1}$. A common approach is to implement $B^{-1}$ as a graph (or more precisely a finite-state acceptor), and run dynamic programming on it.

Since $B$ removes duplicate symbols and blanks, a common approach to implement $B^{-1}$ is to insert zero or more blanks between each symbol and to repeat the symbols one or more times. For example, $B^{-1}(\text{c a t})={\varnothing }^{\ast }{\text{c}}^{+}{\varnothing }^{\ast }{\text{a}}^{+}{\varnothing }^{\ast }{\text{t}}^{+}{\varnothing }^{\ast }$, where I have used regular expression to indicate how many times a symbol can be repeated. However, this implementation causes a problem when there are repeating symbols in the label sequence. For example, if $B^{-1}(\text{m e e t})={\varnothing }^{\ast }{\text{m}}^{+}{\varnothing }^{\ast }{\text{e}}^{+}{\varnothing }^{\ast }{\text{e}}^{+}{\varnothing }^{\ast }{\text{t}}^{+}{\varnothing }^{\ast }$, the sequence $\text{m m e e e t t}$ belongs to $B^{-1}(\text{m e e t})$, but the sequence also belongs to $B^{-1}(\text{m e t})$. In other words, $B^{-1}(\text{m e t})\cap B^{-1}(\text{m e e t})\ne \mathrm{\varnothing }$. This particular implementation not only fails to be faithful to the defition of $B$ (because $B(\text{m m e e e t t})=\text{m e t}$), but also violates the sufficient conditions of properness.²

A more careful implementation of $B^{-1}$ would be to insert one or more blanks between repeating symbols, to insert zero or more blanks otherwise, and to repeat the symbols one or more times. With this implementation, $B^{-1}(\text{m e e t})={\varnothing }^{\ast }{\text{m}}^{+}{\varnothing }^{\ast }{\text{e}}^{+}{\varnothing }^{\mathbf{+}}{\text{e}}^{+}{\varnothing }^{\ast }{\text{t}}^{+}{\varnothing }^{\ast }$. Note that there has to be at least one blank between the two e's.³

Other variants

Since properness depends on the choice of $B$, there are a few other variants that guarantee properness. For example, avoid repeating symbols while allowing blanks (Graves, 2012); only allow repeating symbols and avoid using blanks altogether (Collobert et al., 2016); create a separate symbol for each pair of symbols (Zweig et al., 2017). These variants are considered generalizations of CTC and some are not called CTC anymore.

Acknowledgement