\documentclass[10pt]{article}
\newtheorem{define}{Definition}
%\newcommand{\Z}{{\mathbb{Z}}}
%\usepackage{psfig}
\usepackage{amsmath}

\oddsidemargin=0.15in
\evensidemargin=0.15in
\topmargin=-.5in
\textheight=9in
\textwidth=6.25in

\begin{document}
\input{preamble.tex}

%{lecture number}{lecture date}{Ronitt Rubinfeld}{Your name}
\lecture{20}{April 25, 2012}{Ronitt Rubinfeld}{Jon Schneider}

%%%% body goes in here %%%%

\section{Introduction}

In this lecture, we will demonstrate a procedure to convert any weak learning algorithm to a strong learning algorithm. This procedure is known as the Boosting Algorithm.

Before we state the Boosting Algorithm, let us recall the definitions of weak and strong learning algorithms.

\begin{definition}
The algorithm $A_W$ is a \textbf{weak PAC learning algorithm} for the concept class $\mathcal{C}$ if for every concept $f \in \mathcal{C}$ and distribution $\mathcal{D}$ over the domain of $f$, there exists some $\gamma > 0$ for every $\delta > 0$, such that the algorithm with access to examples $(x, f(x))$ with $x$ drawn from $\mathcal{D}$, returns (with probability at least $1-\delta$) a concept $c$ such that

$$\mathrm{Pr}_{\mathcal{D}}[f(x) \neq c(x)] \leq \frac{1}{2} - \frac{\gamma}{2}$$

\end{definition}

\begin{definition}
The algorithm $A_S$ is a \textbf{strong PAC learning algorithm} for the concept class $\mathcal{C}$ if for every concept $f \in \mathcal{C}$, input distribution $\mathcal{D}$ and $\delta, \epsilon > 0$, the algorithm with access to examples $(x, f(x))$ drawn from $\mathcal{D}$, returns (with probability at least $1-\delta$) a concept $c$ such that

$$\mathrm{Pr}_{\mathcal{D}}[f(x) \neq c(x)] \leq \epsilon$$

\end{definition}

Our main theorem is the following.

\begin{theorem}\label{boost}
If a concept class $\mathcal{C}$ is weakly learnable, then it is strongly learnable.
\end{theorem}

\section{Boosting Algorithm}

To prove Theorem \ref{boost}, we will demonstrate a procedure (called the \textit{Boosting Algorithm}) that transforms a weak learning algorithm into a strong learning algorithm. Below, we will let $f$ be our unknown concept, $\mathcal{D}$ our initial distribution (which we assume to be uniform)\footnote{It turns out that all of our results also hold for the general case where the input distribution is arbitrary (as opposed to uniform). For sake of convenience, however, in this lecture we will assume the input distribution is uniform over the domain of $f$.} , and $WL$ be our weak learning algorithm. 

The algorithm runs in various stages, which we describe below.

\begin{itemize}
\item

\textbf{Initial Stage}: Set $\mathcal{D}_0 = \mathcal{D}$ and use our weak learning algorithm $WL$ to generate a concept $c_0$ such that $\mathrm{Pr}_{\mathcal{D}_0}[f(x) = c_1(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$. 

\item

\textbf{Stage $i$}: Given concepts $c_1, c_2, \dots, c_i$ from the previous $i$ stages, we construct the distribution $\mathcal{D}_i$ via our \textbf{filtering procedure}. We then run $WL$ on $\mathcal{D}_i$ to output a concept $c_{i+1}$ such that $\mathrm{Pr}_{\mathcal{D}_i}[f(x) = c_{i+1}(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$. 

\item

\textbf{Final Stage}: After $T = O(1/\gamma^2\epsilon^2)$ stages, or whenever $\mathrm{maj}(c_1, c_2, \dots, c_{i})$ is correct on a fraction of at least $(1-\epsilon)$ of the inputs, output the concept $c = \mathrm{maj}(c_1, c_2, \dots, c_{i})$.
\end{itemize}

We used a filtering procedure to construct the distribution $\mathcal{D}_i$. We now describe the algorithm for the following filtering procedure, which we repeat until we have the desired number of samples.

\begin{itemize}
\item

Draw an example $(x, f(x))$ from the uniform distribution $\mathcal{D}$. 

\item

If $\mathrm{maj}(c_1, c_2, \dots, c_{i})$ disagree with $f(x)$, keep this sample. 

\item

Let $n_r$ equal the number of $j$ such that $c_j(x) = f(x)$, and let $n_{w}$ equal the number of $j$ such that $c_{j}(x) \neq f(x)$ (so $n_r + n_w = i$). If $n_r - n_w \geq \frac{1}{\gamma\epsilon}$, then toss the sample.

\item

Otherwise, if $n_r - n_w = \frac{\alpha}{\gamma\epsilon}$ for some $0 \leq \alpha < 1$, toss the sample with probability $\alpha$ (and keep it otherwise). 

\end{itemize}

If we look at this filtering procedure in terms of the number $n_w$ of incorrect concepts, then we can see that the probability $p$ of keeping a sample from $\mathcal{D}$ depends piecewise-linearly on the value of $n_w$; specifically, between $0$ and $\frac{i}{2} - \frac{1}{2\gamma\epsilon}$ this function equals $0$, from $\frac{i}{2} - \frac{1}{2\gamma\epsilon}$ to $\frac{i}{2}$ it increases with slope $2\gamma\epsilon$ from $0$ up to $1$, and from $\frac{i}{2}$ to $i$ it stays at $1$.

The main claim we will want to show is that if we need at least $\frac{1}{\epsilon}$ samples of $\mathcal{D}$ to output a sample of $\mathcal{D}_i$, then $\mathrm{Maj}(c_1, c_2, \dots, c_i)$ is $(1-\epsilon)$-close to $f$. We will do this in the following two sections.

\section{Notation}

In the following section, we will establish (most of) a proof of correctness for the Boosting Algorithm. First, however, we will introduce some terminology that will be useful in our following proof.

\begin{definition}
For a concept $c$ and an input $x$, 

$$R_c(x) = \begin{cases} 1, & \mbox{if }  f(x) = c(x)  \\ -1,  & \mbox{if } f(x) \neq c(x) \end{cases} $$

\end{definition}

\begin{definition}
If $c_i$ is the concept introduced at stage $i$ of the Boosting Algorithm, we define

$$N_i(x) = \sum_{1 \leq j \leq i} R_{c_i}(x)$$
\end{definition}

Note that $N_i(x)$ is equal to the value $n_r - n_w$ we saw earlier in the filtering algorithm.

\begin{definition}
We define

$$M_i(x) = \begin{cases} 1, & \mbox{if }  N_i(x) \leq 0  \\ 0, & \mbox{if }  N_i(x) \geq \frac{1}{\epsilon\gamma}  \\ 1-\epsilon\gamma N_i(x),  & \mbox{otherwise} \end{cases}$$

\end{definition}

Similarly, note that $M_i(x)$ is nothing more than the probability of the filtering procedure at stage $i$  keeping $x$.

\begin{definition}
Define

$$|M_i(x)| = \sum_{x} M_i(x)$$
\end{definition}

\begin{definition}
Define

$$D_{M_i}(x) = \frac{M_i(x)}{|M_i|}$$

\end{definition}

The definition for $D_{M_i}(x)$ captures the actual probability of $x$ in the distribution $\mathcal{D}_i$ returned by this filtering procedure. We can think of $|M_i|$ as the total ``mass'' of all of the $M_i(x)$ probabilities.

\begin{definition}
We define the \textit{advantage} of a concept $c$ over one of the distributions $M_j$ as 

$$\mathrm{Adv}_c(M_i) = \sum_{x} R_c(x)M_j(x)$$

\end{definition}

Note that we can construct a large two-dimensional $2^n$ by $i$ matrix, where the rows are indexed by possible inputs $x$, where the columns are indexed by the number $j$ of the current stage, and where element $(x, j)$ is given by $R_{c_{j+1}}(x)M_j(x)$. If this is the case, note that $\mathrm{Adv}_{c_{j+1}}(M_i)$ is just the sum of the $j$th column of this matrix. It is also interesting to consider the row sums of this matrix.

\begin{definition}
Let

$$A_i(x) = \sum_{0 \leq j \leq i-1} R_{c_{j+1}}(x)M_j(x)$$

These $A_i(x)$ are the row sums of the $x$th row of the above matrix.
\end{definition}

We will also need a simple fact about the advantage.

\begin{theorem}\label{lbound}
If $\mathrm{Pr}_{x \in \mathcal{D}_i}[c(x) = f(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$, then $\mathrm{Adv}_c(M_i) \geq \gamma |M|$.
\end{theorem}

\begin{proof}
Note that $\mathrm{Adv}_c(M_i) = |M_i|(\mathrm{Pr}[c_i \mbox{ is right}] - \mathrm{Pr}[c_i \mbox{ is wrong}]) = |M_i|(2\mathrm{Pr}[c_i \mbox{ is right}] - 1) \geq \gamma|M_i|$.
\end{proof}

\begin{corollary}
If $\mathrm{Pr}_{x \in \mathcal{D}_i}[c(x) = f(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$ and $|M| \geq \epsilon 2^n$, then $\mathrm{Adv}_c(M_i) \geq \gamma \epsilon 2^n$.
\end{corollary}

\section{Correctness of the Boosting Algorithm}

In this section, we will provide most of a proof of correctness for the Boosting Algorithm. Our proof will rely on the following theorem, which we will prove next lecture.

\begin{theorem}\label{claim}
For all $x$ we have that

$$A_i(x) \leq \frac{1}{\epsilon\gamma} + \frac{\epsilon}{\gamma}{2}i$$

\end{theorem}



\begin{proof}[Proof of Theorem \ref{boost}]

Assume our algorithm has not terminated after $i$ steps. If this is true, then $|M_{i}| \geq \epsilon 2^i$, or otherwise the majority of the $c_i$ would agree with $f$ at least $(1-\epsilon)$ of the time and our algorithm would have already terminated.

Now, by Theorem \ref{lbound}, we have that

\begin{eqnarray*}
\sum_{x} A_{i+1}(x) &=& \sum_x\sum_{0\leq j \leq i} R_{c_{j+1}}(x)M_j(x)\\
&=& \sum_{0 \leq j \leq i} \mathrm{Adv}_{c_{j+1}}(M_j) \\
& \geq & (i+1)\epsilon\gamma 2^{n}
\end{eqnarray*}

On the other hand, by our unproven Theorem \ref{claim} above, we have that

$$\sum_{x} A_{i+1}(x) \leq 2^n \left(\frac{1}{\epsilon \gamma} + \frac{\epsilon\gamma}{2}(i+1)\right)$$

Combining these two bounds, we get that

$$(i+1)\epsilon\gamma 2^{n} \leq 2^n \left(\frac{1}{\epsilon \gamma} + \frac{\epsilon\gamma}{2}(i+1)\right)$$

Simplifying and solving for $i$, we find that

$$i \leq \dfrac{2}{\epsilon^2\gamma^2} - 1 = O\left(\frac{1}{\epsilon^2\gamma^2}\right)$$

This shows that after $\frac{2}{\epsilon^2\gamma^2}$ stages, our algorithm should terminate.
\end{proof}

\section{Next Lecture}

Next lecture, we will prove Theorem \ref{claim} above. The main idea behind the proof will be ``Elevator Lemma'', which says that if you take an elevator ride, the number of times you go up from the $i$th floor to the $i+1$th floor and the number of times you go down from the $i+1$th floor to the $i$th floor differ by at most $1$. We will use an analogous cancellation involving the $N_i$ functions to establish the lower bound in Theorem \ref{claim}.

We will also discuss the connection between the Boosting Algorithm, strong and weak PAC learning, and Yao's lemma for hardcore predicates.

\end{document}




