\documentclass[10pt]{article}
\usepackage{enumerate, mathrsfs}
\usepackage[pdftex]{graphicx}
\newtheorem{define}{Definition}
%\newcommand{\Z}{{\mathbb{Z}}}
%\usepackage{psfig}

\oddsidemargin=0.15in
\evensidemargin=0.15in
\topmargin=-.5in
\textheight=9in
\textwidth=6.25in

\begin{document}
\input{preamble.tex}

%{lecture number}{lecture date}{Ronitt Rubinfeld}{Your name}
\lecture{19}{April 22, 2012}{Ronitt Rubinfeld}{Joshua Alman}

Today, we will:

\begin{itemize}
\item Finish with weak learning of monotone functions
\item Start proving that weak learnability implies strong learnability
\end{itemize}

\section{Learning Monotone Functions}
Recall that if $x,y \in \{ \pm 1 \} ^n$, then we write $x \leq y$ if for all coordinates $i \in [n]$, we have that $x_i \leq y_i$. Then:

\begin{definition}[Monotone functions]
    A function $f$ is \emph{monotone} if $f(x) \leq f(y)$ whenever $x \leq y$.
\end{definition}

We finish off our discussion of monotone functions from last lecture by proving:

\begin{theorem}
For any monotone function $f : \{ \pm 1 \}^n \rightarrow \{ \pm 1 \}$, there is a $g \in \{ +1, -1, \chi_{\{1\}}, \chi_{\{2\}}, \cdots, \chi_{\{n\}} \}$ such that

$$\Pr_{x \in \{ \pm 1 \}^n } \left[ f(x) = g(x) \right] \geq \frac{1}{2} + \Omega \left( \frac{1}{n} \right).$$
\end{theorem}

\begin{proof}
If $\Pr_{x \in \{ \pm 1 \}^n } \left[ f(x) = 1 \right] > \frac{3}{4}$, then the function $g(x) = 1$ satisfies the desired probability, and if $\Pr_{x \in \{ \pm 1 \}^n } \left[ f(x) = 1 \right] < \frac{1}{4}$, then the function $g(x) = -1$ does. It remains to show the theorem for when

$$\Pr_{x \in \{ \pm 1 \}^n } \left[ f(x) = 1 \right] \in \left[\frac{1}{4}, \frac{3}{4} \right].$$

From problem 5 of problem set 3, we know that since $f$ is monotone:

$$Inf_i(f) = \hat{f}(\{ i \}).$$

Thus, if we can show that for some coordinate $i$, $Inf_i(f) = \Omega \left( \frac{1}{n} \right)$, then we will have that:

$$\Pr \left[ f(x) = \chi_{\{ i \}}(x) \right] = \frac{\hat{f}(\{i\}) + 1}{2} = \frac{1}{2} + \Omega\left( \frac{1}{n} \right),$$

as desired. Thus, we will show that there is an $i$ such that $Inf_i(f) = \Omega \left( \frac{1}{n} \right)$ by using a different interpretation of $Inf_i(f)$.

In particular, we will use the hypercube graph interpretation of $Inf_i(f)$. Elements of $\{ \pm 1 \}^n$ are the vertices of the hypercube graph, and two vertices are adjacent if they have Hamming distance one. Last lecture, we colored the vertices: a vertex $x$ is red if $f(x) = 1$, and blue if $f(x) = -1$. Then, we found that:

$$Inf_i(f) = \frac{\hbox{total \# red-blue edges in }
  i\hbox{th direction}}{\hbox{total \# edges in } i\hbox{th
    direction}} = \frac{\hbox{total \# red-blue edges in }
  i\hbox{th direction}}{2^{n-1}} $$
  
We will bound the number of red-blue edges in the $i$ direction below by using a canonical paths.

\paragraph{Canonical Path Argument Plan}
\begin{enumerate}[1)]
\item Define a canonical path for every pair $(x,y)$ of (not necessarily adjacent) nodes such that $x$ is red and $y$ is blue.
\item Give an upper bound on the number of canonical paths passing through \underline{any} edge of the hypercube.
\item Combine these to conclude a lower bound on the total number of red-blue edges, and thus the number of such edges in a particular direction $i$.
\end{enumerate}

We begin with step 1:

\begin{definition}[Canonical Path]
For all pairs $(x,y)$ of nodes in the hyprcube, a \emph{canonical path from $x$ to $y$} scans bits from left to right, flipping bits where needed. Each flip corresponds to a step in the path.
\end{definition}
    
In this proof, we only care about canonical paths from $x$ to $y$ where $x$ is red and $y$ is blue, but the definition works in general.
    
For instance, if $x = -1,+1,+1,+1$ and $y = +1,-1,+1,-1$, then the canonical path between them is:
    \begin{center}
        \begin{tabular}{rrrrrr}
            $x=$ & -1 & +1 & +1 & +1 \\
            & {\bf +1} & +1 & +1 & +1 \\
            & +1 & {\bf -1} & +1 & +1 \\
            $y=$ & +1 & -1 & +1 & {\bf -1}
        \end{tabular}
    \end{center}
    
Notice that $x$ and $y$ above are not comparable (it is neither true that $x \leq y$ nor $y \leq x$), but there is still a canonical path between them. Furthermore, it is clear that each step in a canonical path is an edge in the hypercube graph, and so this defines a path between any pair f vertices in the hypercube.

How many canonical paths from a red vertex to a blue vertex are there? Since there is exactly one canonical path between every pair of nodes, this is just the number of pairs of one red vertex and one blue vertex. Since $\Pr_{x \in \{ \pm 1 \}^n } \left[ f(x) = 1 \right] \in \left[\frac{1}{4}, \frac{3}{4} \right]$, the number of blue nodes is at least $\frac{1}{4}$ the total number of nodes, as is the number of red nodes. Thus, the number of pairs of a red node and a blue node is at least

$$\left(\frac{1}{4}\cdot 2^n\right)^2 = \frac{2^{2n}}{16} = 2^{2n-4}.$$

Next, for each edge of the hypercube graph, how many canonical paths cross it? Consider any edge $(w, w^{\oplus j})$ of the hypercube. If this is on a canonical path between $x$ and $y$, then it must be the step that changes the $j$th bit. That means that the steps to change bits $1$ through $j-1$ have to have happened before this edge, and so $w_1 = y_1$, $w_2 = y_2$, $\cdots$ $w_{j-1} = y_{j-1}$. Similarly, bits $j+1$ through $n$ have not yet been changed on this canonical path, so they are still the same as $x$, meaning $w_{j+1} = x_{j+1}$, $\cdots$ $w_n = x_n$. Moreover, because flipping the $j$th bit of $w$ is on the path, we know that $x_j = w_j$ and $y_j = -w_j$. The only bits of $x$ and $y$ that we have not accounted for are the first $j-1$ bits of $x$ and the last $n-j-1$ bits of $y$. In fact, for any choice of these bits, $w$ will be on the path from $x$ to $y$. Thus, the total number of canonical paths containing the edge $(w, w^{\oplus j})$ is $2^{j-1} \cdot 2^{n-j-1} = 2^{n-1}$.

Finally, we can combine these two steps by noting that every path from a red node to a blue node must pass through at least one red-blue edge. Thus:

$$\hbox{\# of red-blue edges} \geq \frac{\hbox{\# of red-blue canonical paths}}{\hbox{max \# of canonical paths that cross an edge}} \geq \frac{2^{2n-4}}{2^{n-1}} = 2^{n-3},$$

and so by the Pigeonhole principle, there is a direction $i \in [n]$ such that

$$\hbox{\# of red-blue edges in direction } i \geq \frac{1}{n}2^{n-3}.$$

Thus, for that $i$:

$$Inf_i(f) \geq \frac{\frac{1}{n}2^{n-3}}{2^{n-1}} = \frac{1}{4n} = \Omega \left( \frac{1}{n} \right),$$

as desired.
\end{proof}

\section{Weak Learning}

What we just showed is not that monotone functions are learnable by a PAC, but that they are ``weakly learnable:''

\begin{definition}[Weak Learning]
An algorithm $\mathscr{A}$ \emph{weakly PAC learns} concept class $\mathscr{C}$ if for all $c \in \mathscr{C}$ and all distributions $\mathscr{D}$ on $\mathscr{C}$, there exists a $\gamma > 0$ and $\delta = \frac{1}{4}$ such that with probabilitiy $\geq 1 - \delta$, given examples of $c$ addording to $\mathscr{D}$, the algorithm $\mathscr{A}$ outputs an $h$ such that $\Pr_\mathscr{D} [h(x) = c(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$.
\end{definition}

A few notes about this definition, comparing it to the definition of (strong) learning from lecture 12:

\begin{itemize}
\item Here we set $\delta = \frac{1}{4}$ whereas in the definition of (strong) learning we pick any $\delta > 0$. This is because, like in problem 4 on problem set 3, the sample complexity of $\mathscr{A}$ could be made to be polynomial in $\log \left( \frac{1}{\delta} \right)$, which is relatively small, so we just treat $\delta$ as a constant to simplify the definition.

\item $\gamma$ should be thought of as a constant as well. We use $\frac{1}{2} + \frac{\gamma}{2}$ instead of $\frac{1}{2} + \gamma$ for convenience later; using either would yield the same definition.

\item In (strong) learning, we have that for input $\epsilon$, $\Pr_\mathscr{D} [h(x) = c(x)] \geq 1 - \epsilon$ rather than $\Pr_\mathscr{D} [h(x) = c(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$ (where $\gamma$ is fixed and not necessarily under the control of the user). This is the main difference between the two notions.
\end{itemize}

We put `strong' in parentheses in `(strong) learning' because in lecture 12 we defined it to be called `learning.' The word `strong' just helps us to contrast the two terms.

It seems on first glance that weak learnability is a weaker concept than learnability. In fact, this was conjectured to be the case for many years. However, it turns out to be false.

\begin{theorem}[Schapire]
Weak learnability implies (strong) learnability.
\end{theorem}

We will give the proof of Klivans and Servedio which is based on the techniques of Impagliazzo. We will give a boosting method that converts a weak learning algorithm into a (strong) learning algorithm. However, this method is tricky, and we will not be able to cover it in its entire detail today.

\subsection{An Intuitive Idea}
Here is a first idea for a boosting method: Suppose a weaker learner is only $51\%$ accurate.  We can first learn a weak hypothesis, filter away examples which are correctly classified, and then call the weak learner on the remaining $49\%$ of the data.  To increase the collective coverage of the hypotheses, we can repeat this many times until we have a set of hypotheses that cover most of the data. Unfortunately, this method has a big problem: once our set of hypotheses is complete, then given an unseen example, which hypothesis shall we use to evaluate it at?

The basic idea of the boosting algorithm we give is to construct a filtering mechanism so that the majority vote of the collective hypotheses works out.

\begin{proof}
We are trying to learn a function $f : \{ \pm 1 \}^n \rightarrow \{ \pm 1 \}$ from concept class $\mathscr{C}$. We are given a distribution $\mathscr{D}$ on $\mathscr{C}$ (we assume that $\mathscr{D}$ is the uniform distribution, althoguh this is not required for the theorem), and an algorithm $WL$, which is a weak learning algorithm for $\mathscr{C}$ with parameter $\frac{\gamma}{2}$ for any distribution. We want to design a (strong) learning algorithm for $\mathscr{C}$ with parameter $\epsilon$. Here is an iterative structure for our algorithm:

\vspace{10pt}

\underline{Stage 0}: (initialize) Set $\mathscr{D}_0 = \mathscr{D}$. Then, use $WL$ to generate, with high probability, a $c_0 \in \mathscr{C}$ such that $\Pr_{\mathscr{D}} [f(x) = c_0(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$.

\underline{Stage $i$}: At this point, we already have constructed $\mathscr{D}_0, \cdots, \mathscr{D}_{i-1}$ and $c_0, \cdots, c_{i-1}$. We then use a filtering procedure to construct $\mathscr{D}_i$. This filtering procedure will be fully described next lecture. We then run $WL$ on $\mathscr{D}_i$ to output, with high probability, a $c_i$ such that $\Pr_{\mathscr{D_i}} [f(x) = c_i(x)] \geq \frac{1}{2} + \frac{\gamma}{2}$.

\underline{After $T = O \left( \frac{1}{\gamma^2 \epsilon^2} \right)$ stages}: output $c = MAJ(c_1, c_2, \cdots, c_T)$.

\vspace{10pt}

\includegraphics[scale=0.45]{diagram.png}

We will describe in more detail how the filter works and complete the proof next lecture.
\end{proof}
\end{document}




