\documentclass[10pt]{article}
\newtheorem{define}{Definition}
%\newcommand{\err}{\text{err}}
%\usepackage{psfig}

\oddsidemargin=0.15in
\evensidemargin=0.15in
\topmargin=-.5in
\textheight=9in
\textwidth=6.25in

\begin{document}
\input{preamble.tex}

%{lecture number}{lecture date}{Ronitt Rubinfeld}{Your name}
\lecture{12}{March 19, 2012}{Ronitt Rubinfeld}{Cheung Wang Chi}

Today, we are going to discuss the following:
\begin{itemize}
    \item Models for learning
    \item Brute force algorithm
    \item Learning conjunction functions
    \item Start on Learning theory and Fourier Transform
\end{itemize}
\section{Models for Learning}

In the model, the learner aims to learn an unknown function $f: D \rightarrow R$, where $|D|$ is finite, but $R$ is a general set, e.g. the set of real numbers. She has access to the \emph{example oracle} $\Ex(f)$, which outputs an example $(x, f(x))$, where $x \in_U D$. Upon obtaining a certain number of random examples, she will call an learning algorithm $\mathcal{A}$ that inputs the observed examples and outputs the hypothesis $h: D \rightarrow R$. \\

In general, one cannot hope to learn a random function. Indeed, when there is no restriction on $f$, it is too hard to learn $f$ exactly, i.e. to obtain $h \equiv f$, or even to obtain $h$ that is $\epsilon$-close to $f$. Therefore, we will restrict $f$ to be in a certain class $\mathcal{C}$ of functions, which is also known as the \emph{concept class}. Moreover, the learning algorithm $\mathcal{A}$ is usually specific to the concept class $\mathcal{C}$, as we will see in the case of learning conjunction functions.

\begin{define}
Given a hypothesis $h$ for $f$, the ``error of $h$ with respect to $f$'' is \err$(h) = \Pr_{x \in_{U} D}[f(x) \neq h(x)]$
\end{define}

Equivalently, $\err(h)$ is the smallest $\epsilon >0$ such that $h$ is $\epsilon$-close to $f$.

\begin{define}
A uniform distribution learning algorithm for the concept class $\mathcal{C}$ is an algorithm $\mathcal{A}$ such that 
\begin{enumerate}
\item Given $\epsilon, \delta > 0$, $\mathcal{A}$ accesses $\Ex(f)$ to obtain random samples $(x_1, f(x_1)), ..., (x_m, f(x_m))$.
\item $\mathcal{A}$ outputs hypothesis $h$ such that $\Pr[\err(h) \leq \epsilon] \geq 1 - \delta$, where $\err(h)$ is the error of $h$ with respect to $f$ defined earlier. 
\end{enumerate}
\end{define}

Here are the parameters of interest:
\begin{itemize}
\item The description of $h$:
\begin{itemize}
\item Does $h$ have to be in $\mathcal{C}$? We will not require $h \in \mathcal{C}$ in the future, but under the model for \emph{proper learning} we restrict $h \in \mathcal{C}$.  
\item We require that $h$ is at least relatively compact, i.e. $\size(h) = O(\log(|\mathcal{C}|))$, and $h$ can be efficiently evaluated.
\end{itemize}
\item The accuracy parameter $\epsilon$.
\item The confidence parameter $\delta$.
\item The runtime of $\mathcal{A}$.
\item The sample complexity $m$ of $\mathcal{A}$, which is the number of samples used by $\mathcal{A}$. 
\end{itemize}
It is desirable to have the runtime and the sample complexity bounded by $\ppoly(\log|D|, \frac{1}{\epsilon}, \frac{1}{\delta})$. In fact, we can replace $\frac{1}{\delta}$ with $\log\frac{1}{\delta}$ by repeating $\mathcal{A}$ and picking the best concept learnt.\\

\emph{Comment:} Our uniform distribution learning model generalizes naturally to the PAC model. By replacing the assumption of ``uniform distribution over $D$'' by ``general distribution $\mathcal{D}$ over $D$'', it is the PAC model. To be more precise, in the PAC model, a learner has access to example oracle $\Ex_{\mathcal{D}}(f)$, which generates $(x, f(x))$ with $x\in_\mathcal{D}D$. $\mathcal{D}$ is unknown to the learner. The learner aims to output hypothesis $h$ such that $\Pr_{\mathcal{D}}[h \textnormal{ is $\epsilon$-close to $f$}] \geq 1-\delta$.

\section{Brute Force Algorithm for Learning}

With reasonable sample complexity but no bound on runtime, for all $\mathcal{C}$ and $\epsilon, \delta >0$ there is a brute force algorithm for learning $f \in \mathcal{C}$ that achieves the accuracy parameter $\epsilon$ and the confidence parameter $\delta$:\\

\underline{Brute Force Algorithm}:
\begin{enumerate}
\item Draw $M = \frac{1}{\epsilon}\left(\log|\mathcal{C}| + \log\frac{1}{\delta}\right)$ random examples from $\Ex(f)$.
\item Search over all $h \in \mathcal{C}$ until we find one that is consistent with all labelled examples. Choose an arbitrary one if there are $> 1$ such $h$.
\end{enumerate}
\underline{Behaviour:}
\begin{itemize}
\item $f$ is always a candidate for $h$, since $f$ will always be consistent. Thus, the algorithm will definite output a hypothesis.
\item We would like to show that for a consistent $h$, $\Pr[\err(h)\leq \epsilon] \leq 1-\delta$, i.e. $h$ is not too far from $f$. Now, say $h$ is \emph{bad} if $\err(h) > \epsilon$, and we see that
$$\Pr[\textnormal{bad } h \textnormal{ is consistent the $M$ examples}] \leq (1 - \epsilon)^M$$

By union bound, we have 
\begin{eqnarray}
&&\Pr[\textnormal{Any bad $h$ consistent with the $M$ examples}]\nonumber\\
\leq &&|\mathcal{C}|(1-\epsilon)^M \nonumber\\
\leq &&|\mathcal{C}|\left((1 - \epsilon)^{1/\epsilon}\right)^{\log|\mathcal{C}| + \log\frac{1}{\delta}}\nonumber\\
\leq && \delta. \nonumber
\end{eqnarray}
\end{itemize}

Therefore, it is unlikely to have any bad $h$, and the algorithm has the required property. \qed \\

\emph{Remarks:} 
\begin{itemize}
\item Since $h \in \mathcal{C}$, this algorithm applies for the proper learning model.
\item We did not use the fact that $x \in_U D$, so the brute force algorithm also applies for the PAC model. 
\item Once we have a good $h$, we can
\begin{enumerate}
\item \underline{predict} values of $f$ on new random inputs (with same distribution as in the example oracle).
\item \underline{compress} the description of samples. Given samples $(x_1, f(x_1)), ..., (x_m, f(x_m))$, we can rewrite them as $(x_1, ..., x_m) \cup \left\{\textnormal{Description of $h$}\right\}$. Since the description of $h$ requires $O(\log|\mathcal{C}|)$ bits, it is a genuine compression if $m \geq \log|\mathcal{C}|$.
\end{enumerate} 
\end{itemize}

\section{Learning Conjunction Functions}

Let $\mathcal{C}$ be the family of conjunctions $f$ over $\cube$, for example $f(x) = x_i x_j x_k$. We cannot hope to learn $f\in \mathcal{C}$ with zero error. For example, to distinguish $f_1(x) = \prod^{n}_{i=1}x_i$ from $f_2(x) = 0$, we need to sample almost all points in general, since $f_1, f_2$ coincide for all points but $x = (1, ..., 1)$.\\

If we learn by the brute force algorithm, the sample complexity is $\frac{1}{\epsilon}\left(\log(2^n) + \log\left(\frac{1}{\delta}\right)\right) = \frac{n}{\epsilon} + \frac{1}{\epsilon}\log\left(\frac{1}{\delta}\right)$. But the runtime is an issue. Here is a polynomial time algorithm:

\begin{enumerate}
\item Draw $\ppoly(\frac{1}{\epsilon})$ examples to estimate $\Pr[f(x) = 1]$ to the additive error $\pm \frac{\epsilon}{4}$.
\item If the estimate is $< \frac{\epsilon}{2}$, output $h = 0$.
\item Otherwise, draw enough examples such that there are $k = \log(n/\delta)$ positive examples.
\item Let $V = \left\{i: x_i \textnormal{ is set in the same way in each positive example}\right\}$.
\item Output $h(x) = \bigwedge_{i \in V} y_i$, where $y_i = x_i$ if $x_i = 1$ in the positive examples, $y_i = \bar{x}_i$ if $x_i = 0$ in the positive examples. 
\end{enumerate}

In step (1), it can be shown by the Hoeffding inequality that the estimate has the required additive error bound with high probability. \\

In step (3), we have $\Pr[f(x) = 1] \geq \frac{\epsilon}{4}$, since the estimate $\geq \frac{\epsilon}{2}$ and the additive error is bounded by $\frac{\epsilon}{4}$. Therefore, in every $O(\frac{1}{\epsilon})$ examples, in expectation we will see a new positive example, i.e. $(x, f(x))$ such that $f(x) = 1$. By sampling $O(k/\epsilon)$ times, we will have $k$ positive examples in expectation, and in fact with high probability, for example by Chernoff inequality.\\

\underline{Behaviour:} For $i$ in the conjunction $f$, i.e. $x_i$ or $\bar{x}_i$ appears in $f$, then it must be set in the same way in each positive example. Therefore, $i \in V$. For $i$ not in the conjunction $f$, we know that
\begin{eqnarray}
&&\Pr[i \in V] \nonumber\\
&=& \Pr[x_i \textnormal{ is set in the same way in each of the $K$ examples}]\nonumber\\
&=& \frac{1}{2^{k-1}}.\nonumber
\end{eqnarray}
This implies that 
$$\Pr[\textnormal{Any $i$ not in $f$ survives}] \leq \frac{n}{2^{k-1}} < \delta,$$
since $k = \log(n/\delta)$. Now, the sample complexity is $O\left(\frac{1}{\epsilon}\log\left(\frac{n}{\delta}\right)\right)$, which is better than the brute force algorithm's. By working backwards, we see that this order bound is tight. \qed \\

\section{Learning and Fourier Representation}
For approximating the Fourier coefficient of an unknown Boolean function, we have the following lemma.
\begin{lemma}
For $f:\cube \rightarrow \left\{0, 1\right\}$, we can approximate any Fourier coefficient $\hat{f}(S)$ to within additive error $\gamma$ (i.e. $|\textnormal{Output} - \hat{f}(S)|\leq \gamma$) with probability $\geq 1-\delta$ with $O(\frac{1}{\gamma^2}\log{\frac{1}{\delta}})$ samples.
\end{lemma}
\begin{proof-sketch}
Use Hoeffding inequality and the fact that $\hat{f}(S) = 1 - 2\Pr[f = \chi_S]$
\end{proof-sketch}

It is thought to be unlikely in this model that one can efficiently find the ``heavy'' Fourier coefficients with non-trivial values. Exhaustive search is  intractable, and no other method is immediately clear. If we are dealing with a function class where most of the weight is in the coefficients with small $S$, we can exhaustively check these values. We will see this approach in work in the next lecture.
\end{document}




