CS294-152 Lower Bounds 8/27/18 === Welcome! Course webpage: bit.ly/LB-COURSE Mondays 4:05pm to about 6:30pm. (Start 5 mins later, it's MIT time...) No lecture on the weeks of Labor Day (next week) and Veteran's Day (in November). I think the course webpage gives a lot of information, but let me go over it as well as some extra stuff. -- MAILING LIST: I'll maintain a mailing list for the course. If you are enrolled as a Berkeley student, you're automatically added. Otherwise email rrwill@berkeley.edu or rrw@mit.edu (whichever) to be added to the mailing list. --- Prereqs: You should have had some basic undergrad complexity and algorithms. CS170 and CS172 probably suffice. If you've taken grad complexity, that's great, but it's probably not a big deal if you haven't. More importantly is mathematical maturity, and a drive to learn the material. The first 6-7 chapters of Arora and Barak's "Computational Complexity: A Modern Approach" is probably enough. --- Workload. It will be low. I have tried to design this class so that the course work will directly help you benefit from the Simons semester. Only three requirements: 1. Come to class! We'll generally start around 4:05pm, take a short break around 5:15, and end around 6:30 (or when the instructor is exhausted, whichever comes first). There will also be three workshops during the Lower Bounds semester. Part of coming to class is the workshops! Boolean Devices Sep. 10 – Sep. 14, 2018 Interactive Complexity Oct. 15 – Oct. 19, 2018 Algebraic Methods Dec. 3 – Dec. 7, 2018 Your job: Pick >= 1 talk during each workshop that you'd really like to learn about. Attend the talk (even watching the video would be OK). Write me a short email describing the talk, the main statements, what you liked/disliked, what was unclear/clear. Think of it as writing a (good, detailed) Yelp review for an academic talk :) Your email will be due on the Monday after the workshop. Maybe(?) have partial lectures during the weeks of workshops. E.g. maybe I'll only speak for an hour, from 5:30pm to 6:30pm, when there's a workshop. This will mainly depend on the workshop schedules, which are currently not known (not even for the workshop that's two weeks from now!). This is partly what the mailing list is for: I'll send an announcement to the mailing list about the lecture plan during workshop weeks. 2. Scribe notes. Depending on how many students enroll, you'll have to scribe 1 (or maybe 2) lectures. Now, I will typically have typed-up notes already, so you will have a lot to draw from! Still, you will need to put them in latex format (I won't do that). You should also take your own notes as well: some point might come up that I did not think of in the notes I wrote! 3. Course project: - Must explore a topic related to lower bounds that we did not do in class. Could be related to something you wrote a "review" about! - Ideally, each project would contribute a little something to lower bounds that did not exist before: Could prove a new theorem/lemma/proposition, or write a short survey on a topic that does not have one yet (there are many such topics). - Project proposal a month from last day of course (October 26-ish) - Project milestone two weeks from last day (November 12-ish) - Project presentations on last day of course (November 26). 4. (OPTIONAL) Open problem sessions. - We may have sessions where we get together and try to make progress on an open problem or two. I did this last semester at MIT with fine-grained complexity, and it went pretty well. The tricky thing is that it looks difficult to find an open problem in lower bounds that could potentially be solved "by committee". We'll see... If you see any ideas for such an open problem, please don't hesitate to point them out! At any rate, if/when this happens, it will be totally optional. ==== What is this course about? See the bootcamp lectures :) Proving impossibility results in generic resource-bounded models of computation. The more generic, the better. The list of topics I plan to cover is heavily biased towards things I already know something about (mostly circuit complexity): Time-Space Tradeoffs and Relativization: SAT, simple functions with many outputs, oracles making classes equal Generic Circuit Complexity: ΘTheta(2^n/n) size, ΣSigma2P and MA lower bounds, MAEXP andΣSigma3EXP AC0 Lower Bounds: Hastad's Switching Lemma, PARITY not in AC0 AC0 modulo a prime: Razborov-Smolensky, MAJORITY not in AC0[2] ACC0 Lower Bounds: Beigel-Tarui, Easy Witness Lemmas, SAT Algorithms imply Circuit Lower Bounds TC0 Lower Bounds: Hajnal et al. for MAJ-MAJ, THR-MAJ and MAJ-THR lower bounds, Non-linear-size depth-3 Formula Lower Bounds, Neciporuk's branching program lower bound Valiant's depth reduction and matrix rigidity Natural Proofs: As a Barrier, As an Oracle (MCSP), Constructive and Useful = NEXP Lower Bounds Guest Lectures: Arithmetic Circuit Complexity, Data Structure Lower Bounds, ... ==== Time-Space Lower Bounds and the Relativization Barrier LOGSPACE = decision problems solvable by TMs with only O(log n) extra working space (beyond the input, which we assume is read-only) Q: LOGSPACE = P? LOGSPACE = NP? Widely believed that LOGSPACE != P != NP. Time-Space Tradeoffs: Given only S(n) space to solve a problem, can we prove a T(n)-time lower bound for it? E.g. Given O(log n) space to solve SAT, does it require super-polynomial time? Yes-answer is equivalent to LOGSPACE != NP. === Progress on LOGSPACE vs NP In the following, we may assume a random-access model where we can probe any desired input bit in O(1) time. (So we won't get a lower bound by "sweeping across the input tape" or anything like that.) NTIME[t(n)] = decision problems solvable in nondeterministic t(n)-time Open: 1. NTIME[n] in TIME[n polylog n]? 2. NTIME[n] in LOGSPACE? 3. SAT is in LOGSPACE? Questions 2 and 3 are equivalent, and both are equivalent to NP = LOGSPACE. Claim: NTIME[n] subset of LOGSPACE <=> NP = LOGSPACE. Proof: (<=) is easy. For (=>) direction: Def: TS[t,s] = problems solvable by a single algorithm running in time t and space s. t >= n, s >= log(n) Thm: For every polynomial p(n), If NTIME[n] is in TS[t(n),s(n)] then NTIME[p(n)] subseteq TS[t(p(n)),s(p(n))]. Proof: "By padding". Let f in NTIME[p(n)]. Define the "padded" function g(y) := f(x) if y=x10^{p(|x|)}, = 0 o.w. Note g in NTIME[n]. By assumption, there's TS[t(n),s(n)] alg B for g. So here's a TS[t(p(n)),s(p(n))] algorithm for f: A(x): run B on x10^{p(|x|)}, output answer. Note: A can't write down all of y=x10^{p(|x|)}, and run in s(p(n)) space! (What if s(n) = O(log n)?) But A only needs to simulate B on input y, and note that B has read-only access to y. Algorithm A can use a counter of O(log p(|x|)) bits which keeps track of which input bit of y that its simulation of B is currently reading. Note we can compute any bit of y very efficiently, given random-access to the input x. QED So NTIME[n] subseteq TS[n^k, log n] ==> for all polys p, NTIME[p(n)] in TS[p(n)^k, k log n] subseteq LOGSPACE ==> NP = LOGSPACE. QED Claim: SAT is in LOGSPACE <=> NTIME[n] in LOGSPACE. Follows from: SAT is NP-complete even under logspace reductions. It is known that: [W'07] For all eps > 0, NTIME[n] is *not* in TS[n^{2 cos(pi/7)-eps}, log n]. ==> SAT is not in TS[n^{2 cos(pi/7)-o(1)}, log n] Note: 2 cos(pi/7) is approximately 1.801... I won't show all the details of the proof of this result, but I will show you the major ingredients, and prove a weaker time-space lower bound. ========= Quantifier-Trading Proofs Let A be a decision problem. Recall P^A = problems solvable in ptime with oracle for A NP^A = problems solvable in nondeterministic ptime with oracle for A. For complexity class C, P^C = problems solvable in ptime with oracle for some A in C. NP^C = " nondeterministic ptime " P is "closed under composition": P^P = P. Polynomial Hierarchy (PH): the "closure of NP under composition" We can define a complexity class PH which contains NP and is "closed under composition", i.e. PH^{PH} = PH. PH = union_k NP^{NP^{...k...^NP} Essentially defined so that P=NP => P=PH (because P=NP implies NP is also "closed under composition") The method for proving lower bounds for SAT that I'll discuss shows how PH can be useful as a *computational model* even though it is completely "unrealistic"! [Fortnow '97, FLvMV'05]: Proof by contradiction 1. Assume NTIME[n] is in TS[n^c, log(n)] for some c > 1. Call this assumption (*) [N.B. this is basically equivalent to SAT in TS[n^c, log(n)], thanks to a very strong Cook-Levin theorem.] (*) ==> "Great" algs for NP-hard problems. 2. Use (*) as subroutine to get even greater algs for problems in PH ... 3. Profit! Contradict known LB, e.g. nondet time hierarchy thm. NOTE: This proof strategy is really algorithm design, in a world with "PH machines"! Just as nondeterministic machines are an unrealistic unimplementable model in reality, one can concoct machines based on things like NP^{NP}, things in PH. These machines have no physical counterpart in reality, but they can be used to prove limitations on deterministic algorithms which *do* have a physical counterpart! It's a nice example of how in mathematics one can "extend" the model of the world to allow for "ridiculous" objects, in order to say something about the "realistic" objects. Analogy: developing the complex numbers helps say things about the reals. "The shortest path between two truths in the real domain passes through the complex domain." -- Hadamard Def: f in Sigma_2 TIME[t(n)]: there's an O(n)-time TM M s.t. for all x, f(x) = 1 <=> (exists y of length t(|x|))(forall z of length t(|x|))[M(x,y,z) accepts] [Note: NP^{NP} = union_k Sigma_2 TIME[n^k], but we won't need this in the below.] Start with the complexity class NTIME[n^2]. (1) NTIME[n^2] subset TS[n^{2c},O(log n)], by padding the assumption (*) (2) For c >= 1, TS[n^{2c},O(log n)] subset SIGMA_2 TIME[n^{2c/2} log n]. [SPEED UP rule: Speed up a TS computation by adding two quantifiers] (3) (*) => SIGMA_2 TIME[n^{2c/2}] subset NTIME[n^{2c^2/2}] [We call this "SLOW DOWN" rule: Replace the universal quantifier with a deterministic computation] (4) (1),(2),(3) ==> NTIME[n^2] subset NTIME[n^{2c^2/2}] Suppose c^2 < 2. Then have NTIME[n^2] subset NTIME[n^{2-eps}] for some eps > 0! CONTRADICTION TO NONDETERMINISTIC TIME HIERARCHY! Thm [LV'99] NTIME[n] is not in TS[n^c,O(log n)] for all c < sqrt{2}. ==== (2) SPEED UP THM: Thm: For t(n) >= n and s(n) >= log(n), TS[t, s] is in SIGMA_2 TIME[t^{1/2}*s]. Think of a TS[t,s] computation as a "tableau" of t length and s width. More general problem: given configurations C and D of TM M, want to know if can reach D from C in t steps. Exist Guess L=ceil(t^{1/2}) configurations C0,C2,...,C_L. Univ pick i=0,...,L-1. Erase all but (Ci,Ci+1). if i=0, check C0=C (reject if not) if i=L-1, check CL=D (reject if not) simulate from Ci for ceil(t^{1/2}) steps, accept iff reach cfg Ci+1. ==== (3) SLOW DOWN THM: Thm: (*) ==> SIGMA_2 TIME[n^{kc/2}] subset NTIME[n^{kc^2/2}]. Pf: Very similar to P=NP => P=PH. Let f in Sig2 TIME[n^a]. Want to show f in NTIME[n^{ac}]. For x of length n, f(x) = 1 <=> (Exists n^a bits y)(Forall n^a bits z)[M(x,y,z) accepts] <-- R runs in O(n^a) time. Def. g(x,y) = 1 <=> |y|=|x|^a AND (Forall |x|^a bits z)[M(x,y,z) accepts]. Then, g in coNTIME[n], which is in TS[n^c,O(log n)] by (*). Let M be a TS machine computing g. Here's a new algorithm for f, running in NTIME[n^{ac}]: On x, GUESS y of length n^a. Run M(x,y) and output the answer. === [[[Bonus material, from Williams 2005]]] More sophisticated NTIME[n] lower bound: Suppose NTIME[n] subset TS[n^c,O(log n)] with c >= sqrt{2}. Start with PI_2 TIME[n]. PI_2 TIME[n] is in coNTIME[n^{c}] by slow down is in TS[n^{c^2},O(log n)] by slow down is in Sigma_2 TIME[n^{c^2/2} log n] by speed up Now we have: PI_2 TIME[n] is in Sigma_2 TIME[n^{c^2/2} log n]. If c < sqrt{2} then done (contradiction), o.w. => NEW SLOW DOWN: Sigma_3 TIME[n] is in Sigma_2 TIME[n^{c^2/2} log n] Use this slow-down to derive: Sigma_3 TIME[n] is in Sigma_2 TIME[n^{c^2/2} log n] is in NTIME[n^{c^3/2} log n] is in TS[n^{c^4/2} log n, O(log n)] is in Sigma_3 TIME[n^{c^4/6} log^2 n], improved speed up contradiction if c < 6^{1/4} ~ 1.565... slightly better! ==== Can generalize the lower bounds to a "time-space tradeoff", e.g. [W'13] T-S tradeoff: Suppose SAT is in TS[t(n),s(n)] Then t(n)*s(n) >= n^{phi} > n^{1.61} Note: If could prove t(n)*s(n) > n^2 then SAT not in TS[n,n] = TIME[n]. Can unify all known lower bounds under common "proof system" P The search for lower bound proofs can be feasibly implemented on a computer by solving LPs This reduction to LP can also be used to show limitations on the proof method [Buss-W '13] For all eps > 0, there's no proof in P of "NTIME[n] is *not* in TS[n^{2 cos(pi/7)+eps}, log n]" Nevertheless, it's widely believed that we ought to be able to show that SAT isn't in n^2 time and O(log n) space... ==== Oracles and Relativization: a Blessing and a Barrier. I'll talk about what these terms mean, how they apply to complexity theory, then we'll look at the SAT lower bounds we just proved, in light of relativization. Often, a theorem in complexity theory holds also when all the Turing machines / algs involved have access to a common oracle A. For example, take the universal simulation theorem: Universal Sim Thm: There is an algorithm U such that for all algorithms M, and inputs x and t, U(M,x,t) accepts <=> M(x) accepts within t steps. "Oracle" Universal Sim Thm: For all oracles A, there is an algorithm U^A such that for all algorithms M^A with oracle A, and inputs x and t, U^A(M,x,t) accepts <=> M^A(x) accepts within t steps. When this is possible, we say the theorem "relativizes"; it holds "relative to every oracle". This is a very powerful property! When you show that a theorem relativizes, you're generalizing the theorem to work for uncountably many different computational models, where solving various problems takes only one step in the model. Most computability/complexity theorems you've seen relativize: "Relativized" Halting Problem: HALT^A = {(M,x) | M with oracle for A halts on x} "Relativized" Undecidability: No Turing machine with oracle A decides HALT^A. "Relativized" Time: TIME^A(t(n)) = problems solvable in O(t(n)) time with an oracle for A. "Relativized" T.H.T. Thm: For all oracles A, there's f in TIME^A(t^c(n)) - TIME^A(t(n)). "Relativized" N.T.H.T. Thm: For all oracles A, there's f in NTIME^A(t^c(n)) - NTIME^A(t(n)). Prop: forall A, P^A subseteq NP^A. Prop: P^A = NP^A => PH^A = P^A These are the "blessings" of relativization: when a theorem relativizes, you get many theorems from the proof of one. Now let's talk about the barrier. The barrier comes about when you find two complexity classes C and D with the property: There's A,B such that C^A = D^A and C^B != D^B. What does this mean? It means that, in order to conclude how C compares to D, we require proof techniques that do *not* yield relativized theorems. We need "non-relativizing techniques" that are sensitive enough to the computational model that they *don't* hold wrt arbitrary oracles! Several such techniques are known, but we have not yet applied them to P vs NP.... Interestingly, there *is* an oracle A such that P^A = NP^A. We would say "there is an oracle relative to which P and NP are equal" This simple result already shows that if P != NP, we need non-relativizing techniques to prove it. Def. ExpSim := {(M,x,1^t) | M accepts x in <= 2^t steps} Thm: EXP = P^{ExpSim} = NP^{ExpSim} Pf: EXP is in P^{ExpSim}: Given M running in 2^{n^k} time, can simulate it on x by querying "(M,x,1^{|x|^k}) in ExpSim" P^{ExpSim} in NP^{ExpSim}: by earlier prop. NP^{ExpSim} in EXP: Given N^{ExpSim} running in nondet O(n^k) time, can exhaustively try all 2^{O(n^k)} computation paths, *and* simulate a query of ExpSim on any input of length up to O(n^k), in 2^{O(n^k)} time. QED Maybe this suggests that P = NP, because there's an oracle for which it's true? There's also another oracle relative to which P != NP, so also proving P = NP also requires non-relativizing methods. Thm: There is a B such that P^{B} != NP^B Pf: We'll define a language of the form L_B = {1^n | there's a string of length n in B} Observation: For every B, L_B is in NP^B: "Given 1^n, guess an n-bit string y, if (y in B) then accept else reject." We'll design B so that, for all polynomial-time TMs with an oracle for B, there's an n such that either: -- the ptime TM with B says "YES" (but we put no string of length n in B) -- or the TM says "NO" (but we "hid" some string of length n in B from the poly-time TM). Have to be careful that our definition isn't circular! We first need an enumeration of all languages in P^B, so we can "hide" strings in B from all of them. Let M^B_1,M^B_2,... be an enumeration of all TMs with an oracle for B. (Note: we don't include a description of B in the code! Since all TMs use the same oracle, we will just enumerate all possible TMs that include the special states q_?, q_1, q_0 along with the other usual objects.) For each i, define A^B_i to be M^B_i augmented with a counter that cuts off the machine and rejects after n^i+i steps, on all inputs of length n. Recall L(M^B_i) = set of strings accepted by M^B_i. Claim: P^B = {L(A^B_i} | i=1,2,...} Proof: Clearly, every A^B_i runs in polynomial time, so L(A^B_i) in P. For the other direction, every language in P^B has some machine M^B_i running in time at most n^k+k for some k. In fact, there are infinitely many M^B_{i_1},M^B_{i_2}, ... equivalent to M^B_i, by simply padding the code of M^B_i with "dummy" states that don't do anything; all of them run in time n^k+k. Pick j such that i_j > k, and M^B_{i_j} runs in time n^{i_j}+i_j > n^k+k before rejecting. Thus L(M^B_{i_j}) = L(A^B_i). QED Now that we have an enumeration, let's construct the oracle B. We'll do it in infinitely many stages. Stage i will ensure that A^B_i does not decide L_B. Stage i: (0) In previous stages, we have a record of all strings queried by A^B_1,...,A^B_{i-1} and their answers. (1) Want to get an input length n_i such that A^B_i outputs the wrong answer on 1^{n_i}. Let n_1 = 1. For i > 1, choose an n_i greater than the lengths of all strings queried by A^B_1,...,A^B_{i-1} in previous stages (note, each A^B_j queries at most (n_j)^j+j strings, so n_i is always well-defined). (2) Run A^B_i(1^{n_i}) for (n_i)^i+i steps, and answer "no" to all B-queries that weren't asked in previous stages (otherwise, be consistent with any previous query answers; note these queries must all be shorter than n_i). (3a) If A^B_i(1^{n_i}) accepts, put NO n_i-bit strings in B. (3b) If A^B_i(1^{n_i}) rejects, put SOME n_i-bit string in B that was not queried (this is our string "hiding"). Note: A^B_i can query at most (n_i)^i+i strings, so there are exponentially many n_i-bit strings it did not query! End stage i. Observe: for all i, A^B_i(1^{n_i}) accepts <=> there is no y of length n_i such that y is in B <=> 1^{n_i} is NOT in L_B. Therefore A^B_i does not decide L_B, and L_B is not in P^B. Note: above proof even works to show that for every k, there's an oracle B such that NP^B is not in TIME[2^{n^k}]. === Moral: No proof of P=NP or P!=NP can relativize. === For practically every pair of complexity classes C and D for which there is no separation known, we have oracles A making them equal! Some of the craziest examples are: [Heller'86] There's an B s.t. EXP^{NP^B} = BPP^B (exptime with an NP oracle equals randomized P) even though EXP^A != P^A for all A. [Buhrman-Fenner-Fortnow-Torenvliet '01] There's an B such that NEXP^B = P^{RP^B} even though NEXP^A != NP^A for all A. Therefore, resolving even these embarassing open problems will require techniques that are sensitive to the basic operations of the machine: the techniques must break down when arbitrary oracles are inserted in the model. === Does NTIME[n] in TS[n^{1.4},O(log n)] relativize? === So far I've proved one lower bound for you. Does it relativize? Oracles in TS[n^c,O(log n)]: have a special write-only oracle tape whose content doesn't count towards space bound, and oracle tape is erased after each query. Thm: There is an oracle A s.t. NTIME^A[n] is in TS^A[n^{1.1},O(log n)]. Moral: the time-space lower bound for SAT does not relativize! Pf: Let M_1^?,M_2^?,... be an enumeration of nondeterministic n^{1.01}-time machines with an oracle. In particular, suppose M_i^? runs in at most i*n steps on inputs of length n. Oracle A is constructed in stages. Stage n (handling n-bit strings): ---- For all i=1,...,log(n) and strings x of length n, Run M_i^A on x, gathering all the queries asked over all possible computation paths. Set A to answer "no" on all queries were not asked in previous stages. Let S_n be the set of pairs (i,x) such that M_i^A accepts x. For all (i,x) in S_n, set A to answer "yes" on 0^{n^{1.1}+n*log(n)}1 (i,x). End for Note that none of the strings added to A are too long to have been queried by any M_i on the x's. (They have length at least n^{1.1}+n*log(n), but M_i ran in <= n*log(n) time on inputs of length n.) Thus our oracle A is well-defined on each string. However, for any nondeterministic n^{1.01}-time M_i with an oracle for A, it is easy to simulate it, in time n^{1.1} and space O(log n) with oracle A: On input x of length n, use a counter to write the string 0^{n^{1.1}+n*log(n)}1 (i,x) on the write-only oracle tape, then call A on the string and output the answer. QED We're exploiting the fact that the TS class can call the oracle on strictly longer strings than the NTIME class can. Q: What part of NTIME(n) not in TS(n^{1.4},O(log n)) does not relativize? A: The speed-up theorem! (Think about it...) END OF LECTURE ============================================================================================