\HeadingLevelB{Platform Initialization (Booting)}
\label{sec:booting}

% Outcome: analyzing the chain of trust in a hardware/software co-design

When a computer is powered up, it undergoes a \textit{bootstrapping} process,
also called \textit{booting}, for simplicity. The boot process is a sequence of
steps that collectively initialize all hardware components and
load the system software into DRAM. An analysis of a system's security
properties must be aware of all pieces of software executed during the boot
process, and must account for the trust relationships that are created when a
software module loads another module.

This section outlines the details of the boot process needed to reason about
the security of a system based on the Intel architecture.
\citep{intel2010booting} provides a good reference for many of the booting
process's low-level details. While some specifics of the boot process depend on
the motherboard and components in a computer, this section focuses on the
high-level flow described by Intel's documentation.


\HeadingLevelC{The UEFI Standard}
\label{sec:efi}

The firmware in recent computers with Intel processors implements the
\textit{Platform Initialization} (PI) process in the \textit{Unified
Extensible Firmware Interface} (UEFI) specification \citep{forum2015uefi}. The
platform initialization follows the steps shown in Figure~\ref{fig:uefi} and
described below.

\begin{figure}[htpb]
  \centering
  \includegraphics[width=85mm]{figures/uefi.pdf}
  \caption{
    The phases of the Platform Initialization process in the UEFI
    specification.
  }
  \label{fig:uefi}
\end{figure}

The computer powers up, reboots, or resumes from sleep in the
\textit{Security phase} (SEC). The SEC implementation is responsible for
establishing a temporary memory store and loading the next stage of the
firmware into it. As the first piece of software that executes on the computer,
the SEC implementation is the system's root of trust, and performs the first
steps towards establishing the system's desired security properties.

For example, in a measured boot system (also known as trusted boot), all
software involved in the boot process is measured (cryptographically hashed,
and the measurement is made available to third parties, as described in
\S~\ref{sec:generic_software_attestation}). In such a system, the SEC
implementation takes the first steps in establishing the system's measurement,
namely resetting the special register that stores the measurement result,
measuring the  PEI implementation, and storing the measurement in the special
register.

SEC is followed by the \textit{Pre-EFI Initialization phase} (PEI), which
initializes the computer's DRAM, copies itself from the temporary memory
store into DRAM, and tears down the temporary storage. When the computer is
powering up or rebooting, the PEI implementation is also responsible for
initializing all non-volatile storage units that contain UEFI firmware and
loading the next stage of the firmware into DRAM.

PEI hands off control to the \textit{Driver eXecution Environment phase} (DXE).
In DXE, a loader locates and starts firmware drivers for the various components
in the computer. DXE is followed by a Boot Device Selection (BDS) phase, which
is followed by a Transient System Load (TSL) phase, where an EFI application
loads the operating system selected in the BDS phase. Last, the OS loader
passes control to the operating system's kernel, entering the Run Time (RT)
phase.

When waking up from sleep, the PEI implementation first initializes the
non-volatile storage containing the system snapshot saved while entering the
sleep state. The rest of the PEI implementation may use optimized
re-initialization processes, based on the snapshot contents. The DXE
implementation also uses the snapshot to restore the computer's state, such as
the DRAM contents, and then directly executes the operating system's wake-up
handler.


\HeadingLevelC{SEC on Intel Platforms}
\label{sec:uefi_sec_details}

% Power Sequencing and Reset Signal Timings: X99 Chipset datasheet S 8.5

Right after a computer is powered up, circuitry in the power supply and on the
motherboard starts establishing reference voltages on the power rails in a
specific order, documented as ``power sequencing'' \citep{intel2010bringup} in
chipset specifications such as \citep{intel2015x99chipset}. The rail powering up
the Intel ME (\S~\ref{sec:intel_me}) in the PCH is powered up significantly
before the rail that powers the CPU cores.

When the ME is powered up,  it starts executing the code in its boot ROM, which
sets up the SPI bus connected to the flash memory module
(\S~\ref{sec:motherboard}) that stores both the UEFI firmware and the ME's
firmware. The ME then loads its firmware from flash memory, which contains the
ME's operating system and applications.

% ICC Registers Under Intel ME Control: Series 100 Chipset datasheet S 4.5.1

After the Intel ME loads its software, it sets up some of the motherboard's
hardware, such as the PCH bus clocks, and then it kicks off the CPU's bootstrap
sequence. Most of the details of the ME's involvement in the computer's boot
process are not publicly available, but initializing the clocks is mentioned in
a few public documents~\citep{intel2015cpuio, purism2014meclock,
dice2011meclock, fitpc2014mehw}, and is made clear in firmware bringup guides,
such as the leaked confidential guide~\citep{intel2012mebringup} documenting
firmware bringup for Intel's Series 7 chipset.

% Initialization Overview: SDM S 9.1

The beginning of the CPU's bootstrap sequence is the SEC phase, which is
implemented in the processor circuitry. All logical processors (LPs) on the
motherboard undergo \textit{hardware initialization}, which invalidates the
caches (\S~\ref{sec:caching}) and TLBs (\S~\ref{sec:tlbs}), performs a
\textit{Built-In Self Test} (BIST), and sets all registers
(\S~\ref{sec:registers}) to pre-specified values.

% Multiple-Processor Initialization: SDM S 8.4
% BSP and AP Processors: SDM S 8.4.1
% MP Initialization Protocol Algorithms for MP Systems: SDM S 8.4.3
% An Ivy Bridge CPUID: family 06h, extended model 3, model 58, stepping 9

After hardware initialization, the LPs perform the Multi-Processor (MP)
initialization algorithm, which results in one LP being selected as the
\textit{bootstrap processor} (BSP), and all other LPs being classified as
\textit{application processors} (APs).

According to the SDM, the details of the MP initialization algorithm for recent
CPUs depend on the motherboard and firmware. In principle, after completing
hardware initialization, all LPs attempt to issue a special no-op transaction
on the QPI bus. A single LP will succeed in issuing the no-op, thanks to
the QPI arbitration mechanism, and to the UBox (\S~\ref{sec:cache_coherence})
in each CPU package, which also serves as a ring arbiter. The arbitration
priority of each LP is based on its APIC ID (\S~\ref{sec:interrupts}), which is
provided by the motherboard when the system powers up. The LP that issues the
no-op becomes the BSP. Upon failing to issue the no-op, the other LPs become
APs, and enter the \textit{wait-for-SIPI} state.

% Typical BSP Initialization Sequence: SDM S 8.4.4.1

Understanding the PEI firmware loading process is unnecessarily complicated by
the fact that the SDM describes a legacy process consisting of having the BSP
set its RIP register to 0xFFFFFFF0 (16 bytes below 4~GB), where the firmware is
expected to place a instruction that jumps into the PEI implementation.

% FIT pointer is at 4~GB - 0x18.
%   US 8,429,418 - 3:52-55, Fig 4
%   US 8,321,931 - Fig 5, 12:34-39

Recent processors do not support the legacy approach at all
\citep{reinauer2013fitpatch}. Instead, the BSP reads a word from address
0xFFFFFFE8 (24 bytes below 4~GB) \citep{intel2012uefihypervisor, datta2013acm},
and expects to find the address of a \textit{Firmware Interface Table} (FIT)
in the memory address space (\S~\ref{sec:address_spaces}), as shown in
Figure~\ref{fig:firmware_fit}. The BSP is able to read firmware contents from
non-volatile memory before the computer is initialized, because the initial SAD
(\S~\ref{sec:cache_coherence}) and PCH (\S~\ref{sec:motherboard})
configurations maps a region in the memory address space to the SPI flash module
(\S~\ref{sec:motherboard}) that stores the computer's firmware.

\begin{figure}[htpb]
  \centering
  \includegraphics[width=55mm]{figures/firmware_fit.pdf}
  \caption{
    The Firmware Interface Table (FIT) in relation to the firmware's memory
    map.
  }
  \label{fig:firmware_fit}
\end{figure}

The FIT \citep{qureshi2006fit} was introduced in the context of Intel's Itanium
architecture, and its use in Intel's current 64-bit architecture is described
in an Intel patent \citep{datta2013acm} and briefly documented in an obscure
piece of TXT-related documentation \citep{intel2010txtlab}. The FIT contains
\textit{Authenticated Code Modules} (ACMs) that make up the firmware, and
other platform-specific information, such as the TPM and TXT configuration
\citep{intel2010txtlab}.

The PEI implementation is stored in an ACM listed in the FIT. The processor
loads the PEI ACM, verifies the trustworthiness of the ACM's public key, and
ensures that the ACM's contents matches its signature. If the PEI passes the
security checks, it is executed. Processors that support Intel TXT only accept
Intel-signed ACMs \cite[p. 92]{futral2013servertxt}.


\HeadingLevelC{PEI on Intel Platforms}
\label{sec:uefi_pei_details}

\citep{intel2010booting} and \citep{coreboot2015manual} describe the
initialization steps performed by Intel platforms during the PEI phase, from
the perspective of a firmware programmer. A few steps provide useful context
for reasoning about threat models involving the boot process.

% Preventing Caching: SDM S 11.5.3

When the BSP starts executing PEI firmware, DRAM is not yet initialized.
Therefore, the PEI code starts executing in a \textit{Cache-as-RAM} (CAR) mode,
which only relies on the BSP's internal caches, at the expense of imposing
severe constraints on the size of the PEI's working set.

One of the first tasks performed by the PEI implementation is enabling DRAM,
which requires discovering and initializing the DRAM modules connected to the
motherboard, and then configuring the BSP's memory controllers
(\S~\ref{sec:cache_coherence}) and MTRRs (\S~\ref{sec:memory_io}). Most
firmware implementations use Intel's \textit{Memory Reference Code} (MRC) for
this task.

After DRAM becomes available, the PEI code is copied into DRAM and the BSP is
taken out of CAR mode. The BSP's LAPIC (\S~\ref{sec:interrupts}) is initialized
and used to send a broadcast
\textit{Startup Inter-Processor Interrupt}~(SIPI,~\S~\ref{sec:interrupts})
to wake up the APs. The interrupt vector in a SIPI indicates the memory address
of the AP initialization code in the PEI implementation.

% Typical AP Initialization Sequence: SDM S 8.4.4.2

The PEI code responsible for initializing APs is executed when the APs receive
the SIPI wake-up. The AP PEI code sets up the AP's configuration registers,
such as the MTRRs, to match the BSP's configuration. Next, each AP registers
itself in a system-wide table, using a memory synchronization primitive such
as a semaphore, in order to avoid two APs accessing the table concurrently.
After the AP initialization completes, each AP is suspended again,
and waits to receive an INIT Inter-Processor Interrupt from the OS kernel.

The BSP initialization code waits for all APs to register themselves into the
system-wide table, and then proceeds to locate, load and execute the firmware
module that implements DXE.
