\HeadingLevelB{SGX Implementation Overview}
\label{sec:sgx_implementation_overview}

% ISCA 2015 SGX Slide 204:
%   "Close Collaboration between micro-architecture team and ISA team" -- no
%   mention of other teams e.g. the folks responsible for the execution units

An under-documented and overlooked feat achieved by the SGX design is that
implementing it on an Intel processor has a very low impact on the chip's
hardware design. SGX's modifications to the processor's execution
cores~(\S~\eref{I}{sec:cpu_core}) are either very small or completely nonexistent.
The CPU's uncore~(\S~\eref{I}{sec:cpu_die}, \S~\eref{I}{sec:cache_coherence}) receives
a new module, the  Memory Encryption Engine, which appears to be fairly
self-contained.

The bulk of the SGX implementation is relegated to the processor's
microcode~(\S~\eref{I}{sec:microcode}), which supports a much higher development
speed than the chip's electrical circuitry.


\HeadingLevelC{Execution Core Modifications}
\label{sec:sgx_core_modifications}

% Minimal PMH support for enclaves is MTRR to protect CMA access.
%   US 8,972,746 B2 - 31:65-67, 32:1-9
%   US 8,972,746 B2 - Table 8-1, column 31
% SERR register implemented in PMH
%   US 8,972,746 B2 - 5:59-61
% Outside enclave mode, PMH protects access to EPC using a range register
%   US 8,972,746 B2 - 31:31-36

At a minimum, the SGX design requires a very small modification to the
processor's execution cores~(\S~\eref{I}{sec:cpu_core}), in the Page Miss
Handler~(PMH,~\S~\eref{I}{sec:tlbs}).

The PMH resolves TLB misses, and consists of a fast path that relies on an FSM
page walker, and a microcode assist fallback that handles the edge cases
(\S~\eref{I}{sec:microcode_pmh}). The bulk of SGX's memory access checks, which are
discussed in \S~\ref{sec:sgx_access_protection}, can be implemented in the
microcode assist.

The only modification to the PMH hardware that is absolutely necessary to
implement SGX is developing an ability to trigger the microcode assist for all
address translations when a logical processor~(\S~\eref{I}{sec:cpu_core}) is in
enclave mode~(\S~\eref{I}{sec:sgx_threads}), or when the physical address produced
by the page walker FSM matches the Processor Reserved
Memory~(PRM,~\S~\eref{I}{sec:sgx_prm}) range.

The PRM range is configured by the PRM Range Registers~(\S~\eref{I}{sec:sgx_prm}),
which have exactly the same semantics as the Memory Type Range
Registers~(MTRRs,~\S~\eref{I}{sec:cacheability_config}) used to configure a
variable memory range. The page walker FSM in the PMH is already configured to
issue a microcode assist when the page tables are in uncacheable
memory~(\S~\eref{I}{sec:memory_io}). Therefore, the PRMRR can be represented as an
extra MTRR pair.


\HeadingLevelC{Uncore Modifications}
\label{sec:sgx_uncore_modifications}

% SAD/TAD entry protects against DMA snoops.
%   US 8,972,746 B2 - 6:15-19

The Intel's \textit{Software Development Manual}~(SDM)~\citep{intel2015sdm}
states that DMA transactions~(\S~\eref{I}{sec:motherboard}) that target the
PRM range are aborted by the processor. The SGX patents disclose that the PRMRR
protection against unauthorized DMA is implemented by having the SGX microcode
set up entries in the Source Address Decoder~(SAD) in the uncore CBoxes and in
the Target Address Decoder~(TAD) in the integrated Memory Controller~(MC).

\S~\eref{I}{sec:cache_coherence} mentions that Intel's Trusted Execution
Technology~(TXT)~\citep{grawrock2009txt} already takes advantage of the
integrated MC to protect a DRAM range from DMA. It is highly likely that the
SGX implementation reuses the mechanisms brought by TXT, and only requires the
extension of the SADs and TADs by one entry.

SGX's major hardware modification is the Memory Encryption Engine~(MEE) that is
added to the processor's uncore~(\S~\eref{I}{sec:cpu_die},
\S~\eref{I}{sec:cache_coherence}) to protect SGX's Enclave Page
Cache~(EPC,~\S~\eref{I}{sec:sgx_epc}) against physical attacks.

% ISCA 2015 SGX Slides 163-201

The MEE was first briefly described in the ISCA 2015 SGX
tutorial~\citep{intel2015iscasgx}. According to the information presented there,
the MEE roughly follows the approach introduced by Aegis \citep{suh2003aegis}
\citep{aegis_impl}, which relies on a variation of Merkle trees to provide the
EPC with confidentiality, integrity, and freshness
guarantees~(\S~\eref{I}{sec:crypto_primitives}). Unlike Aegis, the MEE uses
non-standard cryptographic primitives that include a slightly modified AES
operating mode~(\S~\eref{I}{sec:confidentiality_crypto}) and a
Carter-Wegman~\citep{carter1977mac, wegman1981mac}
MAC~(\S~\eref{I}{sec:integrity_crypto}) construction. The MEE was further described
in \citep{gueron2016sgxmee}.

% ISCA 2015 SGX Slide 167

Both the ISCA SGX tutorial and the patents state that the MEE is connected to
to the Memory Controller (MC) integrated in the CPU's uncore. However, all
sources are completely silent on further implementation details. The
MEE overview slide states that ``the Memory Controller detects [the] address
belongs to the MEE region, and routes transaction to MEE'', which suggests that
the MEE is fairly self-contained and has a narrow interface to the rest of the
MC.

% Intel SGX Interactions with S States: SDM S 42.15

% CMA protects EPC data when it is evicted to main memory from CPU package
%   US 8,972,746 B2 - 7:3-9
% Data in CMA lost when powered down
%   US 8,972,746 B2 - 6:37-45
% CMA uses a QPI link-level security protocol in multi-package systems
%   US 9,087,200 B2 - 27:31-37
%   US 8,972,746 B2 - 28:45-51

Intel's SGX patents use the name Crypto Memory Aperture (CMA) to refer to the
MEE. The CMA description matches the MEE and PRM concepts, as follows.
According to the patents, the CMA is used to securely store the EPC, relies on
crypto controllers in the MC, and loses its keys during deep sleep. These
details align perfectly with the SDM's statements regarding the MEE and PRM.

% EPC, EPCM, other SGX structures are in CMA
%   US 9,087,200 B2 - 5:15-25, 27:17-31
%   US 8,972,746 B2 - 5:10-20, 28:31-45

The Intel patents also disclose that the EPCM~(\S~\eref{I}{sec:sgx_epcm}) and
other structures used by the SGX implementation are also stored in the PRM.
This rules out the possibility that the EPCM requires on-chip memory resembling
the last-level cache~(\S~\eref{I}{sec:caching}, \S~\eref{I}{sec:cache_coherence}).

Last, the SGX patents shine a bit of light on an area that the official Intel
documentation is completely silent about, namely the implementation concerns
brought by computer systems with multiple processor chips. The patents state
that the MEE also protects the
Quick-Path Interconnect~(QPI,~\S~\eref{I}{sec:motherboard}) traffic using
link-layer encryption.


\HeadingLevelC{Microcode Modifications}
\label{sec:sgx_microcode_modifications}

% Most of SGX is implemented in microcode
%   US 8,972,746 B2 - 5:20-24, 5:48-58
% EPCM is managed by microcoded instructions
%   US 8,972,746 B2 - 29:43-44

According to the SGX patents, all SGX instructions are implemented in
microcode. This can also be deduced by reading the SDM's pseudocode for all
the instructions, and realizing that it is highly unlikely that any SGX
instruction can be implemented in 4 or fewer
micro-ops~(\S~\eref{I}{sec:out_of_order}), which is the most that can be
handled by the simple decoders used in the hardware fast paths
(S~\eref{I}{sec:microcode_role}).

% AEX implemented in microcode
%   US 8,972,746 B2 - 5:46-48

The Asynchronous Enclave Exit~(AEX,~\S~\eref{I}{sec:sgx_aex}) behavior is also
implemented in microcode. \S~\eref{I}{sec:microcode_structure} draws on an
assortment of Intel patents to conclude that hardware
exceptions~(\S~\eref{I}{sec:faults}), including both faults and interrupts,
trigger microcode events~(\S~\eref{I}{sec:microcode_structure}). It follows that
the SGX implementation can implement AEX by modifying the hardware exception
handlers in the microcode.

% ISCA 2015 SGX Slide 166
% Intel SGX Opt-In Configuration: SDM S 37.7.1

The SGX initialization sequence is also implemented in microcode. SGX is
initialized in two phases. First, it is very likely that the boot sequence in
microcode~(\S~\eref{I}{sec:microcode_sec}) was modified to initialize the registers
associated with the SGX microcode. The ISCA SGX tutorial states that the MEE'
keys are initialized during the boot process. Second, SGX instructions are
enabled by setting a bit in a Model-Specific
Register~(MSR,~\S~\eref{I}{sec:address_spaces}). This second phase involves
enabling the MEE and configuring the SAD and TAD to protect the PRM range. Both
tasks are amenable to a microcode implementation.

The SGX description in the SDM implies that the SGX implementation uses a
significant number of new registers, which are only exposed to microcode.
However, the SGX patents reveal that most of these registers are actually
stored in DRAM.

% Internal CREGs: SDM S 41.1.4
% TCS Layout has SAVE_DR7, SAVE_DEBUGCTL, SAVE_TF
%   US 8,972,746 B2 - Table 4-2, columns 10 and 11

For example, the patents state that each TCS~(\S~\eref{I}{sec:sgx_tcs}) has two
fields that receive the values of the DR7 and IA32\_DEBUGCTL registers when the
processor enters enclave mode~(\S~\eref{I}{sec:sgx_eenter}), and are used to
restore the original register values during enclave
exit~(\S~\eref{I}{sec:sgx_eexit}). The SDM documents these fields as ``internal
CREGs'' (CR\_SAVE\_DR7 and CR\_SAVE\_DEBUGCTL), which are stated to be
``hardware specific registers''.

% Package registers
%   US 8,972,746 B2 - Table 4-14, column 19
% LP registers
%   US 8,972,746 B2 - Table 4-15, column 19

The SGX patents document a small subset of the CREGs described in the SDM,
summarized in Table~\ref{fig:sgx_registers}, as microcode registers. While in
general we trust official documentation over patents, in this case we use the
CREG descriptions provided by the patents, because they appear to be more
suitable for implementation purposes.

% CR_XSAVE_PAGE_0 defined in AEX Operational Detail, SDM S 40.4.1

% EENTER caches TCS physical address
%   US 9,087,200 B2 - 12:19-24

From a cost-performance standpoint, the cost of register memory only seems to
be justified for the state used by the PMH to implement SGX's memory access
checks, which will be discussed in \S~\ref{sec:sgx_access_protection}. The
other pieces of state listed as CREGs are accessed so infrequently that storing
them in dedicated SRAM would make very little sense.

The SGX patents state that SGX requires very few hardware changes, and most of
the implementation is in microcode, as a positive fact. We therefore suspect
that minimizing hardware changes was a high priority in the SGX design, and
that any SGX modification proposals need to be aware of this priority.

\begin{landscape}
\begin{table*}[p]
  \centering
  \caption{
    The CREGs implemented by SGX.
  }
  \label{fig:sgx_registers}
  \begin{tabularx}{500pt}{| l | r | l | X |}
  \hline
  \textbf{SDM Name} & \textbf{Bits} & \textbf{Scope} & \textbf{Description} \\
  \hline
  CSR\_SGX\_OWNEREPOCH & 128 & CPU Chip Package &
      Used by \texttt{EGETKEY}~(\S~\eref{I}{sec:sgx_egetkey}) \\
  \hline
  CR\_ENCLAVE\_MODE & 1 & Logical Processor & 1 when executing code inside an
      enclave \\
  \hline
  CR\_ACTIVE\_SECS & 16 & Logical Processor & The index of the EPC page storing
      the current enclave's SECS \\
  \hline
  CR\_TCS\_LA & 64 & Logical Processor & The virtual address of the
      TCS~(\S~\eref{I}{sec:sgx_tcs}) used to enter~(\S~\eref{I}{sec:sgx_eenter}) the
      current enclave \\
  \hline
  CR\_TCS\_PH & 16 & Logical Processor & The index of the EPC page storing the
      TCS used to enter the current enclave \\
  \hline
  CR\_XSAVE\_PAGE\_0 & 16 & Logical Processor & The index of the EPC page
      storing the first page of the current SSA~(\S~\eref{I}{sec:sgx_ssa}) \\
  \hline
  \end{tabularx}
\end{table*}
\end{landscape}
