\HeadingLevelB{The Memory Layout of an SGX Enclave}
\label{sec:sgx_enclave_layout}

SGX was designed to minimize the effort required to convert application code to
take advantage of enclaves. History suggests this is a wise decision, as a
large factor in the continued dominance of the Intel architecture is its
ability to maintain backward compatibility. To this end, SGX enclaves were
designed to be conceptually similar to the leading software modularization
construct, dynamically loaded libraries, which are packaged as \texttt{.so}
files on Unix, and \texttt{.dll} files on Windows.

For simplicity, we describe the interaction between enclaves and non-enclave
software assuming that each enclave is used by exactly one application process,
which we shall refer to as the enclave's \textit{host process}. We do note,
however, that the SGX design does not explicitly prohibit multiple application
processes from sharing an enclave.


\HeadingLevelC{The Enclave Linear Address Range (ELRANGE)}
\label{sec:sgx_elrange}

% SGX Enclave Control Structure (SECS): SDM S 38.7

Each enclave designates an area in its virtual address space, called the
\textit{enclave linear address range} (ELRANGE), which is used to map the code
and the sensitive data stored in the enclave's EPC pages. The virtual address
space outside ELRANGE is mapped to access non-EPC memory via the same virtual
addresses as the enclave's host process, as shown in
Figure~\ref{fig:sgx_elrange}.

\begin{figure}[htpb]
  \centering
  \includegraphics[width=85mm]{figures/sgx_elrange.pdf}
  \caption{
    An enclave's EPC pages are accessed using a dedicated region in the
    enclave's virtual address space, called ELRANGE. The rest of the virtual
    address space is used to access the memory of the host process. The memory
    mappings are established using the page tables managed by system software.
  }
  \label{fig:sgx_elrange}
\end{figure}

The SGX design guarantees that the enclave's memory accesses inside ELRANGE
obey the virtual memory abstraction~(\S~\ref{sec:paging_concepts}), while
memory accesses outside ELRANGE receive no guarantees. Therefore, enclaves must
store all of their code and private data inside ELRANGE, and must consider the
memory outside ELRANGE to be an untrusted interface to the outside world.

The word ``linear'' in ELRANGE references the linear addresses produced by the
vestigial segmentation feature~(\S~\ref{sec:segments}) in the 64-bit Intel
architecture. For most purposes, ``linear'' can be treated as a synonym for
``virtual''.

ELRANGE is specified using a base (the BASEADDR field) and a size (the SIZE)
in the enclave's SECS~(\S~\ref{sec:sgx_secs}). ELRANGE must meet the same
constraints as a variable memory type range (\S~\ref{sec:cacheability_config})
and as the PRM range~(\S~\ref{sec:sgx_prm}), namely the size must be a power of
2, and the base must be aligned to the size. These restrictions are in place so
that the SGX implementation can inexpensively check whether an address belongs
to an enclave's ELRANGE, in either hardware~(\S~\ref{sec:cacheability_config})
or software.

When an enclave represents a dynamic library, it is natural to set ELRANGE to
the memory range reserved for the library by the loader. The ability to access
non-enclave memory from enclave code makes it easy to reuse existing library
code that expects to work with pointers to memory buffers managed by code in
the host process.

Non-enclave software cannot access PRM memory. A memory access that resolves
inside the PRM results in an aborted transaction, which is undefined at an
architectural level, On current processors, aborted writes are ignored, and
aborted reads return a value whose bits are all set to 1. This comes into play
in the scenario described above, where an enclave is loaded into a host
application process as a dynamically loaded library. The system software maps
the enclave's code and data in ELRANGE into EPC pages. If application software
attempts to access memory inside ELRANGE, it will experience the abort
transaction semantics. The current semantics do not cause the application to
crash (e.g., due to a Page Fault), but also guarantee that the host application
will not be able to tamper with the enclave or read its private information.


\HeadingLevelC{SGX Enclave Attributes}
\label{sec:sgx_secs_attributes}

The execution environment of an enclave is heavily influenced by the value of
the ATTRIBUTES field in the enclave's SECS~(\S~\ref{sec:sgx_secs}). The rest of
this work will refer to the field's sub-fields, shown in
Table~\ref{fig:sgx_secs_attributes}, as \textit{enclave attributes}.

% ATTRIBUTES: SDM S 37.8.1

\begin{table}[hbt]
  \centering
  \caption{
    An enclave's attributes are the sub-fields in the ATTRIBUTES field of the
    enclave's SECS. This table shows a subset of the attributes defined in the
    SGX documentation.
  }
  \label{fig:sgx_secs_attributes}
  \begin{tabularx}{\columnwidth}{| l | r | X |}
  \hline
  \textbf{Field} & \textbf{Bits} & \textbf{Description} \\
  \hline
  DEBUG & 1 & Opts into enclave debugging features. \\
  \hline
  XFRM & 64 & The value of XCR0~(\S~\ref{sec:registers}) while this enclave's
              code is executed. \\
  \hline
  MODE64BIT & 1 & Set for 64-bit enclaves. \\
  \hline
  \end{tabularx}
\end{table}

The most important attribute, from a security perspective, is the DEBUG flag.
When this flag is set, it enables the use of SGX's debugging features for this
enclave. These debugging features include the ability to read and modify most
of the enclave's memory. Therefore, DEBUG should only be set in a development
environment, as it causes the enclave to lose all SGX security guarantees.

SGX guarantees that enclave code will always run with the XCR0
register~(\S~\ref{sec:registers}) set to the value indicated by
\textit{extended features request mask}~(XFRM). Enclave authors are expected to
use XFRM to specify the set of architectural extensions enabled by the compiler
used to produce the enclave's code. Having XFRM be explicitly specified allows
Intel to design new architectural extensions that change the semantics of
existing instructions, such as Memory Protection Extensions (MPX), without
having to worry about the security implications on enclave code that was
developed without an awareness of the new features.

The MODE64BIT flag is set to true for enclaves that use the 64-bit Intel
architecture. From a security standpoint, this flag should not even exist, as
supporting a secondary architecture adds unnecessary complexity to the SGX
implementation, and increases the probability that security vulnerabilities
will creep in. It is very likely that the 32-bit architecture support was
included due to Intel's strategy of offering extensive backwards compatibility,
which has paid off quite well so far.

In the interest of mental sanity, this work does not analyze the behavior of
SGX for enclaves whose MODE64BIT flag is cleared. However, a security
researcher who wishes to find vulnerabilities in SGX may study this area.

Last, the INIT flag is always false when the enclave's SECS is created. The
flag is set to true at a certain point in the enclave lifecycle, which will be
summarized in \S~\ref{sec:sgx_enclave_lifecycle}.


\HeadingLevelC{Address Translation for SGX Enclaves}
\label{sec:sgx_paging}

% Access Control Requirements: SDM S 38.3
% Interactions with VMX: SDM S 42.5, S 42.5.{1,2,3,4,5}

Under SGX, the operating system and hypervisor are still in full control of the
page tables and EPTs, and each enclave's code uses the same address translation
process and page tables~(\S~\ref{sec:paging}) as its host application. This
minimizes the amount of changes required to add SGX support to existing system
software. At the same time, having the page tables managed by untrusted system
software opens SGX up to the address translation attacks described in
\S~\ref{sec:address_translation_attacks}. As future sections will reveal, a
good amount of the complexity in SGX's design can be attributed to the need to
prevent these attacks.

SGX's active memory mapping attacks defense mechanisms revolve around ensuring
that each EPC page can only be mapped at a specific virtual
address~(\S~\ref{sec:segments}). When an EPC page is allocated, its intended
virtual address is recorded in the EPCM entry for the page, in the ADDRESS
field.

When an address translation (\S~\ref{sec:paging}) result is the physical
address of an EPC page, the CPU ensures\footnote{A mismatch triggers a general
protection fault (\#GP, \S~\ref{sec:faults}).} that the virtual address given
to the address translation process matches the expected virtual address
recorded in the page's EPCM entry.

SGX also protects against some passive memory mapping attacks and fault
injection attacks by ensuring that the access permissions of each EPC page
always match the enclave author's intentions. The access permissions for each
EPC page are specified when the page is allocated, and recorded in the
\textit{readable}~(R), \textit{writable}~(W), and \textit{executable}~(X)
fields in the page's EPCM entry, shown in
Table~\ref{fig:sgx_epcm_access_fields}.

\begin{table}[hbt]
  \centering
  \caption{
    The fields in an EPCM entry that indicate the enclave's intended virtual
    memory layout.
  }
  \label{fig:sgx_epcm_access_fields}
  \begin{tabularx}{\columnwidth}{| l | r | X |}
  \hline
  \textbf{Field} & \textbf{Bits} & \textbf{Description}\\
  \hline
  ADDRESS & 48 & the virtual address used to access this page\\
  \hline
  R & 1 & allow reads by enclave code\\
  \hline
  W & 1 & allow writes by enclave code\\
  \hline
  X & 1 & allow execution of code inside the page, inside enclave\\
  \hline
  \end{tabularx}
\end{table}

When an address translation (\S~\ref{sec:paging}) resolves into an EPC page,
the corresponding EPCM entry's fields override the access permission attributes
(\S~\ref{sec:page_table_attributes}) specified in the page tables. For example,
the W field in the EPCM entry overrides the writable (W) attribute, and the X
field overrides the disable execution (XD) attribute.

It follows that an enclave author must include memory layout information along
with the enclave, in such a way that the system software loading the enclave
will know the expected virtual memory address and access permissions for each
enclave page. In return, the SGX design guarantees to the enclave authors that
the system software, which manages the page tables and EPT, will not be able to
set up an enclave's virtual address space in a manner that is inconsistent with
the author's expectations.

The \texttt{.so} and \texttt{.dll} file formats, which are SGX's intended
enclave delivery vehicles, already have provisions for specifying the virtual
addresses that a software module was designed to use, as well as the desired
access permissions for each of the module's memory areas.

Last, a SGX-enabled CPU will ensure that the virtual memory inside
ELRANGE~(\S~\ref{sec:sgx_elrange}) is mapped to EPC pages. This prevents the
system software from carrying out an address translation attack where it maps
the enclave's entire virtual address space to DRAM pages outside the PRM, which
do not trigger any of the checks above, and can be directly accessed by the
system software.


\HeadingLevelC{The Thread Control Structure (TCS)}
\label{sec:sgx_tcs}

% Thread Control Structure (TCS): SDM S 38.8, S 38.8.{1,2,3,4}

The SGX design fully embraces multi-core processors. It is possible for
multiple logical processors~(\S~\ref{sec:cpu_die}) to concurrently execute the
same enclave's code concurrently, via different threads.

The SGX implementation uses a \textit{Thread Control Structure}~(TCS) for each
logical processor that executes an enclave's code. It follows that an enclave's
author must provision at least as many TCS instances as the maximum number of
concurrent threads that the enclave is intended to support.

Each TCS is stored in a dedicated EPC page whose EPCM entry type is PT\_TCS.
The SDM describes the first few fields in the TCS. These fields are considered
to belong to the architectural part of the structure, and therefore are
guaranteed to have the same semantics on all processors that support SGX.
The rest of the TCS is not documented.

% Access Control Requirements: SDM S 38.3
% EDBGRD, EDBGWR: SDM S 41.3

The contents of an EPC page that holds a TCS cannot be directly accessed, even
by the code of the enclave that owns the TCS. This restriction is similar to the
restriction on accessing EPC pages holding SECS instances. However, the
architectural fields in a TCS can be read by enclave debugging instructions.

The architectural fields in the TCS lay out the context
switches~(\S~\ref{sec:registers}) performed by a logical processor when it
transitions between executing non-enclave and enclave code.

For example, the OENTRY field specifies the value loaded in the instruction
pointer (RIP) when the TCS is used to start executing enclave code, so the
enclave author has strict control over the entry points available to enclave's
host application. Furthermore, the OFSBASGX and OFSBASGX fields specify the
base addresses loaded in the FS and GS segment
registers~(\S~\ref{sec:segments}), which typically point to Thread Local
Storage (TLS).


\HeadingLevelC{The State Save Area (SSA)}
\label{sec:sgx_ssa}

% Interactions with the Processor Extended State and Miscellaneous State:
%     SDM S 42.7
% Requirements and Architecture Overview: SDM S 42.7.1

% State Save Area (SSA) Frame: SDM S 38.9

When the processor encounters a hardware exception~(\S~\ref{sec:faults}), such
as an interrupt~(\S~\ref{sec:interrupts}), while executing the code inside an
enclave, it performs a privilege level switch (\S~\ref{sec:faults}) and invokes
a hardware exception handler provided by the system software. Before executing
the exception handler, however, the processor needs a secure area to store the
enclave code's execution context~(\S~\ref{sec:registers}), so that the
information in the execution context is not revealed to the untrusted system
software.

% Relevant Fields in Various Data Structures: SDM S 42.7.2
% SECS.SSAFRAMESIZE: SDM S 42.7.2.2

In the SGX design, the area used to store an enclave thread's execution context
while a hardware exception is handled is called a \texttt{State Save
Area}~(SSA), illustrated in Figure~\ref{fig:sgx_enclave_layout}. Each TCS
references a contiguous sequence of SSAs. The \textit{offset of the SSA
array}~(OSSA) field specifies the location of the first SSA in the enclave's
virtual address space. The \textit{number of SSAs}~(NSSA) field indicates the
number of available SSAs.

\begin{figure}[!htb]
  \centering
  \includegraphics[width=85mm]{figures/sgx_enclave_layout.pdf}
  \caption{
    A possible layout of an enclave's virtual address space. Each enclave has a
    SECS, and one TCS per supported concurrent thread. Each TCS points to a
    sequence of SSAs, and specifies initial values for RIP and for the base
    addresses of FS and GS.
  }
  \label{fig:sgx_enclave_layout}
\end{figure}

Each SSA starts at the beginning of an EPC page, and uses up the number of EPC
pages that is specified in the SSAFRAMESIZE field of the enclave's SECS. These
alignment and size restrictions most likely simplify the SGX implementation by
reducing the number of special cases that it needs to handle.

% Relevant Fields in Various Data Structures: SDM S 42.7.2
% SECS.ATTRIBUTES.XFRM: SDM S 42.7.2.1

An enclave thread's execution context consists of the general-purpose registers
(GPRs) and the result of the XSAVE instruction~(\S~\ref{sec:registers}).
Therefore, the size of the execution context depends on the requested-feature
bitmap~(RFBM) used by to XSAVE. All code in an enclave uses the same RFBM,
which is declared in the XFRM enclave
attribute~(\S~\ref{sec:sgx_secs_attributes}). The number of EPC pages reserved
for each SSA, specified in SSAFRAMESIZE,
must\footnote{\texttt{ECREATE}~(\S~\ref{sec:sgx_ecreate}) fails if SSAFRAMESIZE
is too small.} be large enough to fit the XSAVE output for the feature bitmap
specified by XFRM.

SSAs are stored in regular EPC pages, whose EPCM page type is PT\_REG.
Therefore, the SSA contents is accessible to enclave software. The SSA layout
is architectural, and is completely documented in the SDM. This opens up
possibilities for an enclave exception handler that is invoked by the host
application after a hardware exception occurs, and acts upon the information in
a SSA.
