\HeadingLevelC{Security Monitor}
\label{sec:monitor}

The security monitor receives control after \texttt{mroot} finishes setting up
the attestation measurement chain.

The monitor provides API calls to the OS and enclaves for
\textbf{DRAM region allocation} and \textbf{enclave management}. The monitor
guards sensitive registers, such as the page table base register
(\texttt{ptbr}) and the allowed DMA range (\texttt{dmarbase} and
\texttt{dmarmask}). The OS can set these registers via monitor calls that
ensure the register values are consistent with the current DRAM region
allocation.


\HeadingLevelD{DRAM Regions}

Figure~\ref{fig:dram_region_states} shows the DRAM region allocation state
transition diagram. After the system boots up, all DRAM regions are allocated
to the OS, which can free up DRAM regions so it can re-assign them to enclaves
or to itself. A DRAM region can only become free after it is blocked by its
owner, which can be the OS or an enclave.  While a DRAM region is blocked, any
address translations mapping to it cause page faults, so no new TLB entries
will be created for that region. Before the OS frees the blocked region, it
must flush all TLBs in order to remove any stale entries for the region.

\begin{figure}[hbt]
  \centering
  \includegraphics[width=80mm]{figures/dram_region_states.pdf}
  \caption{
    DRAM region allocation states and API calls.
  }
  \label{fig:dram_region_states}
%\EndFigure
\end{figure}


The monitor ensures that the OS performs TLB shootdowns, using a global
\textit{block clock}. When a region is blocked, the block clock is incremented,
and the current block clock value is stored in the metadata associated with the
DRAM region (shown in Figure~\ref{fig:monitor_layout}). When a core's TLB is
flushed, that core's flush time is set to the current block clock value. When
the OS asks the monitor to free a blocked DRAM region, the monitor verifies
that no core's flush time is lower than the block clock value stored in the
region's metadata. As an optimization, freeing a region owned by an enclave
only requires TLB flushes on the cores running that enclave's threads. No other
core can have TLB entries for the enclave's memory.

\begin{figure}[htb]
  \centering
  \includegraphics[width=80mm]{figures/monitor_layout.pdf}
  \caption{Security monitor data structures.}
  \label{fig:monitor_layout}
%\EndFigure
\end{figure}

The region blocking mechanism guarantees that when a DRAM region is assigned to
an enclave or the OS, no stale TLB mappings associated with the DRAM
region exist. The monitor uses the MMU extensions described in
\S~\ref{sec:pmh_walker_input} and \S~\ref{sec:pmh_walker} to ensure that once a
DRAM region is assigned, no software other than the region's owner may create
TLB entries pointing inside the DRAM region. Together, these mechanisms
guarantee that the DRAM regions allocated to an enclave cannot be accessed by
the operating system or by another enclave.


\HeadingLevelD{Metadata Regions}
\label{sec:monitor_metadata_regions}

Since the security monitor sits between the OS and enclave, and its APIs
can be invoked by both sides, it is an easy target for timing attacks. Sanctum
prevents these attacks with a straightforward policy that states the security
monitor is never allowed to access enclave data, and is not allowed to make
memory accesses that depend on the attestation key material. The rest of the
data handled by the monitor is derived from the OS' actions, so it is already
known to the OS.

A rather obvious consequence of the policy above is that after the security
monitor boots the OS, it cannot perform any cryptographic operations that use
keys. For example, the security monitor cannot compute an attestation
signature directly, and defers that operation to a signing
enclave~(\S~\ref{sec:sanctum_signing_enclave}). While it is possible to
implement some cryptographic primitives without performing data-dependent
accesses, the security and correctness proofs behind these implementations are
non-trivial. For this reason, Sanctum avoids depending on any such
implementation.

A more subtle aspect of the access policy outlined above is that the metadata
structures that the security monitor uses to operate enclaves cannot be stored
in DRAM regions owned by enclaves, because that would give the OS an indirect
method of accessing the LLC sets that map to enclave's DRAM regions, which
could facilitate a cache timing attack.

For this reason, the security monitor requires the OS to set aside at least one
DRAM region for enclave metadata before it can create enclaves. The OS has the
ability to free up the metadata DRAM region, and regain the LLC sets associated
with it, if it predicts that the computer's workload will not involve enclaves.

Each DRAM region that holds enclave metadata is managed independently from the
other regions, at page granularity. The first few pages of each region contain
a page map that that tracks the usage of each metadata page,
specifically the enclave that it is assigned to, and the data structure that it
holds.

Each metadata region is like an EPC region in SGX, with the exception that our
metadata regions only hold special pages, like Sanctum's equivalent of SGX's
Secure Enclave Control Structure (SECS) and the Thread Control Structure (TCS).
These structures will be described in the following sections.

The data structures used to store Sanctum's metadata can span multiple pages.
When the OS allocates such a structure in a metadata region, it must point the
monitor to a sequence of free pages that belong to the same DRAM region. All
the pages needed to represent the structure are allocated and released in one
API call.


\HeadingLevelD{Enclave Lifecycle}
\label{sec:monitor_enclave_lifecycle}

The lifecycle of a Sanctum enclave is very similar to that of its SGX
counterparts, as shown in Figure~\ref{fig:enclave_states}.

\begin{figure}[htb]
  \centering
  \includegraphics[width=80mm]{figures/enclave_states.pdf}
  \caption{
    Enclave states and enclave management API calls.
  }
  \label{fig:enclave_states}
%\EndFigure
\end{figure}

The OS creates an enclave by issuing a \textit{create enclave} call that
creates the enclave metadata structure, which is Sanctum's equivalent of the
SECS. The enclave metadata structure contains an array of mailboxes whose size
is established at enclave creation time, so the number of pages required by the
structure varies from enclave to enclave. \S~\ref{sec:mailboxes} describes the
contents and use of mailboxes.

The \textit{create enclave} API call initializes the enclave metadata fields
shown in Figure~\ref{fig:enclave_layout}, and places the enclave in the LOADING
state. While the enclave is in this state, the OS sets up the enclave's initial
state via monitor calls that assign DRAM regions to the enclave, create
hardware threads and page table entries, and copy code and data into the
enclave. The OS then issues a monitor call to transition the enclave to the
INITIALIZED state, which finalizes its measurement hash. The application
hosting the enclave is now free to run enclave threads.

Sanctum stores a measurement hash for each enclave in its metadata area, and
updates the measurement to account for every operation performed on an enclave
in the LOADING state. The policy described in
\S~\ref{sec:monitor_metadata_regions} does not apply to the secure hash
operations used to update enclave's measurement, because the whole of the data used to
compute the hash is already known to the OS.

Enclave metadata is stored in a metadata
region~(\S~\ref{sec:monitor_metadata_regions}), so it can only be accessed by
the security monitor. Therefore, the metadata area can safely store public
information with integrity requirements, such as the enclave's measurement
hash.

While an OS loads an enclave, it is free to map the enclave's pages, but the
monitor maintains its page tables ensuring all entries point to non-overlapping
pages in DRAM owned by the enclave. Once an enclave is initialized, it can
inspect its own page tables and abort if the OS created undesirable mappings.
Simple enclaves do not require specific mappings. Complex enclaves are expected
to communicate their desired mappings to the OS via out-of-band metadata not
covered by this work.

Sanctum's monitor ensures that page tables do not overlap by storing the last
mapped page's physical address in the enclave's metadata. To simplify the
monitor, a new mapping is allowed if its physical address is greater than that of the
last, constraining the OS to map an enclave's DRAM pages
in monotonically increasing order.


\HeadingLevelD{Enclave Code Execution}
\label{sec:monitor_enclave_entry}

Sanctum closely follows the threading model of SGX enclaves. Each CPU core that
executes enclave code uses a thread metadata structure, which is Sanctum's equivalent
of SGX's TCS combined with SGX's State Save Area (SSA). Thread metadata
structures are stored in a DRAM region dedicated to enclave metadata in order
to prevent a malicious OS from mounting timing attacks against an enclave by
causing AEXes on its threads. Figure~\ref{fig:thread_slot_states} shows the
lifecycle of a thread metadata structure.

\begin{figure}[htb]
  \centering
  \includegraphics[width=80mm]{figures/thread_slot_states.pdf}
  \caption{
    Enclave thread metadata structure states and thread-related API calls.
  }
  \label{fig:thread_slot_states}
%\EndFigure
\end{figure}

The OS turns a sequence of free pages in a metadata region into an uninitialized
thread structure via an \textit{allocate thread} monitor call. During enclave
loading, the OS uses a \textit{load thread} monitor call to initialize the
thread structure with data that contributes to the enclave's measurement.
After an enclave is initialized, it can use an \textit{accept thread} monitor
call to initialize its thread structure.

The application hosting an enclave starts executing enclave code by issuing an
\textit{enclave enter} API call, which must specify an initialized thread
structure. The monitor honors this call by configuring Sanctum's hardware
extensions to allow access to the enclave's memory, and then by loading the
program counter and stack pointer registers from the thread's metadata
structure. The enclave's code can return control to the hosting application
voluntarily, by issuing an \textit{enclave exit} API call, which restores the
application's PC and SP from the thread state area and sets the API call's
return value to \texttt{ok}.

When performing an AEX, the security monitor atomically tests-and-sets the
\textit{AEX state valid} flag in the current thread's metadata. If the flag is
clear, the monitor stores the core's execution state in the thread state's AEX
area.  Otherwise, the enclave thread was resuming from an AEX, so the monitor
does not change the AEX area. When the host application re-enters the enclave,
it will resume from the previous AEX. This reasoning avoids the complexity of
SGX's state stack.

If an interrupt occurs while the enclave code is executing, the security
monitor's exception handler performs an AEX, which sets the API call's return
value to \texttt{async\_exit}, and invokes the standard interrupt handling
code. After the OS handles the interrupt, the enclave's host application
resumes execution, and re-executes the \textit{enter enclave} API call. The
enclave's thread initialization code examines the saved thread state, and
seeing that the thread has undergone an AEX, issues a \textit{resume thread}
API call. The security monitor restores the enclave's registers from the thread
state area, and clears the AEX flag.


\HeadingLevelD{Mailboxes}
\label{sec:mailboxes}

Sanctum's software attestation process relies on \textit{mailboxes}, which are
a simplified version of SGX's local attestation mechanism. Sanctum could not follow
SGX's approach because it relies on key derivation and MAC algorithms, and Sanctum's
timing attack avoidance policy~(\S~\ref{sec:monitor_metadata_regions}) states
that the security monitor is not allowed to perform cryptographic operations
that use keys.

Each enclave's metadata area contains an array of mailboxes, whose size is
specified at enclave creation time, and covered by the enclave's measurement.
Each mailbox goes through the lifecycle shown in
Figure~\ref{fig:sanctum_mailboxes}.

\begin{figure}[htb]
  \centering
  \includegraphics[width=65mm]{figures/mailbox_states.pdf}
  \caption{
    Mailbox states and security monitor API calls related to inter-enclave
    communication.
  }
  \label{fig:sanctum_mailboxes}
%\EndFigure
\end{figure}

An enclave that wishes to receive a message in a mailbox, such as the signing
enclave, declares its intent by performing an \textit{accept message} monitor
call. The API call is used to specify the mailbox that will receive the
message, and the identity of the enclave that is expected to send the message.

The sending enclave, which is usually the enclave wishing to be authenticated,
performs a \textit{send message} call that specifies the identity of the
receiving enclave, and a mailbox within that enclave. The monitor only delivers
messages to mailboxes that expect them. At enclave initialization, the expected
sender for all mailboxes is an invalid value (all zeros), so the enclave will
not receive messages until it calls \textit{accept message}.

When the receiving enclave is notified via an out-of-band mechanism that it has
received a message, it issues a \textit{read message} call to the monitor,
which moves the message from the mailbox into the enclave's memory. If the API
call succeeds, the receiving enclave is assured that the message was sent by
the enclave whose identity was specified in the \textit{accept message} call.

Enclave mailboxes are stored in metadata
regions~(\S~\ref{sec:monitor_metadata_regions}), which cannot be accessed by
any software other than the security monitor. This guarantees the privacy,
integrity, and freshness of the messages sent via the mailbox system.

Sanctum's mailbox design has the downside that both the sending and receiving enclave
need to be alive in DRAM in order to communicate. By comparison, SGX's local
attestation can be done asynchronously. In return, mailboxes do not require any
cryptographic operations, and have a much simpler correctness argument.


\HeadingLevelD{Multi-Core Concurrency}

The security monitor is highly concurrent, with fine-grained locks. API calls
targeting two different enclaves may be executed in parallel on different
cores. Each DRAM region has a lock guarding that region's metadata. An enclave
is guarded by the lock of the DRAM region holding its metadata. Each thread
metadata structure also has a lock guarding it, which is acquired when the
structure is accessed, but also while the metadata structure is used by a core
running enclave code. Thus, the \textit{enter enclave} call acquires a slot
lock, which is released by an \textit{enclave exit} call or by an AEX.

Sanctum avoids deadlocks by using a form of optimistic locking. Each monitor call
attempts to acquire all necessary locks via atomic test-and-set operations,
and errors with a \texttt{concurrent\_call} code if any lock is unavailable.
