OKWS
Note: These lecture notes were slightly modified from the ones posted on the 6.858 course website from 2014.
Today's lecture: how to build a secure web server on Unix.
The design of our lab web server, zookws, is inspired by OKWS.
Privilege separation
- Big security idea
- Split system into modules, each with their own privilege
- Idea: if one module is compromised, then other modules won't be
- Use often:
- Virtual machines (e.g., run web site in its own virtual machine)
- SSH (seperates sshd, agent)
- Challenges:
- Modules need to share
- Need OS support
- Need to use OS carefully to set things up correctly
- Performance
OKWS
- Interesting case study of privilege separation
- Lots of sharing between services
- Strict partitioning doesn't work
- Lots of code
- Not widely used outside of OKcupid
- Many web sites have their privilege separation plan
- But no papers describing their plans
Background: security and protection in Unix
- Typical principals: user IDs, group IDs (32-bit integers).
- Each process has a user ID (uid), and a list of group IDs (gid + grouplist).
- For mostly-historical reasons, a process has a gid + extra grouplist.
- Superuser principal (root) represented by
uid=0, bypasses most checks.
- What are the objects + ops in Unix, and how does the OS do access control?
- Files, directories.
- File operations: read, write, execute, change perms, ..
- Directory operations: lookup, create, remove, rename, change perms, ..
- Each inode has an owner user and group.
- Each inode has read, write, execute perms for user, group, others.
- Typically represented as a bit vector written base 8 (octal);
- octal works well because each digit is 3 bits (read, write, exec).
- Who can change permissions on files?
- Only user owner (process UID).
- Hard link to file: need write permission to file.
- Possible rationale: quotas.
- Possible rationale: prevent hard-linking
/etc/passwd to /var/mail/root with a world-writable /var/mail.
- Execute for directory means being able to lookup names (but not ls).
- Checks for process opening file
/etc/passwd:
- Must be able to look up
'etc' in /, 'passwd' in /etc.
- Must be able to open
/etc/passwd (read or read-write).
- Suppose you want file readable to intersection of group1 and group2.
- Is it possible to implement this in Unix?
- File descriptors.
- File access control checks performed at file open.
- Once process has an open file descriptor, can continue accessing.
- Processes can pass file descriptors (via Unix domain sockets).
- Processes.
- What can you do to a process?
- Debug (ptrace), send signal, wait for exit & get status,
- Debugging, sending signals: must have same UID (almost).
- Various exceptions, this gets tricky in practice.
- Waiting / getting exit status: must be parent of that process.
- Memory.
- One process cannot generally name memory in another process.
- Exception: debug mechanisms.
- Exception: memory-mapped files.
- Networking.
- Operations.
- Bind to a port.
- Connect to some address.
- Read/write a connection.
- Send/receive raw packets.
- Rules:
- only root (UID 0) can bind to ports below 1024;
- (e.g., arbitrary user cannot run a web server on port 80.)
- only root can send/receive raw packets.
- any process can connect to any address.
- can only read/write data on connection that a process has an fd for.
- Additionally, firewall imposes its own checks, unrelated to processes.
- How does the principal of a process get set?
- System calls:
setuid(), setgid(), setgroups().
- Only root (UID 0) can call these system calls (to first approximation).
- Where does the user ID, group ID list come from?
- On a typical Unix system, login program runs as root (UID 0)
- Checks supplied user password against
/etc/shadow.
- Finds user's UID based on
/etc/passwd.
- Finds user's groups based on
/etc/group.
- Calls
setuid(), setgid(), setgroups() before running user's shell
- How do you regain privileges after switching to a non-root user?
- Could use file descriptor passing (but have to write specialized code)
- Kernel mechanism: setuid/setgid binaries.
- When the binary is executed, set process UID or GID to binary owner.
- Specified with a special bit in the file's permissions.
- For example,
su / sudo binaries are typically setuid root.
- Even if your shell is not root, can run
"su otheruser"
su process will check passwd, run shell as otheruser if OK.- Many such programs on Unix, since root privileges often needed.
- Why might setuid-binaries be a bad idea, security-wise?
- Many ways for adversary (caller of binary) to manipulate process.
- In Unix, exec'ed process inherits environment vars, file descriptors, ..
- Libraries that a setuid program might use not sufficiently paranoid
- Historically, many vulnerabilities (e.g. pass
$LD_PRELOAD, ..)
- How to prevent a malicious program from exploiting setuid-root binaries?
- Kernel mechanism: chroot
- Changes what '/' means when opening files by path name.
- Cannot name files (e.g. setuid binaries) outside chroot tree.
- For example, OKWS uses chroot to restrict programs to
/var/okws/run, ..
- Kernel also ensures that '/../' does not allow escape from chroot.
- Why chroot only allowed for root?
- setuid binaries (like
su) can get confused about what's /etc/passwd.
- many kernel implementations (inadvertently?) allow recursive calls
to
chroot() to escape from chroot jail, so chroot is not an effective
security mechanism for a process running as root.
- Why hasn't chroot been fixed to confine a root process in that dir?
- Root can write kern mem, load kern modules, access disk sectors, ..
Background: traditional web server architecture (Apache)
- Apache runs
N identical processes, handling HTTP requests.
- All processes run as user
'www'.
- Application code (e.g. PHP) typically runs inside each of
N apache processes.
- Any accesses to OS state (files, processes, ...) performed by
www's UID.
- Storage: SQL database, typically one connection with full access to DB.
- Database principal is the entire application.
- Problem: if any component is compromised, adversary gets all the data.
- What kind of attacks might occur in a web application?
- Unintended data disclosure (getting page source code, hidden files, ..)
- Remote code execution (e.g., buffer overflow in Apache)
- Buggy application code (hard to write secure PHP code), e.g. SQL injection
- Attacks on web browsers (cross-site scripting attacks)
Back to OKWS: what's their application / motivation?
- Dating web site: worried about data secrecy.
- Not so worried about adversary breaking in and sending spam.
- Lots of server-side code execution: matching, profile updates, ...
- Must have sharing between users (e.g. matching) -- cannot just partition.
- Good summary of overall plan:
- "aspects most vulnerable to attack are least useful to attackers"
Why is this hard?
- Unix makes it tricky to reduce privileges (chroot, UIDs, ..)
- Applications need to share state in complicated ways.
- Unix and SQL databases don't have fine-grained sharing control mechanisms.
How does OKWS partition the web server?
- Figure 1 in paper.
- How does a request flow in this web server?
okd -> oklogd
-> pubd-> svc -> dbproxy-> oklogd
- How does this design map onto physical machines?
- Probably many front-end machines (
okld, okd, pubd, oklogd, svc)
- Several DB machines (
dbproxy, DB)
How do these components interact?
okld sets up socketpairs (bidirectional pipes) for each service.
- One socketpair for control RPC requests (e.g., "get a new log socketpair").
- One socketpair for logging (
okld has to get it from oklogd first via RPC).
- For HTTP services: one socketpair for forwarding HTTP connections.
- For
okd: the server-side FDs for HTTP services' socketpairs (HTTP+RPC).
okd listens on a separate socket for control requests (repub, relaunch).
- Seems to be port 11277 in Figure 1, but a Unix domain socket in OKWS code.
- For repub,
okd talks to pubd to generate new templates,
- then sends generated templates to each service via RPC control channel.
- Services talk to DB proxy over TCP (connect by port number).
How does OKWS enforce isolation between components in Figure 1?
- Each service runs as a separate UID and GID.
- chroot used to confine each process to a separate directory (almost).
- Components communicate via pipes (or rather, Unix domain socket pairs).
- File descriptor passing used to pass around HTTP connections.
- What's the point of
okld?
- Why isn't
okld the same as okd?
- Why does
okld need to run as root? (Port 80, chroot/setuid.)
- What does it take for
okld to launch a service?
- Create socket pairs
- Get new socket to
oklogd
fork, setuid/setgid, exec the service- Pass control sockets to
okd
- What's the point of
oklogd?
- What's the point of
pubd?
- Why do we need a database proxy?
- Ensure that each service cannot fetch other data, if it is compromised.
- DB proxy protocol defined by app developer, depending on what app requires.
- One likely-common kind of proxy is a templatized SQL query.
- Proxy enforces overall query structure (select, update),
- but allows client to fill in query parameters.
- Where does the 20-byte token come from?
- Passed as arguments to service.
- Who checks the token?
- DB proxy has list of tokens (& allowed queries?)
- Who generates token?
- Not clear; manual by system administrator?
- What if token disclosed?
- Compromised component could issue queries.
- Table 1: why are all services and
okld in the same chroot?
- Is it a problem?
- How would we decide?
- What are the readable, writable files there?
- Readable: shared libraries containing service code.
- Writable: each service can write to its own
/cores/<uid>.
- Where's the config file?
/etc/okws_config, kept in memory by okld.
oklogd & pubd have separate chroots because they have important state:
oklogd's chroot contains the log file, want to ensure it's not modified.pubd's chroot contains the templates, want to avoid disclosing them (?).
- Why does OKWS need a separate GID for every service?
- Need to execute binary, but file ownership allows chmod.
- Solution: binaries owned by root, service is group owner, mode 0410.
- Why 0410 (user read, group execute), and not 0510 (user read & exec)?
- Why not process per user?
- Is per user strictly better?
- user X service?
- Per-service isolation probably made sense for okcupid given their apps.
- (i.e., perhaps they need a lot of sharing between users anyway?)
- Per-user isolation requires allocating UIDs per user, complicating
okld
- and reducing performance (though may still be OK for some use cases).
Does OKWS achieve its goal?
- What attacks from the list of typical web attacks does OKWS solve, and how?
- Most things other than XSS are addressed.
- XSS sort-of addressed through using specialized template routines.
- What's the effect of each component being compromised, and "attack surface"?
okld: root access to web server machine, but maybe not to DB.
- attack surface: small (no user input other than svc exit).
okd: intercept/modify all user HTTP reqs/responses, steal passwords.
- attack surface: parsing the first line of HTTP request; control requests.
pubd: corrupt templates, leverage to maybe exploit bug in some service?
- attack surface: requests to fetch templates from okd.
oklogd: corrupt/ignore/remove/falsify log entries
- attack surface: log messages from okd, okld, svcs
service: send garbage to user, access data for svc (modulo dbproxy)
- attack surface: HTTP requests from users (+ control msgs from okd)
dbproxy: access/change all user data in the database it's talking to
- attack surface: requests from authorized services
- requests from unauthorized services (easy to drop)
- OS kernel is part of the attack surface once a single service is compromised.
- Linux kernel vulnerabilities rare, but still show up several times a year.
- OKWS assumes developer does the right thing at design level (maybe not impl):
- Split web application into separate services (not clump all into one).
- Define precise protocols for DB proxy (otherwise any service gets any data).
- Performance?
- Seems better than most alternatives.
- Better performance under load (so, resists DoS attacks to some extent)
- How does OKWS compare to Apache?
- Overall, better design.
okld runs as root, vs. nothing in Apache, but probably minor.- Neither has a great solution to client-side vulnerabilities (XSS, ..)
- How might an adversary try to compromise a system like OKWS?
- Exploit buffer overflows or other vulnerabilities in C++ code.
- Find a SQL injection attack in some
dbproxy.
- Find logic bugs in service code.
- Find cross-site scripting vulnerabilities.
How successful is OKWS?
- Problems described in the paper are still pretty common.
- okcupid.com still runs OKWS, but doesn't seem to be used by other sites.
- C++ might not be a great choice for writing web applications.
- For many web applications, getting C++ performance might not be critical.
- Design should be applicable to other languages too (Python, etc).
- Infact,
zookws for labs in 6.858 is inspired by OKWS, runs Python code.
- DB proxy idea hasn't taken off, for typical web applications.
- But DB proxy is critical to restrict what data a service can access in OKWS.
- Why?
- Requires developers to define these APIs: extra work, gets in the way.
- Can be hard to precisely define the allowed DB queries ahead of time.
- (Although if it's hard, might be a flag that security policy is fuzzy.)
- Some work on privilege separation for Apache (though still hard to use).
- Unix makes it hard for non-root users to manipulate user IDs.
- Performance is a concern (running a separate process for each request).
scripts.mit.edu has a similar design, running scripts under different UIDs.
- Mostly worried about isolating users from one another.
- Paranoid web app developer can create separate locker for each component.
- Sensitive systems do partitioning at a coarser granularity.
- Credit card processing companies split credit card data vs. everything else.
- Use virtual machines or physical machine isolation to split apps, DBs, ..
How could you integrate modern Web application frameworks with OKWS?
- Need to help okd figure out how to route requests to services.
- Need to implement DB proxies, or some variant thereof, to protect data.
- Depends on how amenable the app code is to static analysis.
- Or need to ask programmer to annotate services w/ queries they can run.
- Need to ensure app code can run in separate processes (probably OK).
References