Re: Erlang implementation fun...

[Mike Salib <address@hidden>]

James,
Thanks for the info, but I'm interested in the example that Joe used:
thousands of threads each serving a different network client. I can see
how communication between threads in the same process isn't a big deal
(just use shared memory to implement your message passing ;-)

> As far as I know, all erlang code in one "VM" is run in one thread, and 
> with one network connection to each other "VM". Usually there's only a 
> small number of VMs running, so, a small number of network connections. 
> The erlang message sending is not streams, but packet interfaces, so 
> everything can come in on one endpoint per remote host (or, if they use 
> UDP, it could be one endpoint for everyone). And, if you have only one 
> host, kernel networking isn't involved at all. Everything just gets 
> queued and unqueued in the one memory image.
> (Note: i know almost nothing about erlang other than what i've gleaned 
> from the website when I found out about it and found it interesting a 
> year or two ago)
> 
> > Assuming it is, how can Erlang programs with thousands of threads be
> > scalable? I had thought that network throughput slows dramatically when
> > using the poll system call for more than a few thousand clients; 
> > forcing
> > the kernel to iterate over a very long list of file descriptors when
> > almost all of them will have no activity is very time consuming. I know
> > of a few tricks that more or less fix the problem (/dev/poll on 
> > solaris,
> > kqueue on freebsd, IO completion ports on windows, the e_poll system
> > call on linux), but they're not exactly widely available and they're
> > definitely not POSIX.
> 
> Well, that's all just implementation details, so why does it matter 
> whether it's POSIX or not? And I doubt that'd often be a problem unless 
> you have a *LOT* of erlang systems cooperating in one network.

POSIX matters because its the lowest common denominator. If your Erlang
implementation is written to POSIX, it'll work everywhere. If it makes
use of kqueues, it will only run on FreeBSD. Erlang looked too serious
to be a one-OS language.

The problem here is that if Erlang is portable (only uses POSIX
interfaces), then it should provide terrible performance for network
serving to lots and lots of clients. If that's the case, then why did
Joe highlight it in his talk? If that's not the case, I'm very curious
to hear how Erlang gets scalable network IO in single process. Serving
thousands of network clients at once is a task close to my heart.

Put another way, what I know about this problem tells me that it is
impossible to build a portable system that uses a single process to
handle thousands of different network clients. Joe claims he can do
that. My question is "how?".

> > My other question is whether Erlang offers any realtime garuntees. If
> > so, how do implementations manage interactions between the kernel's
> > scheduling garuntees and the Erlang scheduler garuntees?
> 
> Presumably they must have a real-time system, or else a phone switch 
> wouldn't be possible. I'd guess that phone switches don't run on linux, 
> but rather on a system where erlang is the only thing running, but 
> again, I don't really know anything about it.

Yes, many problems go away if you run on bare metal and don't have to
deal with those pesky operating systems, or, dare I say it, other
programs (perhaps even programs written in unsafe languages). That may
be an option if you're building a telephone switch or some other
embedded system, but most of my work doesn't permit me that luxory. 


-Mike