Re: Extensible syntax

[Michael Vanier <address@hidden>]

If you want to go all the way with an environment that allows you to define
arbitrary DSLs, then you'll have to use a dedicated lexer and parser (like
lex/yacc allow you to do with C, and innumerable variants of those programs
allow you to do with other languages).  This will give you absolute control
(up to the limits of the tools e.g. LALR(1) languages only), but is a lot
of work (certainly not "lightweight" by any standard I can think of).

The brilliance of the lisp/haskell/dylan approach is exactly what you
describe: you can write syntactic extensions that are meaningful and save
you from writing reams of boilerplate code, but because they are
constrained by the host syntax, they are easy to define and fit nicely
within the host language.  Rebol (http://www.rebol.com) also has some kind
of a language extension system that is similar in spirit.

What has occurred to me is that it should be easier to access languages at
the abstract syntax tree (AST) level, with surface syntaxes provided for
convenience and also definable by users.  Interoperability is trivial at
(say) the byte-code level.  The only example of this that I know of is
camlp4 for ocaml.  Even there, I have worked with one large system designed
using camlp4 to extend the syntax, and the syntax quickly got VERY
unwieldy.  I'm not sure if this was just due to laziness on the part of the
implementors or some deeper limitation of camlp4 that makes it easier to
design syntaxes that are similar to the ocaml syntax.

Mike

> From: "Tom Locke" <tom@livelogix.com>
> Date: Thu, 11 Dec 2003 13:12:28 +0530
> 
> Hi,
> 
> ...seeking links / advice...
> 
> I'm currently working on a language that is extensible in the same way
> that "Lisp is a programmable programming language". The intention is to
> come up with a language that can easily morph into a wide range of DSLs.
> 
> Lisp/Scheme provide exactly this facility, but every DSL must have
> Lisp's syntax. What about lisp-like extensibility, plus extensible
> syntax - i.e. a fully extensible set of prefix/infix/postfix operators
> and compound statements (e.g. control structures)?
> 
> There's Dylan, with its template based macro system, but Dylan syntax is
> only extensible within a fairly strict framework. It always looks like
> Dylan. Agree?
> 
> Then there's Smalltalk et al, using blocks to provide extensible control
> structures, but again the syntax is highly constrained.
> 
> Haskell has extensible prefix and infix operators, with user control
> over precedence. This gets you quite far. I once built an XPath like
> library in Haskell that tried to mimic regular XPath syntax, but you can
> only get part of the way there.
> 
> Of course there's a ton of preprocessors that allow one to extend a
> languages grammar, but this is something of a heavyweight approach, and
> may suffer from lack of modularity?? I'm after something lightweight and
> built-in, similar to the way Lisp lets me just fire off a defmacro as
> easily as a defun.
> 
> Also of great interest is the ability to freely embed DSLs inside each
> other. I don't think I've seen this done (except in Lisp where the
> syntax doesn't change).
> 
> Thoughts? Links?
> 
> Thanks very much.
> 
> Tom.
>