I'm sorry, I don't really have any insightful perspective on debugging. But maybe I can elaborate a bit on what goes wrong if you make a new DSL without keeping in mind debugging considerations.
- You'll want to maintain a separate stack for your embedded DSL. It's really, really hard to debug when Lisp's stack and your DSL's get intertwined in a complex way in your stack trace. When I just had my straightforward HTML and CSS "closure-compilers" this wasn't such a big deal but when you make a language with more funky semantics where the order of evaluation is less obvious it's a real problem.
- When you compile something (say, your semantic representation of HTML into optimized closures), you should always keep the more semantic representation of what you're compiling around. As an example of this, at first I made a macro to directly generate Lisp code that generates HTML source code. That was bad because the only thing I could do with my HTML is output it.
But it turns out there are many things you can do if instead you make a macro that generates Lisp code that will build a semantic representation of HTML as Lisp objects: you can interpret it directly or optimize it or compile it directly or make a single-stepping debugger... The possibilities are limitless!
Also, to preserve semantics as much as possible I think a rewriting strategy is great. You start with your super abstract representation, and then you rewrite in successive steps to an ever more concrete and low-level representation until you're done. For example, in an HTML system you'd have a login-box object that rewrites to HTML lisp objects that rewrites to HTML source code. It would be great to have a way of linking the various pieces at the different semantics level. For example, having some feature Explain that when you point to a certain character in the HTML source, it could tell: "This character was generated as part of the value of the text attribute of a img node that was generated in that page because it's part of the sidebar that's in the template of this part of the site"... Anyway, what I mean is that sometimes you want to focus on the high-level semantics and other times on the low-level, and you should be able to switch contexts at will.
PS: I'm not even sure what exactly I was talking about anymore. Maybe I should start a blog...
That's very plausible, and your thoughts on intermediate representations are quite interesting. Now I'm going to change the subject back to my tangent. :)
Maybe it wouldn't be so "really, really hard to debug when Lisp's stack and your DSL's get intertwined in a complex way in your stack trace" if Lisp stack traces weren't so hard to debug in the first place. At least, they are for me. If anyone has any hints, please chime in. I'm getting tired of BREAK-and-FORMAT-driven-debugging. When Visual Studio does anything better than a Lisp environment, that's a goddamn emergency.
In the meantime, I think I'm going to try Clozure-with-a-Z and see if life is better.
p.s. If you do write a blog about this stuff, post it here. I'll read it.
- You'll want to maintain a separate stack for your embedded DSL. It's really, really hard to debug when Lisp's stack and your DSL's get intertwined in a complex way in your stack trace. When I just had my straightforward HTML and CSS "closure-compilers" this wasn't such a big deal but when you make a language with more funky semantics where the order of evaluation is less obvious it's a real problem.
- When you compile something (say, your semantic representation of HTML into optimized closures), you should always keep the more semantic representation of what you're compiling around. As an example of this, at first I made a macro to directly generate Lisp code that generates HTML source code. That was bad because the only thing I could do with my HTML is output it.
But it turns out there are many things you can do if instead you make a macro that generates Lisp code that will build a semantic representation of HTML as Lisp objects: you can interpret it directly or optimize it or compile it directly or make a single-stepping debugger... The possibilities are limitless!
Also, to preserve semantics as much as possible I think a rewriting strategy is great. You start with your super abstract representation, and then you rewrite in successive steps to an ever more concrete and low-level representation until you're done. For example, in an HTML system you'd have a login-box object that rewrites to HTML lisp objects that rewrites to HTML source code. It would be great to have a way of linking the various pieces at the different semantics level. For example, having some feature Explain that when you point to a certain character in the HTML source, it could tell: "This character was generated as part of the value of the text attribute of a img node that was generated in that page because it's part of the sidebar that's in the template of this part of the site"... Anyway, what I mean is that sometimes you want to focus on the high-level semantics and other times on the low-level, and you should be able to switch contexts at will.
PS: I'm not even sure what exactly I was talking about anymore. Maybe I should start a blog...