Nano-pass/serde like framework for rust.

[mamcx]

When I see some of the topics here and elsewhere about building languages for rust, most of it become talks about performance, deals with quirks of platforms, GC, and other matters that could obscure how actually implement a language. For some users, all that details become a wall that communicate "this is not for you". Yet, a simple language can be done in a few hours.

Also, what is best for an author/lang is not for other. Ones need fast JIT compilers, others transpilers, others fast parse -> execute loops, others are just exploring and researching, etc. This mean that even things like LLVM are hardly usefull for a lot of cases.

Under the https://nanopass.org idea, go to ast -> ?? -> execute can be changed at will. So, instead of a coupling like:

parsing -> AST ->  execute
//Change minds, rewrite a lot:
parsing -> AST ->  gen bytecode -> parsing bytecode -> execute

Everything is "just" chain another pass:

steps = [parsing, AST, execute]
steps.iter....
//Change minds, just add more specialization
steps = [parsing, AST,  [gen bytecode, parsing bytecode ], execute]

This mean the under the nanopass style, the authors define/build "a pass" and the framework is only concerned in how combine and transform them. This allow for a lot of flexibility (ie: You can swap the "generate bytecode" for "format the code like rust-fmt" and still retain part of the effor (lexer, build ast).

So the idea is have something like:

trait CompilerPass {...}

impl CompilerPass for Lexer {}
impl CompilerPass <Lexer> for LossLessAST {}
impl CompilerPass <LossLessAST> for SugaredAST  {}

Other places where this style show:

• Middlewares like in actix
• Serde
• Scala LMS?
• Transducers and iterators?
• Unix-like pipelines

This is at least on production with Chez scheme:
https://news.ycombinator.com/item?id=15156027

I think the case of middlewares show clearly the benefits:

#[actix_rt::main]
async fn main() {
    let app = App::new()
        .wrap_fn(|req, srv| { <-- Inject freely. Easy introspection and quick debugging
            println!("Hi from start. You requested: {}", req.path());
            srv.call(req).map(|res| {
                println!("Hi from response");
                res
            })
        })
       .wrap(Logger::default()) <-- Common middlewares allow to reuse efforts across the community
      .wrap(
                CookieSession::signed(&[0; 32])
                    .secure(false),
         ) <-- Inject state and thread across the pipeline (for example, debugging info)
        ...<--And finally, add/remove at will. Not pay for what you don't need.
        .route(
            "/index.html",
            web::get().to(|| async {
                "Hello, middleware!"
            }),
        );
}

Originally posted by @mamcx in #1 (comment)

[brendanzab]

One thing I've been wondering about is whether it would be possible for certain passes to support a 'pull-based', streaming model of compilation. So the rust compiler could fuse multiple 'nanopasses' into a single pass, in a similar way to how iterators, streams, and futures work. For source-to-source optimization passes you might need to allocate intermediate data structures, but it would be super cool if this could be avoided as much as possible.

[mamcx]

My understanding is that transducers allow to do that (and support both styles with the same logic applied)