Version 0.15.0

0.15.0 (22 June 2021) - "We say 'howdy' around these parts" edition:

Beside introducing dialects (a.k.a. whole-module code transforms), this edition concentrates on upgrading our dependencies, namely the macro expander, and the Python language itself, to ensure unpythonic keeps working for the next few years. This introduces some breaking changes, so we have also taken the opportunity to apply any such that were previously scheduled.

We have sneaked in some upgrades for other subsystems, too. Particularly curry, the multiple dispatch system (@generic), and the integration between these two have been improved significantly.

IMPORTANT:

• Minimum Python language version is now 3.6.
  • We support 3.6, 3.7, 3.8, 3.9 and PyPy3 (language versions 3.6 and 3.7).
  • For future plans, see our Python language version support status.
• The optional macro expander is now mcpyrate.

If you still need unpythonic for Python 3.4 or 3.5, use version 0.14.3, which is the final version of unpythonic that supports those language versions.

The same applies if you need the macro parts of unpythonic (i.e. import anything from unpythonic.syntax) in your own project that uses MacroPy. Version 0.14.3 of unpythonic works up to Python 3.7.

New:

• Dialects! New module unpythonic.dialects, providing some example dialects that demonstrate what can be done with a dialects system (i.e. full-module code transformer) together with a kitchen-sink language extension macro package such as unpythonic.

  • These dialects have been moved from the now-obsolete pydialect project and ported to use mcpyrate.

• Improved robustness: several auxiliary syntactic constructs now detect at macro expansion time if they appear outside any valid lexical context, and raise SyntaxError (with a descriptive message) if so.

  • The full list is:
    • call_cc[], for with continuations
    • it, for aif[]
    • local[]/delete[], for do[]
    • q/u/kw, for with prefix
    • where, for let[body, where(k0=v0, ...)] (also for letseq, letrec, let_syntax, abbrev)
    • with expr/with block, for with let_syntax/with abbrev
  • Previously these constructs could only raise an error at run time, and not all of them could detect the error even then.

• Syntactic consistency: allow env-assignment notation and brackets to declare bindings in the let family of macros. The preferred syntaxes for the let macro are now:

  let[x << 42, y << 9001][...]         # lispy expr
  let[[x << 42, y << 9001] in ...]     # haskelly let-in
  let[..., where[x << 42, y << 9001]]  # haskelly let-where

  If there is just one binding, these become:

  let[x << 42][...]
  let[[x << 42] in ...]
  let[..., where[x << 42]]

  Similarly for letseq, letrec, and the decorator versions; and for the expr forms of let_syntax, abbrev. The reason for preferring this notation is that it is consistent with both unpythonic's env-assignments (let bindings live in an env) and the use of brackets to denote macro invocations.

  To ease backwards compatibility, we still accept the syntax used up to v0.14.3, too.

  Also, from symmetry and usability viewpoints, if a mix of brackets and parentheses are used, it hardly makes sense to require some specific mix - so this has been extended so that the choice of delimiter doesn't matter. All the following are also accepted, with the meaning exactly the same as above:

  let[[x, 42], [y, 9001]][...]  # best visual consistency
  let[(x, 42), (y, 9001)][...]
  let([x, 42], [y, 9001])[...]
  let((x, 42), (y, 9001))[...]  # like up to v0.14.3
  let[[[x, 42], [y, 9001]] in ...]  # best visual consistency
  let[[(x, 42), (y, 9001)] in ...]
  let[([x, 42], [y, 9001]) in ...]
  let[((x, 42), (y, 9001)) in ...]  # like up to v0.14.3
  let[(x << 42, y << 9001) in ...]
  let[..., where[[x, 42], [y, 9001]]]  # best visual consistency
  let[..., where[(x, 42), (y, 9001)]]
  let[..., where([x, 42], [y, 9001])]
  let[..., where((x, 42), (y, 9001))]  # like up to v0.14.3
  let[..., where(x << 42, y << 9001)]

  For a single binding, these are also accepted:

  let[x, 42][...]
  let(x, 42)[...]  # like up to v0.14.3
  let[[x, 42] in ...]
  let[(x, 42) in ...]  # like up to v0.14.3
  let[(x << 42) in ...]
  let[..., where[x, 42]]
  let[..., where(x, 42)]  # like up to v0.14.3
  let[..., where(x << 42)]

  These alternate syntaxes will be supported at least as long as we accept parentheses to pass macro arguments; but in new code, please use the preferred syntaxes.

• Miscellaneous.

  • with namedlambda now understands the walrus operator, too. In the construct f := lambda ...: ..., the lambda will get the name f. (Python 3.8 and later.)
  • with namedlambda now auto-names lambdas that don't have a name candidate using their source location info, if present. This makes it easy to see in a stack trace where some particular lambda was defined.
  • Multiple-dispatch system unpythonic.dispatch:
    • Use consistent terminology:
      • The function that supports multiple call signatures is a generic function.
      • Its individual implementations are multimethods.
    • Add decorator @augment: add a multimethod to a generic function defined elsewhere.
    • Add function isgeneric to detect whether a callable has been declared @generic.
    • Add function methods: display a list of multimethods of a generic function.
    • It is now possible to dispatch on a homogeneous type of contents collected by a **kwargs parameter.
    • curry now supports @generic functions. This feature is experimental. Semantics may still change.
    • The utilities arities, required_kwargs, and optional_kwargs now support @generic functions. This feature is experimental. Semantics may still change.
  • curry now errors out immediately on argument type mismatch.
  • Add partial, a type-checking wrapper for functools.partial, that errors out immediately on argument type mismatch.
  • Add unpythonic.excutil.reraise_in (expr form), unpythonic.excutil.reraise (block form): conveniently remap library exception types to application exception types. Idea from Alexis King (2016): Four months with Haskell.
  • Add variants of the above for the conditions-and-restarts system: unpythonic.conditions.resignal_in, unpythonic.conditions.resignal. The new signal is sent using the same error-handling protocol as the original signal, so that e.g. an error remains an error even if re-signaling changes its type.
  • Add resolve_bindings_partial, useful for analyzing partial application.
  • Add triangular, to generate the triangular numbers (1, 3, 6, 10, ...).
  • Add partition_int_triangular to answer a timeless question concerning stackable plushies.
  • Add partition_int_custom to answer unanticipated similar questions.
  • All documentation files now have a quick navigation section to skip to another part of the docs. (For all except the README, it's at the top.)
  • Python 3.8 and 3.9 support added.

Non-breaking changes:

• Changes to how some macros expand.

  • Some macros, notably letseq, do0, and lazyrec, now expand into hygienic macro captures of other macros. The continuations macro also outputs a hygienically captured aif when transforming an or expression that occurs in tail position.

    • This allows mcpyrate.debug.step_expansion to show the intermediate result, as well as brings the implementation closer to the natural explanation of how these macros are defined. (Zen of Python: if the implementation is easy to explain, it might be a good idea.)
    • The implicit do (extra bracket syntax) also expands as a hygienically captured do, but e.g. in let[] it will then expand immediately (due to let's inside-out expansion order) before control returns to the macro stepper. If you want to see the implicit do[] invocation, use the "detailed" mode of the stepper, which shows individual macro invocations even when expanding inside-out: step_expansion["detailed"][...], with step_expansion["detailed"]:.

  • The do[] and do0[] macros now expand outside-in. The main differences from a user perspective are:

    • Any source code captures (such as those performed by test[]) show the expanded output of do and do0, because that's what they receive. (For tests, you may want to use the macro with expand_testing_macros_first, which see.)
    • mcpyrate.debug.step_expansion is able to show the intermediate result after the do or do0 has expanded, but before anything else has been done to the tree.

• Miscellaneous.

  • Resolve issue #61: curry now supports kwargs properly.
    • We now analyze parameter bindings like Python itself does, so it should no longer matter whether arguments are passed by position or by name.
    • Positional passthrough works as before. Named passthrough added.
    • Any remaining arguments (that cannot be accepted by the initial call) are passed through to a callable intermediate result (if any), and then outward on the curry context stack as a Values. Since curry in this role is essentially a function-composition utility, the receiving curried function instance unpacks the Values into args and kwargs.
    • If any extra arguments (positional or named) remain when the top-level curry context exits, then by default, TypeError is raised. To override, use with dyn.let(curry_context=["whatever"]), just like before. Then you'll get a Values object.
  • The generator instances created by the gfuncs returned by gmemoize, imemoize, and fimemoize, now support the __len__ and __getitem__ methods to access the already-yielded, memoized part. Asking for the len returns the current length of the memo. For subscripting, both a single int index and a slice are accepted. Note that memoized generators do not support all of the collections.abc.Sequence API, because e.g. __contains__ and __reversed__ are missing, on purpose.
  • fup/fupdate/ShadowedSequence can now walk the start of a memoized infinite replacement backwards. (Use imemoize on the original iterable, instantiate the generator, and use that generator instance as the replacement.)
  • When using the autoreturn macro, if the item in tail position is a function definition or class definition, return the thing that was defined.
  • The nb macro now works together with autoreturn.
  • unpythonic.conditions.signal, when the signal goes unhandled, now returns the canonized input condition, with a nice traceback attached. This feature is intended for implementing custom error protocols on top of signal; error already uses it to produce a nice-looking error report.
  • The internal exception types unpythonic.conditions.InvokeRestart and unpythonic.ec.Escape now inherit from BaseException, so that they are not inadvertently caught by except Exception handlers.
  • The modules unpythonic.dispatch and unpythonic.typecheck, which provide the @generic and @typed decorators and the isoftype function, are no longer considered experimental. From this release on, they receive the same semantic versioning guarantees as the rest of unpythonic.
  • CI: Automated tests now run on Python 3.6, 3.7, 3.8, 3.9, and PyPy3 (language versions 3.6, 3.7).
  • CI: Test coverage improved to 94%.
  • Full update pass for the user manual written in Markdown.
    • Things added or changed in 0.14.2 and later are still mentioned as such, and have not necessarily been folded into the main text. But everything should be at least up to date now.

Breaking changes:

• New macro expander mcpyrate; MacroPy support dropped.

  • API differences.
    • Macro arguments are now passed using brackets, macroname[args][...], with macroname[args], @macroname[args], instead of parentheses.
      • Parentheses are still available as alternative syntax, because up to Python 3.8, decorators cannot have subscripts (so e.g. @dlet[(x, 42)] is a syntax error, but @dlet((x, 42)) is fine). This has been fixed in Python 3.9.
      • If you already only run on Python 3.9 and later, please use brackets, that is the preferred syntax. We currently plan to eventually drop support for parentheses to pass macro arguments in the future, when Python 3.9 becomes the minimum supported language version for unpythonic.
    • If you write your own macros, note mcpyrate is not drop-in compatible with MacroPy or mcpy. See its documentation for details.
  • Behavior differences.
    • mcpyrate should report test coverage for macro-using code correctly; no need for # pragma: no cover in block macro invocations or in quasiquoted code.

• Previously scheduled API changes.

  • As promised, names deprecated during 0.14.x have been removed. Old name on the left, new name on the right:
    • m → imathify (consistency with the rest of unpythonic)
    • mg → gmathify (consistency with the rest of unpythonic)
    • setescape → catch (Lisp family standard name)
    • escape → throw (Lisp family standard name)
    • getvalue, runpipe → exitpipe (combined into one)
      • CAUTION: exitpipe already existed in v0.14.3, but beginning with v0.15.0, it is now an unpythonic.symbol.sym (like a Lisp symbol). This is not compatible with existing, pickled exitpipe instances; it used to be an instance of the class Getvalue, which has been removed. (There's not much reason to pickle an exitpipe instance, but we're mentioning this for the sake of completeness.)
  • Drop support for deprecated argument format for raisef. Now the usage is raisef(exc) or raisef(exc, cause=...). These correspond exactly to raise exc and raise exc from ..., respectively.

• Other backward-incompatible API changes.

  • Multiple-return-value handling changed. Resolves issue #32.
    • Multiple return values are now denoted as Values, available from the top-level namespace of unpythonic.
    • The Values constructor accepts both positional and named arguments. Passing in named arguments creates named return values. This completes the symmetry between argument passing and returns.
    • Most of the time, it's still fine to return a tuple and destructure that; but in contexts where it is important to distinguish between a single tuple return value and multiple return values, it is preferable to use Values.
    • In any utilities that deal with function composition, if your intent is multiple-return-values, it is now mandatory to return a Values instead of a tuple:
      • curry
      • pipe family
      • compose family
      • unfold
      • iterate
      • All multiple-return-values in code using the with continuations macro. (The continuations system essentially composes continuation functions.)
  • The lazy evaluation tools lazy, Lazy, and the quick lambda f (underscore notation for Python) are now provided by unpythonic as unpythonic.syntax.lazy, unpythonic.lazyutil.Lazy, and unpythonic.syntax.fn (note name change!), because they used to be provided by macropy, and mcpyrate does not provide them.
    • API differences.
      • The quick lambda is now named fn[] instead of f[] (as in MacroPy). This was changed because f is often used as a function name in code examples, local temporaries, and similar. Also, fn[] is a less ambiguous abbreviation for a syntactic construct that means function, while remaining shorter than the equivalent lambda. Compare fn[_ * 2] and lambda x: x * 2, or fn[_ * _] and lambda x, y: x * y.
        • Note that in mcpyrate, macros can be as-imported, so this change affects just the default name of fn[]. But that is exactly what is important: have a sensible default name, to remove the need to as-import so often.
      • The macros lazy and fn can be imported from the syntax interface module, unpythonic.syntax, and the class Lazy is available at the top level of unpythonic.
      • Unlike macropy's Lazy, our Lazy does not define __call__; instead, it defines the method force, which has the same effect (it computes if necessary, and then returns the value of the promise). You can also use the function unpythonic.force, which has the extra advantage that it passes through a non-promise input unchanged (so you don't need to care whether x is a promise before calling force(x); this is sometimes useful).
      • When you import the macro quicklambda, you must import also the macro fn.
      • The underscore _ is no longer a macro on its own. The fn macro treats the underscore magically, as before, but anywhere else it is available to be used as a regular variable.
    • Behavior differences.
      • fn[] now respects nesting: an invocation of fn[] will not descend into another nested fn[].
      • The with quicklambda macro is still provided, and used just as before. Now it causes any fn[] invocations lexically inside the block to expand before any other macros in that block do.
      • Since in mcpyrate, macros can be as-imported, you can rename fn at import time to have any name you want. The quicklambda block macro respects the as-import, by internally querying the expander to determine the name(s) the macro fn is currently bound to.
  • For the benefit of code using the with lazify macro, laziness is now better respected by the compose family, andf and orf. The utilities themselves are marked lazy, and arguments will be forced only when a lazy function in the chain actually uses them, or when an eager (not lazy) function is encountered in the chain.
  • Rename the curry macro to autocurry, to prevent name shadowing of the curry function. The new name is also more descriptive.
  • Move the functions force1 and force from unpythonic.syntax to unpythonic. Make the Lazy class (promise implementation) public. (They actually come from unpythonic.lazyutil.)
  • Change parameter ordering of unpythonic.it.window to make it curry-friendly. Usage is now window(n, iterable).
    • This was an oversight when this function was added; most other functions in unpythonic.it have been curry-friendly from the beginning.
  • Change output format of resolve_bindings to return an inspect.BoundArguments instead of the previous OrderedDict that had a custom format. Change the input format of tuplify_bindings to match.
  • Change parameter name from l to length in the functions in_slice and index_in_slice (in the unpythonic.collections module).
    • These are mostly used internally, but technically a part of the public API.
    • This change fixes a flake8 E741 warning, and the new name for the parameter is more descriptive.

• Miscellaneous.

  • Robustness: the with continuations macro now raises SyntaxError if async constructs (async def or await) appear lexically inside the block, because interaction of with continuations with Python's async subsystem has never been implemented. See issue #4.
  • The functions raisef, tryf, equip_with_traceback, and async_raise now live in unpythonic.excutil. They are still available in the top-level namespace of unpythonic, as usual.
  • The functions call and callwith now live in unpythonic.funutil. They are still available in the top-level namespace of unpythonic, as usual.
  • The functions almosteq, fixpoint, partition_int, and ulp now live in unpythonic.numutil. They are still available in the top-level namespace of unpythonic, as usual.
  • Remove the internal utility class unpythonic.syntax.util.ASTMarker. We now have mcpyrate.markers.ASTMarker, which is designed for data-driven communication between macros that work together. As a bonus, no markers are left in the AST at run time.
  • Rename contribution guidelines to CONTRIBUTING.md, which is the modern standard name. Old name was HACKING.md, which was correct, but nowadays obscure.
  • Python 3.4 and 3.5 support dropped, as these language versions have officially reached end-of-life.

Fixed:

• Make unpythonic.misc.callsite_filename ignore our call helpers. This allows the testing framework report the source code filename correctly when testing code using macros that make use of these helpers (e.g. autocurry, lazify).

• In aif, it is now only valid in the then and otherwise parts, as it should always have been.

• Fix docstring of test: multiple the[] marks were already supported in 0.14.3, as the macro documentation already said, but the docstring claimed otherwise.

• Fix bug in with namedlambda. Due to incorrect function arguments in the analyzer, already named lambdas were not detected correctly.

• Fix bug: fup/fupdate/ShadowedSequence now actually accept an infinite-length iterable as a replacement sequence (under the obvious usage limitations), as the documentation has always claimed.

• Fix bug: memoize is now thread-safe. Even when the same memoized function instance is called concurrently from multiple threads. Exactly one thread will compute the result. If f is recursive, the thread that acquired the lock is the one that is allowed to recurse into the memoized f.