Avoid emptying the whole of region_highlight

[okapia]

By emptying region_highlight, z-sy-h can't be used so easily together with other plugins that use highlighting. Ideally, it would only remove it's own entries.

See, e.g okapia/zsh-viexchange#1

As discussed over e-mail a potential solution could take advantage of zsh ignoring unknown keys. e.g

region_highlight+=( '1 3 fg=red,owner=z-sy-h' )

[danielshahaf]

That idea wouldn't work: region_highlight+=( '0 1 fg=green,owner=z-sy-h' ) causes $region_highlight to contain 0 1 fg=green; the owner is dropped.

The region_highlight serializer (get_region_highlight()) is entirely injection-free; there's no way to have it echo a user-specified string.

[danielshahaf]

Issue sweep: any suggestions on how to approach this? Brainstorming:

• Patch zsh to support the owner key.
• Separation of concerns: have a mode in which _zsh_highlight only returns the list of entries to be appended to $region_highlight, and <handwave>something</handwave> is responsible for clearing it between _zsh_highlight invocations. (TBD whether that something would be a plugin, or a zle feature, or …)
• (Something else?)

[okapia]

I like the idea of the owner key. And actually ignoring unknown keys makes some sense.

[danielshahaf]

Ack. Don't have brainwidth to drive that currently, though.

[danielshahaf]

Right, here it is: zsh-users/zsh#57

And the corresponding z-sy-h patch:

(EDIT: See next comment for a replacement z-sy-h patch. The original thing I posted here is visible in the comment edit history if it's ever needed.)

I've checked that these two, together, fix okapia/zsh-viexchange#1. Could you confirm, please?

[danielshahaf]

Revised z-sy-h patch (cf. zsh-users/zsh#57 (review)):

Edit 2020-06-18: s/owner/memo/g

Revised z-sy-h patch

diff --git a/highlighters/pattern/pattern-highlighter.zsh b/highlighters/pattern/pattern-highlighter.zsh
index 054eff7..ce926f7 100644
--- a/highlighters/pattern/pattern-highlighter.zsh
+++ b/highlighters/pattern/pattern-highlighter.zsh
@@ -54,7 +54,7 @@ _zsh_highlight_pattern_highlighter_loop()
   local -a match mbegin mend
   local MATCH; integer MBEGIN MEND
   if [[ "$buf" == (#b)(*)(${~pat})* ]]; then
-    region_highlight+=("$((mbegin[2] - 1)) $mend[2] $ZSH_HIGHLIGHT_PATTERNS[$pat]")
+    region_highlight+=("$((mbegin[2] - 1)) $mend[2] $ZSH_HIGHLIGHT_PATTERNS[$pat], memo=zsh-syntax-highlighting")
     "$0" "$match[1]" "$pat"; return $?
   fi
 }
diff --git a/highlighters/regexp/regexp-highlighter.zsh b/highlighters/regexp/regexp-highlighter.zsh
index 26f9da3..c5806d0 100644
--- a/highlighters/regexp/regexp-highlighter.zsh
+++ b/highlighters/regexp/regexp-highlighter.zsh
@@ -55,7 +55,7 @@ _zsh_highlight_regexp_highlighter_loop()
   local -a match mbegin mend
   while true; do
     [[ "$buf" =~ "$pat" ]] || return;
-    region_highlight+=("$((MBEGIN - 1 + OFFSET)) $((MEND + OFFSET)) $ZSH_HIGHLIGHT_REGEXP[$pat]")
+    region_highlight+=("$((MBEGIN - 1 + OFFSET)) $((MEND + OFFSET)) $ZSH_HIGHLIGHT_REGEXP[$pat], memo=zsh-syntax-highlighting")
     buf="$buf[$(($MEND+1)),-1]"
     OFFSET=$((MEND+OFFSET));
   done
diff --git a/tests/test-highlighting.zsh b/tests/test-highlighting.zsh
index d0698d6..1ce6178 100755
--- a/tests/test-highlighting.zsh
+++ b/tests/test-highlighting.zsh
@@ -190,7 +190,7 @@ run_test_internal() {
     if
       [[ $start != $exp_start ]] ||
       [[ $end != $exp_end ]] ||
-      [[ $highlight_zone[3] != $expected_highlight_zone[3] ]]
+      [[ ${highlight_zone[3]%,} != ${expected_highlight_zone[3]%,} ]]
     then
       print -r -- "not ok $i - $desc - expected ($exp_start $exp_end ${(qqq)expected_highlight_zone[3]}), observed ($start $end ${(qqq)highlight_zone[3]}). $todo"
     else
diff --git a/zsh-syntax-highlighting.zsh b/zsh-syntax-highlighting.zsh
index bb85d33..19bed18 100644
--- a/zsh-syntax-highlighting.zsh
+++ b/zsh-syntax-highlighting.zsh
@@ -122,12 +122,32 @@ _zsh_highlight()
   # Make it read-only.  Can't combine this with the previous line when POSIX_BUILTINS may be set.
   typeset -r ret
 
+  (( ${+region_highlight} )) || typeset -ga region_highlight
+
+  # Probe the memo= feature, once.
+  (( ${+zsh_highlight__memo_feature} )) || {
+    region_highlight+=( "0 0 fg=red, memo=zsh-syntax-highlighting" )
+    if [[ ${region_highlight[-1]} == *memo=zsh-syntax-highlighting* ]]; then
+      integer -gr zsh_highlight__memo_feature=1
+    else
+      integer -gr zsh_highlight__memo_feature=0
+    fi
+    region_highlight[-1]=()
+  }
+
+  # Reset region_highlight to build it from scratch
+  if (( zsh_highlight__memo_feature )); then
+    region_highlight=( "${(@)region_highlight:#*memo=zsh-syntax-highlighting*}" )
+  else
+    # Legacy codepath.  Not very interoperable with other plugins (issue #418).
+    region_highlight=()
+  fi
+
   # Remove all highlighting in isearch, so that only the underlining done by zsh itself remains.
   # For details see FAQ entry 'Why does syntax highlighting not work while searching history?'.
   # This disables highlighting during isearch (for reasons explained in README.md) unless zsh is new enough
   # and doesn't have the pattern matching bug
   if [[ $WIDGET == zle-isearch-update ]] && { $zsh_highlight__pat_static_bug || ! (( $+ISEARCHMATCH_ACTIVE )) }; then
-    region_highlight=()
     return $ret
   fi
 
@@ -163,10 +183,6 @@ _zsh_highlight()
   # Do not highlight if there are pending inputs (copy/paste).
   [[ $PENDING -gt 0 ]] && return $ret
 
-  # Reset region highlight to build it from scratch
-  typeset -ga region_highlight
-  region_highlight=();
-
   {
     local cache_place
     local -a region_highlight_copy
@@ -285,7 +301,8 @@ _zsh_highlight_apply_zle_highlight() {
   else
     start=$second end=$first
   fi
-  region_highlight+=("$start $end $region")
+  # The comma is there to make zsh's that don't support the memo= key silently ignore it.
+  region_highlight+=("$start $end $region, memo=zsh-syntax-highlighting")
 }
 
 
@@ -325,7 +342,7 @@ _zsh_highlight_add_highlight()
   shift 2
   for highlight; do
     if (( $+ZSH_HIGHLIGHT_STYLES[$highlight] )); then
-      region_highlight+=("$start $end $ZSH_HIGHLIGHT_STYLES[$highlight]")
+      region_highlight+=("$start $end $ZSH_HIGHLIGHT_STYLES[$highlight], memo=zsh-syntax-highlighting")
       break
     fi
   done