Video stream modifications via a new VideoStreamProcessor (#135)

* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame`

* Add `VideoStreamService` architecture and setup one in `VideoStreamFrame`

* Rename `VideoStreamEnhancers` to `Overlays`, make `ImageDrawOverlay` editable

* `ClickListener`: make `callback` optional and allow activating via `with` context

* `MediaGrabber`: fix bug: `get_media` cleared `continuousCollectionEvent`

* `calibrate_beamshift.py`: make imports absolute, allow float beamshift

* Instamatic GUI: temporarily print coords of clicked point, revert later

* Move `instamatic.typing` to `typing_` to avoid import conflicts

* `stage`: Add `get_rotation_speed`, improve typing

* Add `instamatic.calibrate_stage_rotation` for greater rotation control

* Add `instamatic.calibrate_stage_rotation` for greater rotation control 2

* Add first ideas for RATS-like generalized data collection frame

* `ImageDrawOverlay`: Don't print `self.operations` on every redraw

* GUI: FIX couldn't initialize with non-steamable camera

* `VideoStream`: Generalize real & fake streams under common base class

* GUI: Allow VideoStreamFrame for non-streamable cameras via the fake

* Allow customizing `NoOverwriteDict`'s `KeyOverwriteError` message

* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame`

* `ClickListener`: make `callback` optional and allow activating via `with` context

* Add `typing_extensions` as a dependency

* Move potential `typing_extensions` types to `instamatic._typing`

* Feature usefulness of new `instamatic._typing.Self`

* Move potential `typing_extensions` types to `instamatic._typing`

* Feature usefulness of new `instamatic._typing.Self`

* Exclude changes irrelevant to the `click_service` branch

* Add a custom `ClickEvent.__repr__`

* Add a custom `ClickEvent.__repr__`

* Move `instamatic.collections` to `instamatic._collections`

* Move `instamatic.collections` to `instamatic._collections`

* `ClickListener`: call `callback` only after putting event in `self.queue`

* `ClickListener`: call `callback` only after putting event in `self.queue`

* `_VS`: rename type variable to a more conventional `VideoStream_T`

* `_VS`: rename type variable to a more conventional `VideoStream_T`

* `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing`

* `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing`

* Do not pass numpy objects to the microscope.

* `CameraSimu.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet 2

* `TEMController.get_movie`: convert into `Generator`

* `ctrl.get_image`: fix `header_keys` could hold only 1 key as `str` not `tuple`

* `CalibMovieRate`: first draft of the movie delay calibration

* `TEMController.get_movie`: collect common metadata before the main loop for time precision

* `TEMController`: move ugly explicit `self.cam` check to a method decorator

* `CalibMovieDelays`: simplify and add multi-attempt mechanism

* `CalibMovieDelaysMapping`: add a common class for reading/writing mappings

* `CalibMovieDelaysMapping`: fix I/O, script, bugs, add custom `CalibError`

* FEI Tecnai upper speed limit seem to be around 0.2

* `CameraServal.get_movie`: rewrite as generator, streamline by PIL->tifffile

* `CameraBase.get_movie`: rewrite as generator

* `CameraMerlin.get_movie`: rewrite as generator

* `showcase_movie.py`: Add a temp script to feature get_movie streaming feature

* `CameraServal.get_movie`: Actually replace previous implementation this time

* `CalibMovieDelays`: vastly simplify, do for specific exposure only

* `CalibMovieDelays`: fix documentation, warning criteria

* Register `instamatic.calibrate.calibrate_movie_delays` as script

* Fix bugs, typos

* Adapt `test_get_movie` to the new structure

* Add Daniel Tchoń to `CITATION.cff`

* Implement simple cRED protocol in rats frame using `get_movie`

* Fix `write_tiff` raising when writing to non-existing directory

* Fix `DiffractionRun.middles` off-by-one error

* Wrap generators in `try/finally` to correctly close them when not exhausted

* RATS cRED: find and rotate with closest possible rotation speed

* Reorganize rats frame, lock unusable buttons

* `RatsFrame`: clean code formatting

* `RatsFrame`: make `Run.experiments` a property

* `RatsFrame`: can now extend/collect experiments in parts

* These things don't work on Tecnai (fix better)

* WHen using center, convert numpy types to python

* Disable plotting until its fixed.

* Adapt RATS experiment to updated movie calibration

* Add typing, align `stage.rotating_speed` to `stage.rotation_speed` to match already-used wording

* Fix beamshift deflector so that it passes `int` or `float` instead of `np.int`

* Fix wrong definitions of goniometer speed vs pace

* Fix mismatched axes order for the purpose of crystal tracking

* Generalize image, tracking, diffraction config re/store

* Correctly change config during RATS experiment

* Add docstrings, streamline blanking

* Add `with Beam().blanked()` and `with Beam().unblanked()`

* Apply the new beam `un/blanked` context in various places

* Various RATS frame fixes needed to run cRED on Tecnai

* New console print of carriage return, with debug statements

* Remove ANSI ESCAPE characters after all; this version handles tqdm

* Remove debug statements

* Improve RATS experiment: go to 0 at start, end, also collect image

* Add beam blank button, block can when collecting stills,

* Systematize `setStagePosition` signature, make `None` a sentinel for speed

* Fix TEMController alignments being incorrectly stored and restored

* Add generalized handling for PETS input using a dedicated factory

* Since we need to support python 3.7 and 3.8, move csv file inside py

* Produce just a single PETS file in RATS experiment

* Add generalized handling for PETS input using a dedicated factory

* Since we need to support python 3.7 and 3.8, move csv file inside py

* Fix name of produced pets file to `pets.pts`

* Fix name of produced pets file to `pets.pts`

* Fix creating `pets.pts` if prefix or suffix are not defined

* Fix creating `pets.pts` if prefix or suffix are not defined

* Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix

* Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix

* Add doc, tests, move csv to resources, bump min Python version to 3.9

* GitHub tests fail for Python 3.13, may be not supported by some libraries yet

* Simplify, accelerate the video provider, now `VideoServiceProcessor`

* Further simplify, make faster, allow `VideoStreamProcessor` to display figures

* Remove `[tool.setuptools.package-data]` from `pyproject.toml`

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* Fix typo in src/instamatic/processing/ImgConversion.py

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* Move `PetsInputWarning` to the top of the `PETS_input_factory.py` file

* Make `PETS_KEYWORDS` uppercase, fix `PetsKeywords.from_file` type hint

* Bump the suggested conda-forge Python version to 3.12

* Improve docstrings in `DeferredImageDraw` & `VideoStreamProcessor`

* `on_frame`: scale 2-fold, clear a bunch of empty lines

* `ExperimentalRats`: fix beam un/blanker

* `ExperimentalRats`: export tracking to its own function

* Merge `fix_pets_input` into `video_services`

* Revert code unrelated to strictly stream processor updates

* Improve `stream.processor.temporary_*`, docstring, add example code

* Fix bug: clicking on VideoStreamFrame.panel.image was 2px off

* Remove debug timing instructions

* Fix clicked point was incorrect if window scrolled or partially visible

* Rename `save_image` -> `save_frame`, new `save_image` job saves edited as PNG, remove unused imports

* Remove discontinued showcase script

* Fix incorrect yield placement in `processor.temporary` context

* Make `DeferredImageDraw.Instruction` mutable dataclass (& .5% slower)

* Make `DeferredImageDraw.Instruction` mutable dataclass (& .2% slower)

* Forcing keyword args here for safety.

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* Rewrite `VideoStreamProcessor.frame` return into a bit neater one-liner

* Fix unsightly docstring, remove unused import

* Improve type hints, docstrings

* Revert "Forcing keyword args here for safety."

This reverts commit 08806ff.

* Remove code left "ONLY FOR DEVELOPMENT SHOWCASE PURPOSES"

---------

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

Author: Baharis

src/instamatic/gui/jobs.py

@@ -9,7 +9,10 @@
 import time
 from datetime import datetime
 
+import PIL.Image
+
 from instamatic.formats import read_tiff, write_tiff
+from instamatic.processing.flatfield import apply_flatfield_correction
 
 
 def microscope_control(controller, **kwargs):
@@ -31,17 +34,15 @@ def collect_flatfield(controller, **kwargs):
     flatfield.collect_flatfield(controller.ctrl, confirm=False, drc=drc, **kwargs)
 
 
-def save_image(controller, **kwargs):
+def save_frame(controller, **kwargs):
     frame = kwargs.get('frame')
 
     module_io = controller.app.get_module('io')
 
     drc = module_io.get_experiment_directory()
     drc.mkdir(exist_ok=True, parents=True)
 
-    timestamp = datetime.now().strftime('%H-%M-%S.%f')[
-        :-3
-    ]  # cut last 3 digits for ms resolution
+    timestamp = datetime.now().strftime('%H-%M-%S.%f')[:-3]  # cut last 3 digits for ms res.
     outfile = drc / f'frame_{timestamp}.tiff'
 
     try:
@@ -55,6 +56,15 @@ def save_image(controller, **kwargs):
     print('Wrote file:', outfile)
 
 
+def save_image(controller, image: PIL.Image.Image, **_):
+    drc = controller.app.get_module('io').get_experiment_directory()
+    drc.mkdir(exist_ok=True, parents=True)
+    timestamp = datetime.now().strftime('%H-%M-%S.%f')[:-3]  # cut last 3 digits for ms res.
+    out_path = drc / f'image_{timestamp}.png'
+    image.save(out_path, format='PNG')
+    print('Wrote file:', out_path)
+
+
 def toggle_difffocus(controller, **kwargs):
     toggle = kwargs['toggle']
 
@@ -85,6 +95,7 @@ def relax_beam(controller, **kwargs):
 JOBS = {
     'ctrl': microscope_control,
     'flatfield': collect_flatfield,
+    'save_frame': save_frame,
     'save_image': save_image,
     'toggle_difffocus': toggle_difffocus,
     'relax_beam': relax_beam,

src/instamatic/gui/videostream_frame.py

@@ -2,17 +2,18 @@
 
 import threading
 import time
-from datetime import datetime
 from tkinter import *
+from tkinter import Label as TkLabel
 from tkinter.ttk import *
+from typing import Union
 
 import numpy as np
-from PIL import Image, ImageEnhance, ImageTk
+from PIL import Image, ImageTk
+from PIL.Image import Resampling
 
-from instamatic.formats import read_tiff, write_tiff
 from instamatic.gui.base_module import BaseModule
 from instamatic.gui.click_dispatcher import ClickDispatcher
-from instamatic.processing.flatfield import apply_flatfield_correction
+from instamatic.gui.videostream_processor import VideoStreamProcessor
 from instamatic.utils.spinbox import Spinbox
 
 
@@ -69,6 +70,7 @@ def __init__(self, parent, stream, app=None):
 
         self.click_dispatcher = ClickDispatcher()
         self.panel.bind('<Button>', self.on_click)
+        self.processor = VideoStreamProcessor(self)
 
     def init_vars(self):
         self.var_fps = DoubleVar()
@@ -92,8 +94,17 @@ def init_vars(self):
         self.var_auto_contrast.trace_add('write', self.update_auto_contrast)
 
     def buttonbox(self, master):
-        btn = Button(master, text='Save image', command=self.saveImage)
-        btn.pack(side='bottom', fill='both', padx=10, pady=10)
+        btn_frame = Frame(master)
+        btn_frame.pack(side='bottom', fill=BOTH, padx=10, pady=10)
+        btn1 = Button(btn_frame, text='Save frame', command=self.save_frame)
+        btn1.pack(side=LEFT, expand=True, fill='both')
+        btn2 = Button(btn_frame, text='Save image', command=self.save_image)
+        btn2.pack(side=LEFT, expand=True, fill='both')
+
+    @property
+    def frame(self) -> Union[np.ndarray, None]:
+        """Raw image frame from the camera."""
+        return self.processor.frame
 
     def header(self, master):
         ewidth = 8
@@ -169,14 +180,10 @@ def makepanel(self, master, resolution=(512, 512)):
         if self.panel is None:
             image = Image.fromarray(np.zeros(resolution))
             image = ImageTk.PhotoImage(image)
-
-            self.panel = Label(master, image=image)
+            self.panel = TkLabel(master, image=image, borderwidth=0)
             self.panel.image = image
             self.panel.pack(side='left', padx=10, pady=10)
 
-    def setup_stream(self):
-        pass
-
     def update_resize_image(self, name, index, mode):
         # print name, index, mode
         try:
@@ -214,9 +221,14 @@ def update_display_range(self, name, index, mode):
         except BaseException:
             pass
 
-    def saveImage(self):
-        """Dump the current frame to a file."""
-        self.q.put(('save_image', {'frame': self.frame}))
+    def save_frame(self):
+        """Save currently shown raw frame from the stream to a file in cwd."""
+        self.q.put(('save_frame', {'frame': self.frame}))
+        self.triggerEvent.set()
+
+    def save_image(self):
+        """Save currently shown, modified, & scaled image to a file in cwd."""
+        self.q.put(('save_image', {'image': self.processor.image}))
         self.triggerEvent.set()
 
     def set_trigger(self, trigger=None, q=None):
@@ -234,33 +246,17 @@ def start_stream(self):
         self.after(500, self.on_frame)
 
     def on_frame(self, event=None):
-        self.stream.lock.acquire(True)
-        self.frame = frame = self.stream.frame
-        self.stream.lock.release()
-
-        if frame is not None:
-            # the display range in ImageTk is from 0 to 255
-            if self.display_range != 255.0 or self.brightness != 1.0:
-                if self.auto_contrast:
-                    display_range = 1 + np.percentile(frame[::4, ::4], 99.5)
-                else:
-                    display_range = self.display_range
-                frame = (self.brightness * 255 / display_range) * frame
-            frame = np.clip(frame.astype(np.int16), 0, 255).astype(np.uint8)
-            image = Image.fromarray(frame)
-
+        """Get the newest image from `processor`, adapt to GUI and display."""
+        if self.frame is not None:
+            image = self.processor.image
             if self.resize_image:
-                image = image.resize((950, 950))
-
+                size = [2 * dim for dim in image.size]
+                image = image.resize(size=size, resample=Resampling.NEAREST)
             image = ImageTk.PhotoImage(image=image)
-
             self.panel.configure(image=image)
             # keep a reference to avoid premature garbage collection
             self.panel.image = image
-
         self.update_frametimes()
-        # self.parent.update_idletasks()
-
         self.after(self.frame_delay, self.on_frame)
 
     def update_frametimes(self):
@@ -290,13 +286,15 @@ def on_click(self, event: Event) -> None:
         if not self.click_dispatcher.active:
             return
 
+        # Correct for window offset due to scrolling or not fitting on screen
+        p = self.panel
+        offset_x = (p.winfo_width() - p.image.width()) // 2
+        offset_y = (p.winfo_height() - p.image.height()) // 2
+
         # Convert window coordinates to image coordinates
-        panel_width: int = self.panel.winfo_width()
-        panel_height: int = self.panel.winfo_height()
         array_shape = self.frame.shape
-        x = round(event.x * array_shape[1] / panel_width)
-        y = round(event.y * array_shape[0] / panel_height)
-
+        x = round((event.x - offset_x) * array_shape[1] / p.image.width())
+        y = round((event.y - offset_y) * array_shape[0] / p.image.height())
         self.click_dispatcher.handle_click(x=x, y=y, button=event.num)
 
 

src/instamatic/gui/videostream_processor.py

@@ -0,0 +1,187 @@
+from __future__ import annotations
+
+import io
+from collections import deque
+from contextlib import contextmanager
+from dataclasses import dataclass
+from functools import wraps
+from typing import Any, Iterator, Literal, Optional, Protocol, Union
+
+import numpy as np
+import PIL.Image
+from matplotlib.figure import Figure
+from PIL import Image, ImageDraw
+
+from instamatic.camera.videostream import VideoStream
+
+
+class VideoStreamFrameProtocol(Protocol):
+    """Mimics the `VideoStreamFrame` interface to avoid circular import."""
+
+    auto_contrast: bool = True
+    brightness: float = 1.0
+    display_range: int = 255
+    stream: VideoStream
+
+
+class DeferredImageDraw:
+    """Defer `ImageDraw` method calls: put them in deque, draw using `on`."""
+
+    @dataclass
+    class Instruction:
+        """Stores info about `ImageDraw` calls deferred by `__getattr__`."""
+
+        attr_name: str
+        args: tuple[Any, ...]
+        kwargs: dict[str, Any]
+
+    def __init__(self, draw: Optional[ImageDraw.ImageDraw] = None) -> None:
+        self._drawing = draw if draw else ImageDraw.Draw(Image.new('RGB', (1, 1)))
+        self.instructions: deque[DeferredImageDraw.Instruction] = deque()
+
+    def __getattr__(self, attr_name: str) -> Any:
+        """Get the first of `self.attr_name` and `self._drawing.attr_name`.
+
+        If the attribute is a method of the internal `ImageDraw` object,
+        return its wrapped version that defers it by appending a corresponding
+        `Instruction` to `self.instructions` to be run at render time instead.
+        `DeferredImageDraw.Instruction` instance returned this way is mutable
+        and can be deleted by calling `self.instructions.remove(instruction)`.
+        Otherwise, return the attribute of `DeferredImageDraw` instance as-is.
+        """
+        try:
+            attr = object.__getattribute__(self, attr_name)
+        except AttributeError as e:
+            reraise_on_fail = e
+            try:
+                attr = getattr(self._drawing, attr_name)
+            except AttributeError:
+                raise reraise_on_fail
+
+            if callable(attr):
+
+                @wraps(attr)
+                def wrapped(*args, **kwargs) -> DeferredImageDraw.Instruction:
+                    instruction = self.Instruction(attr_name, args, kwargs)
+                    self.instructions.append(instruction)
+                    return instruction
+
+                return wrapped  # do not call attr - delay it until _redraw()
+        return attr  # non-callable attr of self (if exists) or self._drawing
+
+    def on(self, image: Image.Image) -> Image.Image:
+        """Core method: draws all deferred `self.instructions` on image."""
+        self._drawing = ImageDraw.Draw(image)
+        for ins in self.instructions:
+            getattr(self._drawing, ins.attr_name)(*ins.args, **ins.kwargs)
+        return image
+
+    def circle(
+        self,
+        xy: tuple[int, int],
+        radius: float,
+        fill: Optional[Union[str, tuple[int, int, int]]] = None,
+        outline: Optional[Union[str, tuple[int, int, int]]] = None,
+        width: int = 1,
+    ) -> DeferredImageDraw.Instruction:
+        """Draw a circle by wrapping a call to `ImageDraw.ellipse`.
+
+        Since `ImageDraw.circle` was added only in Pillow 10.4.0, this
+        provides a backward-compatible way to draw circles using ellipses.
+        """
+        ellipse_xy = (xy[0] - radius, xy[1] - radius, xy[0] + radius, xy[1] + radius)
+        return self.ellipse(ellipse_xy, fill=fill, outline=outline, width=width)
+
+
+class VideoStreamProcessor:
+    """Encapsulate complex `VideoStreamFrame` frame/image processing.
+
+    This class handles converting mathematical operations behind efficiently
+    converting raw frames into images, rendering matplotlib figures as static
+    images in the video stream window, as well as drawing on top of images.
+
+    Streamed view can be altered by setting a temporary frame/image/figure.
+    Each of these can be set/reset via corresponding attribute, or set
+    temporarily via `with processor.temporary(frame/image/figure=...)` syntax.
+
+    Drawing is handled via the `draw` attribute of the `DeferredImageDraw`
+    class which acts as a deferred proxy for PIL.ImageDraw. Instructions,
+    instead of being applied directly on one frame only, are saved into the
+    `draw.instructions` deque and efficiently re-applied continuously.
+    """
+
+    def __init__(self, vsf: VideoStreamFrameProtocol) -> None:
+        self.vsf: VideoStreamFrameProtocol = vsf
+        self.draw: DeferredImageDraw = DeferredImageDraw()
+        self.color_mode: Literal['L', 'RGB'] = 'RGB'
+        self.temporary_frame: Optional[np.ndarray] = None
+        self.temporary_image: Optional[Image.Image] = None
+        self._temporary_figure: Optional[Figure] = None
+
+    @property
+    def frame(self) -> Union[np.ndarray, None]:
+        """The raw `np.ndarray` frame from the stream or `_temporary_frame`"""
+        return self.vsf.stream.frame if (t := self.temporary_frame) is None else t
+
+    @property
+    def image(self) -> Union[Image.Image, None]:
+        """Processed image with `draw.instructions`, or `_temporary_image`."""
+        if (temporary_image := self.temporary_image) is not None:
+            return temporary_image
+        if (frame := self.frame) is not None:
+            if self.vsf.display_range != 255.0 or self.vsf.brightness != 1.0:
+                if self.vsf.auto_contrast:
+                    display_range = 1 + np.percentile(frame[::4, ::4], 99.5)
+                else:
+                    display_range = self.vsf.display_range
+                frame = (self.vsf.brightness * 255 / display_range) * frame
+            frame = np.clip(frame.astype(np.int16), 0, 255).astype(np.uint8)
+        if self.draw.instructions:
+            image = Image.fromarray(frame).convert(self.color_mode)
+            self.draw.on(image)
+        else:
+            image = Image.fromarray(frame)
+        return image
+
+    def render_figure(self, figure: Figure) -> Image.Image:
+        """Convert a `Figure` into an `Image` to allow rendering it in GUI."""
+        buffer = io.BytesIO()
+        dpi = min(self.vsf.stream.frame.shape / figure.get_size_inches())
+        figure.savefig(buffer, format='png', dpi=dpi, bbox_inches='tight', pad_inches=0)
+        buffer.seek(0)
+        return Image.open(buffer).convert('RGBA')
+
+    @contextmanager
+    def temporary(
+        self,
+        *,
+        frame: Optional[np.ndarray] = None,
+        image: Optional[Image.Image] = None,
+        figure: Optional[Figure] = None,
+    ) -> Iterator[None]:
+        """Temporarily override the current frame/image/figure for rendering.
+
+        Use via context manager using a `with` statement with one of the args:
+            with processor.temporary(frame=..., image=..., figure=...):
+                ...
+        """
+        pre_context_values = self.temporary_frame, self.temporary_image
+        try:
+            if frame is not None:
+                self.temporary_frame = frame
+            if image is not None:
+                self.temporary_image = image
+            elif figure is not None:
+                self.temporary_image = self.render_figure(figure)
+            yield
+        finally:
+            self.temporary_frame, self.temporary_image = pre_context_values
+
+    @property
+    def temporary_figure(self) -> Figure:
+        return self._temporary_figure
+
+    @temporary_figure.setter
+    def temporary_figure(self, figure: Union[Figure, None]) -> None:
+        self._temporary_figure = figure
+        self.temporary_image = self.render_figure(figure) if figure else None

src/instamatic/microscope/components/stage.py

@@ -76,7 +76,7 @@ def set_with_speed(
             speed=speed,
         )
 
-    def set_rotation_speed(self, speed: Union[float, int] = 1) -> None:
+    def set_rotation_speed(self, speed: Number = 1) -> None:
         """Sets the stage (rotation) movement speed on the TEM."""
         self._tem.setRotationSpeed(value=speed)