Add stage rotation calib for a better control over rotation speed (#136)

* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame`

* Add `VideoStreamService` architecture and setup one in `VideoStreamFrame`

* Rename `VideoStreamEnhancers` to `Overlays`, make `ImageDrawOverlay` editable

* `ClickListener`: make `callback` optional and allow activating via `with` context

* `MediaGrabber`: fix bug: `get_media` cleared `continuousCollectionEvent`

* `calibrate_beamshift.py`: make imports absolute, allow float beamshift

* Instamatic GUI: temporarily print coords of clicked point, revert later

* Move `instamatic.typing` to `typing_` to avoid import conflicts

* `stage`: Add `get_rotation_speed`, improve typing

* Add `instamatic.calibrate_stage_rotation` for greater rotation control

* Add `instamatic.calibrate_stage_rotation` for greater rotation control 2

* Add first ideas for RATS-like generalized data collection frame

* `ImageDrawOverlay`: Don't print `self.operations` on every redraw

* GUI: FIX couldn't initialize with non-steamable camera

* `VideoStream`: Generalize real & fake streams under common base class

* GUI: Allow VideoStreamFrame for non-streamable cameras via the fake

* Allow customizing `NoOverwriteDict`'s `KeyOverwriteError` message

* Add `ClickDispatcher` architecture and setup one in `VideoStreamFrame`

* `ClickListener`: make `callback` optional and allow activating via `with` context

* Add `typing_extensions` as a dependency

* Move potential `typing_extensions` types to `instamatic._typing`

* Feature usefulness of new `instamatic._typing.Self`

* Move potential `typing_extensions` types to `instamatic._typing`

* Feature usefulness of new `instamatic._typing.Self`

* Exclude changes irrelevant to the `click_service` branch

* Add a custom `ClickEvent.__repr__`

* Add a custom `ClickEvent.__repr__`

* Move `instamatic.collections` to `instamatic._collections`

* Move `instamatic.collections` to `instamatic._collections`

* `ClickListener`: call `callback` only after putting event in `self.queue`

* `ClickListener`: call `callback` only after putting event in `self.queue`

* `_VS`: rename type variable to a more conventional `VideoStream_T`

* `_VS`: rename type variable to a more conventional `VideoStream_T`

* `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing`

* `typing_extensions`: uses typing if available so import directly, not via `instamatic._typing`

* Do not pass numpy objects to the microscope.

* `CameraSimu.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: convert into `Generator`

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet

* `VideoStream.get_movie`: Optimize thread release, remove unused event = best fps yet 2

* `TEMController.get_movie`: convert into `Generator`

* `ctrl.get_image`: fix `header_keys` could hold only 1 key as `str` not `tuple`

* `CalibMovieRate`: first draft of the movie delay calibration

* `TEMController.get_movie`: collect common metadata before the main loop for time precision

* `TEMController`: move ugly explicit `self.cam` check to a method decorator

* `CalibMovieDelays`: simplify and add multi-attempt mechanism

* `CalibMovieDelaysMapping`: add a common class for reading/writing mappings

* `CalibMovieDelaysMapping`: fix I/O, script, bugs, add custom `CalibError`

* FEI Tecnai upper speed limit seem to be around 0.2

* `CameraServal.get_movie`: rewrite as generator, streamline by PIL->tifffile

* `CameraBase.get_movie`: rewrite as generator

* `CameraMerlin.get_movie`: rewrite as generator

* `showcase_movie.py`: Add a temp script to feature get_movie streaming feature

* `CameraServal.get_movie`: Actually replace previous implementation this time

* `CalibMovieDelays`: vastly simplify, do for specific exposure only

* `CalibMovieDelays`: fix documentation, warning criteria

* Register `instamatic.calibrate.calibrate_movie_delays` as script

* Fix bugs, typos

* Adapt `test_get_movie` to the new structure

* Add Daniel Tchoń to `CITATION.cff`

* Implement simple cRED protocol in rats frame using `get_movie`

* Fix `write_tiff` raising when writing to non-existing directory

* Fix `DiffractionRun.middles` off-by-one error

* Wrap generators in `try/finally` to correctly close them when not exhausted

* RATS cRED: find and rotate with closest possible rotation speed

* Reorganize rats frame, lock unusable buttons

* `RatsFrame`: clean code formatting

* `RatsFrame`: make `Run.experiments` a property

* `RatsFrame`: can now extend/collect experiments in parts

* These things don't work on Tecnai (fix better)

* WHen using center, convert numpy types to python

* Disable plotting until its fixed.

* Adapt RATS experiment to updated movie calibration

* Add typing, align `stage.rotating_speed` to `stage.rotation_speed` to match already-used wording

* Fix beamshift deflector so that it passes `int` or `float` instead of `np.int`

* Fix wrong definitions of goniometer speed vs pace

* Fix mismatched axes order for the purpose of crystal tracking

* Generalize image, tracking, diffraction config re/store

* Correctly change config during RATS experiment

* Add docstrings, streamline blanking

* Add `with Beam().blanked()` and `with Beam().unblanked()`

* Apply the new beam `un/blanked` context in various places

* Various RATS frame fixes needed to run cRED on Tecnai

* New console print of carriage return, with debug statements

* Remove ANSI ESCAPE characters after all; this version handles tqdm

* Remove debug statements

* Improve RATS experiment: go to 0 at start, end, also collect image

* Add beam blank button, block can when collecting stills,

* Systematize `setStagePosition` signature, make `None` a sentinel for speed

* Fix TEMController alignments being incorrectly stored and restored

* Add generalized handling for PETS input using a dedicated factory

* Since we need to support python 3.7 and 3.8, move csv file inside py

* Produce just a single PETS file in RATS experiment

* Add generalized handling for PETS input using a dedicated factory

* Since we need to support python 3.7 and 3.8, move csv file inside py

* Fix name of produced pets file to `pets.pts`

* Fix name of produced pets file to `pets.pts`

* Fix creating `pets.pts` if prefix or suffix are not defined

* Fix creating `pets.pts` if prefix or suffix are not defined

* Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix

* Split PETS affix to blocks pre-adding: rejects only duplicate lines, not all affix

* Add doc, tests, move csv to resources, bump min Python version to 3.9

* GitHub tests fail for Python 3.13, may be not supported by some libraries yet

* Simplify, accelerate the video provider, now `VideoServiceProcessor`

* Further simplify, make faster, allow `VideoStreamProcessor` to display figures

* Remove `[tool.setuptools.package-data]` from `pyproject.toml`

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* Fix typo in src/instamatic/processing/ImgConversion.py

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

* Move `PetsInputWarning` to the top of the `PETS_input_factory.py` file

* Make `PETS_KEYWORDS` uppercase, fix `PetsKeywords.from_file` type hint

* Bump the suggested conda-forge Python version to 3.12

* Improve docstrings in `DeferredImageDraw` & `VideoStreamProcessor`

* `on_frame`: scale 2-fold, clear a bunch of empty lines

* `ExperimentalRats`: fix beam un/blanker

* `ExperimentalRats`: export tracking to its own function

* Merge `fix_pets_input` into `video_services`

* Revert code unrelated to strictly stream processor updates

* Improve `stream.processor.temporary_*`, docstring, add example code

* Fix bug: clicking on VideoStreamFrame.panel.image was 2px off

* Remove debug timing instructions

* Fix clicked point was incorrect if window scrolled or partially visible

* Rename `save_image` -> `save_frame`, new `save_image` job saves edited as PNG, remove unused imports

* Remove discontinued showcase script

* Fix incorrect yield placement in `processor.temporary` context

* Revert "Revert code unrelated to strictly stream processor updates"

This reverts commit a607a77.

* Align with changes suggested in #135 (temporary as common context)

* Fix bug: continuous geometry not interpreted correctly for PETS

* Improve CalibStageRotation docstrings & naming

* Switch from json to yaml, fix plotting, add docstring

* Clear calibration of comments undesired in instamatic, shorten code

* Revert changes unrelated directly to the stage rotation calibration

* Revert code left for development showcase purposes only

* Revert changes associated with PR #135 specifically

* Python 3.7, 3.8's `typing` does not feature `Self`

* Address @stefsmeets review: dataclasses & simplify

* Trim out unnecessary method `CalibStageRotation._alpha_poly`

* Add greater control over speed settings restrictions

* Improve `instamatic.calibrate_stage_rotation` docstring

* Minor fixes to documentation, toml found at final review.

---------

Co-authored-by: Stef Smeets <stefsmeets@users.noreply.github.com>

Author: Baharis

docs/programs.md

@@ -28,6 +28,8 @@ These tools help calibrate instamatic for some experiments.
 - [instamatic.calibrate_movie_delays](#instamaticcalibrate_movie_delays) (`instamatic.calibrate.calibrate_movie_delays:main_entry`)
 - [instamatic.flatfield](#instamaticflatfield) (`instamatic.processing.flatfield:main_entry`)
 - [instamatic.stretch_correction](#instamaticstretch_correction) (`instamatic.processing.stretch_correction:main_entry`)
+- [instamatic.calibrate_stage_rotation](#instamaticcalibrate_stage_rotation) (`instamatic.calibrate.calibrate_stage_lowmag:main_entry`)
+
 
 **Tools**
 
@@ -176,6 +178,32 @@ instamatic.calibrate_stage_lowmag [-h] [IMG [IMG ...]]
 `-h`, `--help`
 : Show this help message and exit  
 
+## instamatic.calibrate_stage_rotation
+
+Program to calibrate and plan the rotation pace (in seconds / angle) against speed settings available on the microscope.
+
+**Usage:**  
+```bash
+instamatic.calibrate_stage_rotation [-h] [-a ALPHAS] [-s SPEEDS] [-m MODE] [-o OUTDIR]
+```
+
+**Optional arguments:**  
+
+`-h`, `--help`
+: Show this help message and exit  
+
+`-a ALPHAS`, `--alphas ALPHAS`
+: Comma-delimited list of alpha spans to calibrate (default: "1,2,3,4,5,6,7,8,9,10").
+
+`-s SPEEDS`, `--speeds SPEEDS`
+: Comma-delimited list of speed settings to calibrate (default: "0.1,0.2,0.3,0.4,0.5,0.6,0.7,0.8,0.9,1.0" or "1,2,3,4,5,6,7,8,9,10,11,12", whichever is accepted by the microscope).
+
+`-m MODE`, `--mode MODE`
+: Calibration mode to be used: "auto" - auto-determine upper and lower speed limits based on TEM response (default); "limited" - restrict TEM goniometer speed limits between min and max of --speeds; or "listed" - restrict TEM goniometer speed settings exactly to --speeds provided.
+
+`-o OUTDIR`, `--outdir OUTDIR`
+: Path to the directory where calibration file should be output (default: "%appdata%/calib" on Windows or "$AppData/calib" on Unix).
+
 
 ## instamatic.calibrate_stage_mag1
 

pyproject.toml

@@ -104,6 +104,7 @@ publishing = [
 "instamatic.calibrate_movie_delays" = "instamatic.calibrate.calibrate_movie_delays:main_entry"
 "instamatic.flatfield" = "instamatic.processing.flatfield:main_entry"
 "instamatic.stretch_correction" = "instamatic.processing.stretch_correction:main_entry"
+"instamatic.calibrate_stage_rotation" = "instamatic.calibrate.calibrate_stage_rotation:main_entry"
 # tools
 "instamatic.browser" = "scripts.browser:main"
 "instamatic.viewer" = "scripts.viewer:main"

src/instamatic/calibrate/__init__.py

@@ -5,5 +5,6 @@
 from .calibrate_directbeam import CalibDirectBeam
 from .calibrate_movie_delays import CalibMovieDelays
 from .calibrate_stage_lowmag import CalibStage
+from .calibrate_stage_rotation import CalibStageRotation
 
 # from .calibrate_stage_mag1 import CalibStageMag1

src/instamatic/calibrate/calibrate_movie_delays.py

@@ -73,10 +73,12 @@ def from_file(
         calib_conditions = (exposure, header_keys, header_keys_common)
         calib = calib_map.get(calib_conditions, None)
         if calib is None:
+            header_keys_str = ','.join(header_keys)
+            header_keys_common_str = ','.join(header_keys_common)
             raise KeyError(
                 f'Calibration for conditions {calib_conditions} not found. '
                 f"Please run 'instamatic.calibrate_movie_delays -e {exposure:.6f} "
-                f"-a {header_keys} -c {header_keys_common}' first."
+                f'-a "{header_keys_str}" -c "{header_keys_common_str}" first.'
             )
         return calib
 
@@ -188,8 +190,8 @@ def from_file(cls, path: Optional[str] = None) -> Self:
         if path is None:
             path = Path(calibration_drc) / CALIB_MOVIE_DELAYS
         try:
-            with open(Path(path), 'r') as json_file:
-                return cls.from_list(yaml.safe_load(json_file))
+            with open(Path(path), 'r') as yaml_file:
+                return cls.from_list(yaml.safe_load(yaml_file))
         except OSError as e:
             prog = 'instamatic.calibrate_movie_delays'
             raise OSError(f'{e.strerror}: {path}. Please run {prog} first.')
@@ -200,6 +202,7 @@ def to_file(self, path: Optional[str] = None) -> None:
             path = Path(calibration_drc) / CALIB_MOVIE_DELAYS
         with open(Path(path), 'w') as json_file:
             yaml.safe_dump(self.to_list(), json_file)
+        log(f'{self} saved to {path}.')
 
 
 class MovieTimes: