Resolved #107 -- Added support for line & cell magics by subclassing MetaKernel #109

a-apple1234 · 2025-05-23T18:32:27Z

Subclassed MetaKernel to implement custom line and cell magic commands
Updated output methods to follow MetaKernel’s conventions
Adjusted execution logic to align with MetaKernel's do_execute function

MaxCan-Code · 2025-05-23T18:48:18Z

Thanks for the PR. You can add the MetaKernel requirement at:

Lines 23 to 25 in ef152d3

    
           install_requires=[ 
        
               "ipykernel>=6.20.0", 
        
           ],

a-apple1234 · 2025-05-23T20:01:02Z

done

MaxCan-Code · 2025-05-23T20:51:33Z

Try squashing the remove import commit: https://youtu.be/42392W7SgnE?t=334

a-apple1234 · 2025-05-23T21:03:22Z

done

MaxCan-Code · 2025-05-23T22:05:11Z

There's an extra commit by @martanit. You can delete that commit in git rebase -i.

a-apple1234 · 2025-05-24T04:10:48Z

done

setup.py

a-apple1234 · 2025-05-24T17:52:58Z

done

MaxCan-Code · 2025-05-26T13:29:10Z

My notebook hangs when I try to run a cell. Is there a working demo I can try?

`jupyter notebook` output

[I 2025-05-26 09:10:42.242 ServerApp] Connecting to kernel dc8407c1-2312-4ce1-87cf-ed61e0fdc3d9.
[W 2025-05-26 09:10:42.254 ServerApp] The websocket_ping_timeout (90000) cannot be longer than the websocket_ping_interval (30000).
    Setting websocket_ping_timeout=30000
[W 2025-05-26 09:10:42.537 ServerApp] Timeout waiting for kernel_info reply from dc8407c1-2312-4ce1-87cf-ed61e0fdc3d9
[E 2025-05-26 09:10:42.537 ServerApp] Uncaught exception GET /api/kernels/dc8407c1-2312-4ce1-87cf-ed61e0fdc3d9/channels?session_id=4b49a55f-a99c-4b30-b627-b1c68abad581 (127.0.0.1)
    HTTPServerRequest(protocol='http', host='localhost:8888', method='GET', uri='/api/kernels/dc8407c1-2312-4ce1-87cf-ed61e0fdc3d9/channels?session_id=4b49a55f-a99c-4b30-b627-b1c68abad581', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/tornado/web.py", line 1848, in _execute
        result = await result
                 ^^^^^^^^^^^^
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_server/services/kernels/websocket.py", line 66, in get
        await super().get(kernel_id=kernel_id)
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/tornado/websocket.py", line 277, in get
        await self.ws_connection.accept_connection(self)
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/tornado/websocket.py", line 890, in accept_connection
        await self._accept_connection(handler)
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/tornado/websocket.py", line 930, in _accept_connection
        self.selected_subprotocol = handler.select_subprotocol(subprotocols)
                                    ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
      File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_server/services/kernels/websocket.py", line 88, in select_subprotocol
        preferred_protocol = self.connection.kernel_ws_protocol
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    AttributeError: 'NoneType' object has no attribute 'kernel_ws_protocol'
[E 2025-05-26 09:10:42.574 ServerApp] {
      "Host": "localhost:8888",
      "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) QtWebEngine/6.8.2 Chrome/122.0.6261.171 Safari/537.36"
    }
[E 2025-05-26 09:17:43.076 ServerApp] 500 GET /api/kernels/c17a455b-c217-4a9d-9c34-dbdcb22425c8/channels?session_id=be0bbd1f-a5d1-446e-9ae6-e8

a-apple1234 · 2025-05-26T20:42:34Z

Entering the following commands into Terminal on macOS successfully set up the kernel for me:

git clone -b implement-magics https://github.com/dwijy-piggy/dyalog-jupyter-kernel
jupyter kernelspec install dyalog-jupyter-kernel/dyalog_kernel/dyalog_apl --user

If you are using just a kernel.py and kernel.json file to test this locally, the error might occur if you aren't starting the kernel using IPKernelApp.launch_instance(kernel_class=DyalogKernel) within the kernel.py file. However, cloning the implement-magics branch of my forked repo and installing the kernel from there should get it working.

MaxCan-Code · 2025-05-27T20:38:13Z

Still the same problem. I'm on Linux (NixOS).

However it works when I tried it on Windows.

a-apple1234 · 2025-05-27T20:52:45Z

Could you send the output from this command: jupyter console --kernel=dyalog_apl --debug?

MaxCan-Code · 2025-05-27T21:31:44Z

The main branch displays the same SyntaxWarning: invalid escape sequence message, then proceeds to work normally.

`jupyter console --kernel=dyalog_apl --debug`

$ jupyter console --kernel=dyalog_apl --debug
[ZMQTerminalIPythonApp] Searching ['/home/user/p/ad/dyalog-jupyter-notebooks/.venv/etc/jupyter', '/home/user/.jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[ZMQTerminalIPythonApp] Looking for jupyter_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /home/user/.jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_config in /home/user/p/ad/dyalog-jupyter-notebooks/.venv/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /usr/local/etc/jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /home/user/.jupyter
[ZMQTerminalIPythonApp] Looking for jupyter_console_config in /home/user/p/ad/dyalog-jupyter-notebooks/.venv/etc/jupyter
[ZMQTerminalIPythonApp] Connection File not found: /home/user/.local/share/jupyter/runtime/kernel-21830.json
[ZMQTerminalIPythonApp] Instantiating kernel 'Dyalog APL' with kernel provisioner: local-provisioner
[ZMQTerminalIPythonApp] Starting kernel: ['/home/user/p/ad/dyalog-jupyter-notebooks/.venv/bin/python3', '-m', 'dyalog_kernel', '-f', '/home/user/.local/share/jupyter/runtime/kernel-21830.json']
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:46129
[ZMQTerminalIPythonApp] connecting iopub channel to tcp://127.0.0.1:35829
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:35829
[ZMQTerminalIPythonApp] connecting shell channel to tcp://127.0.0.1:37835
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:37835
[ZMQTerminalIPythonApp] connecting stdin channel to tcp://127.0.0.1:35037
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:35037
[ZMQTerminalIPythonApp] connecting heartbeat channel to tcp://127.0.0.1:44739
[ZMQTerminalIPythonApp] connecting control channel to tcp://127.0.0.1:46129
[ZMQTerminalIPythonApp] Connecting to: tcp://127.0.0.1:46129
/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/dyalog_kernel/kernel.py:205: SyntaxWarning: invalid escape sequence '\d'
  if re.match('^Dyalog-\d+\.\d+\.app$', d):
/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/dyalog_kernel/kernel.py:209: SyntaxWarning: invalid escape sequence '\d'
  if re.match('^\d+\.\d+$', v):
/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/dyalog_kernel/kernel.py:298: SyntaxWarning: invalid escape sequence '\s'
  match = re.search('^%suspend\s+(\w+)$',lines[0].lower(), re.IGNORECASE)
Traceback (most recent call last):
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_core/utils/__init__.py", line 154, in wrapped
    asyncio.get_running_loop()
    ~~~~~~~~~~~~~~~~~~~~~~~~^^
RuntimeError: no running event loop

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_console/ptshell.py", line 434, in init_kernel_info
    reply = self.client.get_shell_msg(timeout=1)
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_core/utils/__init__.py", line 158, in wrapped
    return loop.run_until_complete(inner)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/home/user/.local/share/uv/python/cpython-3.13.2-linux-x86_64-gnu/lib/python3.13/asyncio/base_events.py", line 725, in run_until_complete
    return future.result()
           ~~~~~~~~~~~~~^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_client/client.py", line 143, in _async_get_shell_msg
    return await ensure_async(self.shell_channel.get_msg(*args, **kwargs))
                              ~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_client/channels.py", line 232, in get_msg
    raise Empty
_queue.Empty

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/bin/jupyter-console", line 10, in <module>
    sys.exit(main())
             ~~~~^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_core/application.py", line 284, in launch_instance
    super().launch_instance(argv=argv, **kwargs)
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
    ~~~~~~~~~~~~~~^^^^^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_console/app.py", line 136, in initialize
    self.init_shell()
    ~~~~~~~~~~~~~~~^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_console/app.py", line 106, in init_shell
    self.shell = ZMQTerminalInteractiveShell.instance(parent=self,
                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                    manager=self.kernel_manager,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                    client=self.kernel_client,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
                    confirm_exit=self.confirm_exit,
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/traitlets/config/configurable.py", line 583, in instance
    inst = cls(*args, **kwargs)
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_console/ptshell.py", line 352, in __init__
    self.init_kernel_info()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/home/user/p/ad/dyalog-jupyter-notebooks/.venv/lib/python3.13/site-packages/jupyter_console/ptshell.py", line 437, in init_kernel_info
    raise RuntimeError("Kernel didn't respond to kernel_info_request") from e
RuntimeError: Kernel didn't respond to kernel_info_request

MaxCan-Code · 2025-05-28T12:39:11Z

Still the same problem. I'm on Linux (NixOS).

However it works when I tried it on Windows.

@xpqz can you see if it works on your end? Thanks

MaxCan-Code · 2025-06-02T14:29:31Z

Still the same problem. I'm on Linux (NixOS).
However it works when I tried it on Windows.

@martanit @sloorush can you test this as well?

martanit · 2025-06-04T06:35:45Z

Works for me on Windows 👍!

xpqz · 2025-06-11T08:56:22Z

It would be good to see an outline of the consequences of switching from ipykernel.kernelbase to MetaKernel as the baseclass. Whilst convenient, are there any implications or future limitations we should be aware of?

martanit · 2025-07-21T11:41:24Z

Hi @MaxCan-Code and @dwijy-piggy, I successfully installed the kernel on a Ubuntu VM (by following more or less https://github.com/Dyalog/dyalog-jupyter-kernel/blob/master/DEVELOPMENT.md and testing it locally). @MaxCan-Code did you managed to install it on your OS? Anyway, before a merge, it would be nice to have more details on advantages/disadvantages of using MetaKernel, as @xpqz is asking. Thanks!

MaxCan-Code · 2025-07-22T02:52:50Z

I'm still getting the same problem on my end: #109 (comment).
The issue is likely just my setup.

MaxCan-Code · 2025-07-26T11:39:57Z

Got it to work by bypassing all magic by adding

def register_magics(self, magic_klass):
    return []

to the DyalogKernel class in kernel.py as described in Calysto/metakernel#287.

Looks like an upstream issue related to Nix.

…ssing MetaKernel - Subclassed MetaKernel to implement custom line and cell magic commands - Updated output methods to follow MetaKernel’s conventions - Adjusted execution logic to align with MetaKernel's do_execute function

…deleted comments

MaxCan-Code · 2025-07-27T18:12:38Z

Got it to work by bypassing all magic by adding
def register_magics(self, magic_klass):
    return []
to the DyalogKernel class in kernel.py as described in Calysto/metakernel#287.

Looks like an upstream issue related to Nix.

Never mind. The .inputrc¹ file I'm using was messing with the shell magic setup. Everything worked as expected after moving that elsewhere.

https://gist.github.com/0racle/bdaeb945cf42de317a48db7b1529f0fe ↩

a-apple1234 · 2025-07-28T00:51:09Z

It would be good to see an outline of the consequences of switching from ipykernel.kernelbase to MetaKernel as the baseclass. Whilst convenient, are there any implications or future limitations we should be aware of?

@xpqz MetaKernel is actively maintained and is used by several other kernels. It essentially just checks for and executes any magic commands before deferring to the code defined in the do_execute_direct method, so it doesn’t interfere with the core functionality of the Dyalog Kernel.

rikedyp · 2025-10-14T08:46:38Z

I'm trying this with Jupyter lab and get extra output.

FWIW, the kernel magic seems to be working

I installed by checking out this pull request and doing

python -m pip install .
python -m 'dyalog_kernel' install

a-apple1234 changed the title ~~Resolved #107 -- Added support for line & cell magics by subclassing MetaKernel~~ removed unnecessary import May 23, 2025

a-apple1234 changed the title ~~removed unnecessary import~~ Resolved #107 -- Added support for line & cell magics by subclassing MetaKernel May 23, 2025

a-apple1234 force-pushed the implement-magics branch 2 times, most recently from 80342d9 to f3cebc6 Compare May 23, 2025 19:38

a-apple1234 mentioned this pull request May 23, 2025

Resolve 107 -- Add support for line & cell magics by subclassing MetaKernel #108

Closed

a-apple1234 force-pushed the implement-magics branch from f3cebc6 to a402e3d Compare May 23, 2025 21:02

a-apple1234 force-pushed the implement-magics branch from a402e3d to 719b48d Compare May 24, 2025 04:10

MaxCan-Code reviewed May 24, 2025

View reviewed changes

setup.py Show resolved Hide resolved

a-apple1234 force-pushed the implement-magics branch from bdbe457 to b26bc21 Compare May 24, 2025 19:31

dwijm12 added 2 commits July 27, 2025 00:55

added requirement for MetaKernel

3c50b89

removed redundant ipykernel requirement and restored unintentionally …

232a283

…deleted comments

a-apple1234 force-pushed the implement-magics branch from b26bc21 to 232a283 Compare July 27, 2025 05:55

MaxCan-Code approved these changes Sep 9, 2025

View reviewed changes

Resolved #107 -- Added support for line & cell magics by subclassing MetaKernel #109

Are you sure you want to change the base?

Resolved #107 -- Added support for line & cell magics by subclassing MetaKernel #109

Uh oh!

Conversation

a-apple1234 commented May 23, 2025

Uh oh!

MaxCan-Code commented May 23, 2025

Uh oh!

a-apple1234 commented May 23, 2025

Uh oh!

MaxCan-Code commented May 23, 2025

Uh oh!

a-apple1234 commented May 23, 2025

Uh oh!

MaxCan-Code commented May 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

a-apple1234 commented May 24, 2025

Uh oh!

Uh oh!

a-apple1234 commented May 24, 2025

Uh oh!

MaxCan-Code commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

a-apple1234 commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MaxCan-Code commented May 27, 2025

Uh oh!

a-apple1234 commented May 27, 2025

Uh oh!

MaxCan-Code commented May 27, 2025

Uh oh!

MaxCan-Code commented May 28, 2025

Uh oh!

MaxCan-Code commented Jun 2, 2025

Uh oh!

martanit commented Jun 4, 2025

Uh oh!

xpqz commented Jun 11, 2025

Uh oh!

martanit commented Jul 21, 2025

Uh oh!

MaxCan-Code commented Jul 22, 2025

Uh oh!

MaxCan-Code commented Jul 26, 2025

Uh oh!

MaxCan-Code commented Jul 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Footnotes

Uh oh!

a-apple1234 commented Jul 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rikedyp commented Oct 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

MaxCan-Code commented May 23, 2025 •

edited

Loading

MaxCan-Code commented May 26, 2025 •

edited

Loading

a-apple1234 commented May 26, 2025 •

edited

Loading

MaxCan-Code commented Jul 27, 2025 •

edited

Loading

a-apple1234 commented Jul 28, 2025 •

edited

Loading