WebDriver commands to enable and disable full accessibility support in browser

[spectranaut]

Problem:

There is no cross-browser, cross-platform way to enable accessibility for a browser.

Context:

In order for a browser to work with assistive technologies, the browser must implement various accessibility APIs. Because of the extra computation needed to expose these accessibility APIs for every web page, accessibility support is not enabled by default.

Browsers use a variety of different heuristics to decide whether or not to enable the accessibility support, and these heuristics are not only different between browsers but are also often platform dependent -- for example, testing whether a screen reader is running.

Browsers are also capable of enabling only partial support. For example, the WebDriver commands getComputedRole and getComputedLabel will only enable minimal, platform-independent accessibility support -- it does not enable everything that a screen reader needs to provide access to content. This is true for the "accessibility tree" view in DevTools as well. As a result, we currently lack a single, reliable way to perform complete testing of accessibility implementations for all platforms.

Because there are many levels of accessibility support, I'm going to use the term "full accessibility" to mean that the browser can be operated with a screen reader.

Proposal:

A set of WebDriver endpoints to enable full accessibility within a browser.

Ideally the endpoints would be able to do the following:

1. Query to see whether or not full accessibility is enabled.
2. Turn on full accessibility.
3. Turn off full accessibility.

Possibly:

/session/{session id}/accessibility/status
/session/{session id}/accessibility/enable
/session/{session id}/accessibility/disable

How this API could be used in testing

This API would enable two related but different testing scenarios:

1. Make it possible to reliably turn on full accessibility support during tests.
   a. This is particularly in test environments where, for example, you won't find a screen reader running.
2. During the run of a test suite, make it possible to turn on full accessibility support for only a subset of tests without having to restart the browser or slow down non-accessibility related tests
   a. This is particularly critical for running accessibility tests in the Web Platform Test suite.

The ability to turn on accessibility in any test suite that uses WebDriver would also allow for:

1. Testing differences in performance with accessibility on and off.
2. Catching crashes that might originate from accessibility support code.

Adding this API would improve testing of a part of the web platform that is absolutely critical for many individuals. Accessibility testing is often far behind the rest of browser testing, and this feature could get us a little closer to creating stable access to the web for assistive technology users.

This solution will also help solve immediate problems with running the new accessibility API tests in the wpt.fyi infrastructure: web-platform-tests/wpt#53733

[whimboo]

See w3c/webdriver-bidi#443 for the WebDriver BiDi related issue. As of this point I wonder if such a feature should only be present in BiDi.

[spectranaut]

Thanks for the link to the bidi issue -- the discussion in the bidi issue is only about what we call the "internal accessibility tree", and exposing that tree only requires enabling partial accessibility support in the browser. The "internal accessiblity tree" is what is displayed in dev tools and what would be used for the kind of tree dump discussed in that issue. For the issues being discussed in the bidi issue, the feature introduced here (turning on and off full accessibility) is not necessary.

I'd like to understand more the benefits/reasons for introducing an API like this one to classic vs bidi?

[jcsteh]

3. Turn off full accessibility.

...

2. During the run of a test suite, make it possible to turn on full accessibility support for only a subset of tests without having to restart the browser

We may run into implementation limitations in browser accessibility engines here. For Gecko, while it's possible to shut down the accessibility engine if it was only initialised for partial, platform independent usage (e.g. getComputedLabel/Role), it is not currently possible to shut down the accessibility engine if it was instantiated for full, platform specific usage (what you're requesting here). We've never put time into supporting this because outside of a testing context, it's not feasible to shut down the accessibility engine for clients. This is because various layers of the accessibility stack (clients, OS components, etc.) outside of our control might hold onto references to our objects for longer (potentially much, much longer) than we expect. While it should theoretically be possible to detach the OS objects so that all queries just fail gracefully, it's somewhat risky in practice if we get anything wrong. It's also unclear when we should shut down, although that's less relevant here because there would be an explicit shut down request. I'm sure we could find a way to implement an explicit shut down, but I suspect we may hit snags along the way that may make it non-trivial. I know Chromium has done some work in this area, but my (possibly flawed) understanding is that Chromium ran into some similar issues.

[spectranaut]

Thanks for the thoughts @jcsteh. It's a good point that turning off accessibility will be more difficult to implement..

Also we talked about this in the wpt accessibility interop investigation meeting -- for the WPT case, it's possible that turning off accessibility for speed consideration is not that necessary, because when running all of wpt, the tests are "batched". If you turn on accessibility, it will only be on for the rest of the batch.

So for now, I might want to just focus on adding an API end point for turning on accessibility. If there is a more motivating use case for turning it off, we can do that later.

[jcsteh]

If we're only allowing this to be turned on and not off (at least initially), what is the drawback of using command line flags or prefs when starting the browser (abstracted by the harness) as the WPT PR already does? What is the need for (or advantage of) using WebDriver for this?

Another thing to consider is that starting the accessibility engine after the browser has already started may not be synchronous. That is, the WebDriver method might return, but there may be a short delay before the accessibility tree is "ready" due to asynchronous pieces of accessibility engine startup and tree building.

[zcorpan]

Another approach we discussed was for tests to declare themselves as needing a11y APIs enabled, for example as a file name flag. The test runner can then handle enabling and batching as appropriate.

[spectranaut]

If we're only allowing this to be turned on and not off (at least initially), what is the drawback of using command line flags or prefs when starting the browser (abstracted by the harness) as the WPT PR already does? What is the need for (or advantage of) using WebDriver for this?

The goal is that we can have tests in WPT that require accessibility APIs to be enabled, and for those tests to run successfully when someone is running a full run of WPT while not having to enable accessibility for every single test. When doing a full run, once the wpt test runner gets to the accessibility API tests, the browser will have already been started, so the commandline flags aren't very useful.

Also I like the idea of have a cross-browser, public way of enabling accessibility, rather than obscure commandline flags, but I'm not sure whether this use case is motivating enough to get that, we can continue to discuss :)

Related to whether it's worth it, @zcorpan helped me realized I should look into how WPT does "chunking" -- when a full run of WPT happens, the tests are typically chunked for parallelization, and there is some meta data about the chunks, like test type. I think it would be possible to say this chunk has accessibility API tests, so for this chunk, start the browser with the accessibility commandline flag.

Another thing to consider is that starting the accessibility engine after the browser has already started may not be synchronous. That is, the WebDriver method might return, but there may be a short delay before the accessibility tree is "ready" due to asynchronous pieces of accessibility engine startup and tree building.

That is true, there is no way around having to poll to wait for accessibility to be turned on, as far as I know. But we already have to poll to wait for the page to appear in the accessibility API when starting the browser with the accessibility flags.

[jcsteh]

The goal is that we can have tests in WPT that require accessibility APIs to be enabled, and for those tests to run successfully when someone is running a full run of WPT while not having to enable accessibility for every single test.

Understood. But once the accessibility tests run, because we can't (currently) shut down the engine, the tests after the accessibility tests will still have accessibility enabled. As I understand it, that means we aren't really solving the problem at hand here.

Also I like the idea of have a cross-browser, public way of enabling accessibility, rather than obscure commandline flags, but I'm not sure whether this use case is motivating enough to get that, we can continue to discuss :)

I'd like to understand other motivations for that. I think this is only useful in the context of automated testing, and for that case, there might be easier solutions; e.g. test metadata for chunking.

[sadym-chromium]

Can WPT test feature help solving the problem?