[Accessibility-testing] Defining our list of a11y tests

Jason McIntosh jmac at jmac.org
Fri Mar 2 23:09:33 EST 2018


This email got long. Zarf successfully nudged me back onto the path y’all had already established last year, and I proceed to review and digest the autumn 2017 list discussion out loud. Please bear with me...

On Mar 2, 2018, at 5:14 PM, Andrew Plotkin <erkyrath at eblong.com> wrote:

> The point of WCAG is to give *high*-level guidelines. Not low-level.

Got it; I had a misunderstanding of WCAG, but I’ve read up a bit on it now, and at least have the gist of it down. Thanks for the nudge!

In fact… this helps gel for me a lot of discussion from later last year regarding WCAG, and the Testathon’s use of it. I see that Deborah and Zack specifically suggested testing for WCAG 2.0 level A, and then discussion went on from there, through the late-September meeting that Dan thoroughly summarized in his post of October 5.

So we were talking about basing our final list of tests on WCAG-2-A, and holding work like Includification more as an example application of WCAG than something to use as our own basis. Right?

Let me quote some key passages from Dan’s October post, just to refresh them into the present (and since, modulo Furkle’s Twine example, I believe that it was the last major bit of activity on this list in 2017).

> We started the meeting by talking through the four WCAG "Principles of Accessibility:" Perceivable, Operable, Understandable, and Robust. How do these principles apply to IF and to our testing?
> 
> 	• Perceivable: Can you start up the game and read the content? It doesn't have to be automatically read.
> 	• Operable: Can you communicate with the game and play the game to completion? Can you do any necessary out of world processing, e.g. use menus?
> 	• Understandable: This sounds similar to "Perceivable," but "Perceivable" might mean that you're aware that there's an image in the game, you just can't understand it. An ASCII map would be perceivable but not understandable.
> 	• Robust: Solid against future technological developments; mostly outside the scope of this group.

This sounds great to me.

> We decided that we wanted to focus on screen readers specifically for our testing.

This, I’d like to know a little more about. Is the implication here that our testing would focus primarily on accessibility issues for players with visual impairments (and thus screen-reader users), and not any other disability (e.g. motor impairments, with attendant tests for non-mouse play)?

> We imagined a few different kinds of document:

(And I’m going to quote these only in excerpt…)

> 	• An "happy path" introduction to testing parser IF, for users of screen readers. It would document how to download and install WinFrotz, and how to use it to open "Cloak of Darkness.”


Since the start of this year we’ve modified this to developing special games in the systems we are testing (Twine 1/2, Inform, ChoiceScript), and distributing these with additional training docs (as Dan suggests) & surveys to fill out. I think as much as possible I’d like the games to be self-documenting, though, since we have total control of their content!

> 	• A "happy path" introduction to testing hypertext IF for users of screen readers.

Furkle delivered a draft of this! I plan to revisit it when we’re all in the mode of designing the test-games, in the not-too-distant future.

> 	• A list of possible/expected parser bugs to watch out for, preferably gesturing to mitigations. Issues like: mismanaging focus when submitting a command, images without alt text, ASCII art, ASCII charts and tables, non-standard punctuation (e.g. mispronounced > prompt), status bar, bold/italic/colored text, text that changes without user interaction


Our Inform test game should, of course, deliberately hit all these triggers! (And honestly we can make them a little fun… maybe with a theme of running through a literal obstacle course, AT-challenging “puzzles” popping up that the player can either surmount with their AT, or not. Maybe? Eh!)

> 	• A list of possible/expected hypertext bugs to watch out for. Issues like: mismanaging focus when clicking a link, changing text effects on click, changing text effects without clicks, images without alt text, non-standard punctuation, bold/italic/colored text, hover effects, text that changes without user interaction


And our Twine and ChoiceScript games should hit all these triggers.

> 	• A list of links to games to test.

We’ll be supplying our own games, so this isn’t on the list any more.

So it seems to me that, melding last fall’s discussion with 2018 conversation, our next set of tasks as a group involves creating those friendly Twine, Inform, and ChoiceScript games that exercise both WCAG-based and IF-specific accessibility challenges. Creating these games carries the sub-task of deciding and documenting what challenges they should and can contain.

Does that sound about right?




More information about the Accessibility-testing mailing list