[Accessibility-testing] "Gut check"

[Jason McIntosh]

Quoting from the skeletal schedule that we agreed upon last month:

> 3. (Sep 10) Gut check: Should we continue with wide testing, or should we present what we have (including testing and observations provided from our own team’s accessibility experts) and call it good?
> 
> If we decide to call it a day, then we take an abbreviated path that sidesteps the remainder of this schedule. We’ll start preparing the final public deliverables early, and otherwise wrap up.
> 
> If we decide to carry on with wider-scale testing, then we should start preparing the surveys as well as any other preparatory material we plan to distribute to testers.

I’d like to gauge the group’s thoughts on this question, enough to answer it definitively within the first half of September.

We discussed this a little bit a couple of meetings ago, so I’ll recap here. My prompt to ask this was a question that Dan Fabulich raised on this list two years ago (August 29, 2016), in response to Zarf sharing the brand-new committee’s charter:

> Granting that the goal is to test and document the state of IF software, my only remaining question here is whether the name "Testathon" (which I understand to mean, "hackathon for testers") assumes in advance that:
> 
> 1) a large crowd of testers is the right way to begin, as opposed to, say, having just me and/or Deborah run from Marathon to Athens by ourselves
> 
> 2) testing a large matrix of screen readers, interpreters, IF systems, operating systems, and games is immediately desired, as opposed to testing a smaller number of platforms, finding the critical bugs, and then broadening the scope of testing once those bugs are fixed

This question — which the list declined to immediately answer, according to the archives — surprised me at the time, but it’s stuck with me ever since. Since I took up the chair at the start of the year, I’ve let it set part of the tone for the direction that we have subsequently taken, favoring pragmatic delivery over thorough perfection.

So, I’d like to definitively answer what Dan had marked as the first part of this question: should we follow through on our initial, 2016-vintage assumptions to organize a wide testathon, or should we make the actual testing a much more modest affair, limiting the testing population to the membership of this committee — specifically, the with-disabilities and accessibility-expert sub-membership? In the latter scenario, I also imagine the option of bringing in a handful of hand-picked volunteer testers to fill in disability coverage gaps, but in any event it would skip entirely the public recruitment drive that we’d envisioned for years.

I’d see choosing the more modest route as a trade off: the final stages of this project would end up much easier to organize, since we’d have far fewer volunteers to manage, and we wouldn’t have to set up systems to communicate with the whole group, assist them with setup and testing, and then manage all their subsequent data. But in doing so, we’d be knowingly sacrificing a far wider range of tester viewpoints. I suppose it comes down to asking what we’d accept as a win condition: would it be enough to deliver a documented, repeatable process, even if the scope and scale of our performed iteration ends up a lot smaller than we had originally hoped?