[Accessibility-testing] 2018 goals review

[Dan Fabulich]

Hi, sorry I've been quiet on this thread.

> On Jan 30, 2018, at 9:19 AM, Jason McIntosh <jmac at jmac.org> wrote:
> 
> Hullo team,
> 
> I'd like to begin the year by reviewing and confirming this project's goals. After that, I'd like to lay out an overall schedule, and then start defining and assigning our next tasks. (I realize that we're due to carry some tasks over from last year, and do plan to review those with y'all when we get to it.)
> 
> To start, I'd ask for y'all's confirmation on some assertions about this project I feel comfortable making. These mix my reading of the committee charter, my review of this mailing list's 2016-2017 content, and my own understanding:
> 
> • The project's mission remains as stated in its committee's charter (http://iftechfoundation.org/documents/access-committee.pdf).
> 
>    • We will organize a "Testathon" event, which is understood to mean crowd-sourced testing of given software for accessibility compliance.
> 
>    • Testers can be anyone with user-level access to the software being tested. They needn't be assistive technology users, or even people with disabilities -- though of course we'll welcome and encourage participation from AT users & PwDs.
> 
>    • Generally speaking, we'll generate a big matrix that splits up every piece of software being tested into small-as-possible discrete actions that anyone with access to the software can attempt, each with pass/fail results.
> 
>    • We will publish documentation about the methods we used to organize said Testathon, as well as the results of the crowd-sourced tests and the subsquent conclusions of the accessibility and platform experts on the team.
> 
>    • This project will conclude with the release of all the above.
> 
> • This project can and should deliver its mission-declared publications by the end of calendar year 2018.
> 
> Is all the above correct?


I want to raise a point that I've raised a few times on the mailing list in the past, and I want to re-raise it each time the team comes under new management, but no more often than that. 😉

As I think you've probably seen in the mailing list archives, I've been skeptical of the "Testathon" goal from the start. In particular, I remain very afraid that we'll drag ourselves over the finish line to run a Testathon and file a number of bugs, only to let the bugs sit unfixed for a long time (quite possibly years), wasting the effort.

As a result, I feel that the true value of the project is the documentation we generate on how to test for accessibility issues in general. This is the documentation that developers will use in future years to repeat the testing, reproduce bugs, and verify their fixes.

I especially wince at the idea of covering a "big matrix." We should at least start with a small matrix.

We've really struggled to get individuals to do real work for this project. I'm as guilty as anyone; in September, I thought I'd have a lot of time in November of 2017, but life ate up all my volunteer time and then some.

The problem isn't that we can't get organized. The problem is that nobody's sitting down to do work. We don't have a big list of tasks to put into Trello because we aren't doing any tasks. 😞

Instead of building a matrix and finding volunteers to cover the matrix, we should start by figuring out our volunteer time budget: how much time is anyone in this group dedicated to doing this week?

If history is any guide, we're going to have approximately one dedicated person at a time, who can spare about one work-day of time at approximately monthly intervals.

On that budget, I claim that we should just have one dedicated person do the testing on a very small matrix. If it were still September, I'd say that this person should be me (and it's possible that this person could still be me, but probably not this month or next).

Overall, in light of our non-accomplishment to date, I think we should not assume that there will be a big increase in volunteer output, and we should plan accordingly.

-Dan