Monday, September 25, 2017

Watir is What You Use Instead When Local Conditions Make Automated Browser Testing Otherwise Difficult.

I spent last weekend in Toronto talking to Titus Fortner, Jeff "Cheezy" Morgan, Bret Pettichord, and a number of other experts involved with the Watir project. There are a few things you should know:

The primary audience and target user group for Watir is people who use programming languages other than Ruby, and also people who do little or no programming at all. Let's say that again:

The most important audience for Watir is not Ruby programmers 

Let's talk about "local conditions":

it may be that the language in which you work does not support Selenium

I have been involved with Watir since the very beginning, but I started using modern Watir with the Wikimedia Foundation to test Wikipedia software. The main language of Wikipedia is PHP, in which Selenium is not fully supported, and in which automated testing in general is difficult. Watir/Ruby was a great choice to do browser testing.  At the time we started the project, there were no selenium bindings for Javascript. Now that Selenium is fully supported in Node.js/, I understand that WMF is porting the Watir-based tests in 20+  repositories to Node/ It is very exciting.

Today I use Watir at, the philanthropic arm of the much larger company Salesforce has its own proprietary programming language APEX, which does not have Selenium bindings and never will. Watir is the best choice to do automated browser testing in this environment.

it may be that even if your language supports Selenium, automated browser testing is still really hard

Selenium is fully supported in C#, but many Windows organizations do not use C#. Even when they do, the amount of scaffolding required for robust browser testing can be daunting. Creating an assertion framework, a logging structure, a Page Object  model, a test runner, all of these take significant investment. The Watir project allows you to short-circuit all that work.

Java and Python are in a similar situation. Watir and Ruby provide an off-the-shelf answer that is remarkably  well-documented, with a proven history of excellent design and implementation decisions as well as an active and polite user community. (Watir as a concept is actually older than Selenium!)

it may be that the browser test automation staff do not have access to the feature code

Certainly this is an anti-pattern, but it is sometimes the case that the people who need to do browser test automation are not in regular contact with the people creating the code that needs testing.

Two Paths for Watir

Right now today there are two approaches to using Watir, each with their own design philosophy, each with their own
(overlapping) sets of tools.

Simple, straightforward, and easy; but powerful

The first approach is exemplified by the project at watir_install, that I will call Team Titus. Team Titus wants to provide a single installation that yields an instantly usable Watir testing instance with no configuration, with examples in place for  anyone to follow. Regardless of what programming language you find most comfortable, Team Titus wants to provide the most  readable, understandable framework possible.

The second approach is exemplified by the project at page-object, that I will call Team Cheezy. Team Cheezy wants to provide the most powerful application of Watir for browser testing possible. Team Cheezy gives you a framework that needs to be tweaked a little to work, and has more layers requiring understanding. To understand the internals for Team Cheezy you need to know something about how Ruby works.

You should know that the barriers to entry for Team Cheezy were already low; Team Titus wants to make them even lower, but at the cost of some power in the framework.

Note that these approaches are not incompatible. They do not compete, they cooperate. Bits of each framework can be switched in and out.

Me, I started with Team Cheezy about six years ago and I'm sticking with it. I need that power. However, my heart is with Team Titus, and I intend to contribute as that project grows.

Choosing between Team Titus and Team Cheezy

Team Titus (watir_install) Team Cheezy (page-object)
Don't care about Cucumber Want Cucumber
Need easy install Willing to tweak some directories and settings
Only encounter standard HTML and DOM Needs to address non-standard page elements
Probably don't need to scale across multiple repos Needs to work across multiple repos
Want simple internal code to analyze Willing to learn some Ruby magic for internals

I went to Toronto to talk these people because I saw changes happening and I need to know the context of those changes. I
learned what I needed to know:

  • Everyone agrees that the primary audience and most important users for Watir are people who need to do automated browser testing BUT who are not primarily Ruby programmers.
  • We need to provide a framework that works for everyone upon installing it.
  • We need to provide power to people who need power.
  • We need to make it easy to start, but also easy to improve.

Appendix: Why not target Ruby programmers?

For historical reasons, the Ruby web development community mostly uses Capybara and not Watir.  We might argue which is
better, but it would be pointless. Watir has always been intended for use by everyone, not just Ruby programmers, and we
carry on that design philosophy.

Monday, February 20, 2017

Use "Golden Image" to test Big Ball Of Mud software systems

So I had a brief conversation on Twitter with Noah Sussman about testing a software system designed as a "Big Ball Of Mud" (BBOM).

We could talk about the technical definition of BBOM, but in practical terms a BBOM is a system where we understand and expect that changing one part of the system is likely to cause unknown and unexpected results in other, unrelated parts of the system. Such systems are notoriously difficult to test, but I have tested them long ago in my career, and I was surprised that Noah hadn't encountered this approach of using a "Golden Image" to accomplish that.

Let's assume that we're creating an automated system here. Every part of the procedure I describe can be automated.

First you need some tests. And you'll need a test environment. BBOM systems come in many different flavors, so I won't specify a test environment too closely. It might be a clone of the production system, or a version of prod with fewer data.  It might be something different than that.

Then you need to be able to make a more-or-less exact copy of your test environment. This may mean putting your system on a VM or a Docker image, or it may be a matter of simply copying files. However you accomplish it, you need to be able to make faithful "Golden Image" copies of your test environment at a particular point in time.

Now you are ready to do some serious testing of a BBOM system using Golden Images:

Step One: Your test environment right now is your Golden Image. Make a copy of your Golden Image.

Step Two: Install the software to be tested on the copy of your Golden Image. Run your tests. If your tests pass, deploy the changes to production. Check to make sure that you don't have to roll back any of the production changes. If your tests fail or if your changes to production get rolled back, go back to Step One.

Step Three: the copy of your first Golden Image with the successful changes is your new Golden Image. You may or may not want to discard the now obsolete original Golden Image, see Step Five below.

Step Four: Add more tests for the system. Repeat the procedure at Step One.

Step Five (optional) You may want to be able to compare aspects of a current Golden Image test environment with previous versions of the Golden Image. Differences in things like test output behavior, file sizes, etc. may be useful information in your testing practice.

Monday, February 13, 2017

Open Letter about Agile Testing Days cancelling US conference

I sent the following by email contact pages to Senator John McCain, Senator Jeff Flake, and Representative Martha McSally of Arizona in regard to Agile Testing Days cancelling their US conference on 13 February.

Agile Testing Days is a top-tier tech conference about software testing and Quality Assurance in Europe. They had planned their first conference in the USA to be held in Boston MA, with a speaker lineup from around the world. They cancelled the entire conference on 13 February because of the "current political situation" in the USA. Here is their statement:

Although I was not scheduled to attend or to speak at this particular conference, it is conferences such as Agile Testing Days where the best ideas in my field are presented, and it is from conferences such as Agile Testing Days that many of my peers get those ideas, and I rely on conversations from those who do speak and attend in order to stay current in my field.

As a resident of Arizona, cancelling such conferences affects me directly. I have enough expertise and skill to live anywhere I choose. I choose to live in Arizona, but my work absolutely depends on the free flow of people and information across national and state borders.

It is shameful that such a prestigious and respected multi-national software organization finds it necessary to cancel their first ever conference in the USA because of the outrageous policies of the current administration. I urge you to take measures to make organizations such as Agile Testing Days and their attendees and speakers feel safe and welcome, as they should be.

Chris McMahon
Senior Member of Technical Staff, Quality Assurance
Tucson, AZ

Wednesday, August 03, 2016

Sanction Keith Klain

I have asked the Association for Software Testing to make a statement regarding the relationship of AST with their former Board officer Keith Klain in regard to the lawsuit for fraud filed against Klain by his former employer Doran Jones.

The lawsuit alleges that Klain did a number of reprehensible things. What should concern the AST and the software testing community in particular is that Klain is alleged to have been a party to sabotaging and undermining the training in software testing given to disadvantaged people by Per Scholas in New York City.

Klain filed a "LETTER addressed to Judge Analisa Torres from A. Goldenberg dated July 20, 2016 re: Request for a Pre-Motion Conference Regarding Anticipated Motion to Dismiss".  Of concern to the software testing community is that this letter in no way disputes or denies Klain's behavior alleged in the law suit; this letter merely attempts to make the case that Klain's abominable behavior should not meet the letter of the statute itself for actual fraud. (1)

Then Doran Jones answered that letter with one of its own "FIRST LETTER addressed to Judge Analisa Torres from JASON H. KISLIN dated August 1, 2016 re: Plaintiff's Response to Defendant Klain's Request for a Pre-Motion Conference."  (2), which concludes in part:

"In addition, should the Court dismiss Doran Jones' CFAA claim, its remaining state law claims -- for breach of contract, breach of the duty of good faith and fair dealing, breach of fiduciary duty, tortious interference with prospective economic advantage, tortious interference with contractual relations, misappropriation of trade secrets, fraudulent inducement, and declaratory judgment -- will be properly before this Court because they do not require the Court to rule on novel or unsettled issues of state law."

This is reprehensible at best, immoral at worst. Not only should the AST sanction Keith Klain, but TechWell, SoftwareTestPro, and the conference organizations at which Klain presents should seriously reconsider their current and past association with Klain, and anyone else with whom he has any ties should consider the value of that association as well.

(1) Anyone may get direct access to court documents for this case for free or for a nominal fee by creating an account at PACER  The case number is Case 1:16-cv-02843-AT . Notices of actions are available publicly for free.
(2) This PDF does not render in my browser, but seems OK after downloading. For an original copy see (1) above.

Thursday, July 14, 2016

Open letter to "CDT Test Automation" reviewers


Tim Western
Alan Page
Keith Klain
Ben Simo
Paul Holland
Alan Richardson
Christin Wiedemann
Albert Gareev
Noah Sussman
Joseph Quaratella

Apropos of my criticism of "Context Driven Approach to Automation in Testing" (I reviewed version 1.04), I ask you to join me in condemning publicly both the tone and the substance of that paper.

If you do support the paper, I ask you to do so publicly.

And regardless of your view, I request that you ask the authors of the paper bearing your names to remove that paper from public view as well as to remove the copy that Keith Klain hosts here.  For the reasons I pointed out, this paper is an impediment to reasonable discussion and it has no place in the modern discourse about test automation.

Wednesday, July 06, 2016

Open letter to the Association for Software Testing

To the Association for Software Testing:

Considering the discussion in the software testing community with regard to my blog post "Test is a Ghetto", I ask the Board of the AST  to release a statement regarding the relationship of the AST with Keith Klain and Per Scholas, particularly in regard to the lawsuit for fraud filed by Doran Jones (PDF download link) .

The AST has a Code of Ethics  and I also ask the AST Board to release a public statement on whether the AST would consider creating an Ethics Committee similar to, or as a part of the recently created Committee on Standards and Professional Practices.

The yearly election for the Board of the AST happens in just a few weeks, and I hope that the candidates for the Board and the voting members of the Association for Software Testing will consider these requests with the gravity they deserve.

Thursday, June 30, 2016

Test is a Ghetto

If you read software testing news aimed at the general public, you might be of the opinion that software testing is done by, and *properly* done by:

The key of course is "minimal training". There is a class of software testers who have minimal programming skills, or system administration skills, or database skills, or any technical computer skills at all. These testers do honorable work and can be valuable members of a software development team. They have been my colleagues; I have helped hire them; and I have trained them in test automation. And I still do that sort of work myself sometimes, although others are better at it than I am.

However, their lack of technical skills mean that they tend to have lower status, lower income, and are often considered fungible, easily swapped out as economy dictates, or replaceable by automation. At least at the start of their careers, these testers can be caught in a vicious circle of not having the technical skills or critical abilities to advance their careers, while also lacking enough understanding of modern software development to see a way to improve their skills. Many stay in this circle indefinitely, where a whole mythology of the value of codified ignorance evolves. This is the software testing ghetto.

Ghettos exclude their occupants from the general discourse, but it is also true that ghettos are exploited by agents of the greater culture. Software testers, particularly junior-level software testers, especially such testers that are because of their circumstances ignorant of modern software development discourse, are especially prone to exploitation. If they are represented by an agent, then someone is paid to recruit them and train them. Someone is paid to negotiate their contracts. Someone sells their employers the tools they have been trained to use.

I have not found out anything about the training that autistic people, Aboriginals, or Malaysians receive; the original draft of this essay was to have provided a detailed analysis of the training provided by Doran Jones, Per Scholas , and Keith Klain in New York City. There is a wealth of material available, and I urge you to search online yourself.

However, in the course of doing that research, I discovered that Doran Jones has sued Keith Klain and Per Scholas for fraud over that software testing training program. The entire document (PDF download link)  is a fascinating look at the inner workings of those agencies that sell software testing services. The part of the lawsuit relevant to the software testing training begins at paragraph 89, for those who wish to read it. Also of interest given the stated list of Per Scholas partners is paragraph 37.

Under the circumstances, and given the nature of the claims against Klain and Per Scholas, I think it is not appropriate for me to publish my comments, but I would like to point out one fact not contained in the lawsuit documentation: the Association for Software Testing committed their resources  to the Per Scholas training program in NYC while Keith Klain was a member of the AST Board of Directors. I expect that the current and recent officers of the AST are extremely interested in the outcome of this lawsuit.

Klain says this about software testers: "Software testing is a strange business. It’s commoditized (sic), devalued, misunderstood, and goes through cycles of being chopped, changed, and lives at the front lines of imminent takeover by our robot overlords. Why anyone would want to be a professional software tester is even harder to understand." Read the whole thing

Interestingly, Klain and the people involved in this Per Scholas project are also the most vocal opponents of software testing certification, sometimes with questionable approaches to gutting certification efforts.

It makes sense that these agents of minimally trained software testers would oppose certification. A global, generally-accepted, inexpensive certification in software testing would allow those entry-level software testers with limited knowledge of modern software development culture to more easily be their own agents in that culture. The market for this sort of exploitation might shrink considerably. In hindsight, I wish I had said this explicitly when I tackled the topic in 2010.  As your career matures, your CV becomes more important than your certifications, but getting certified early on is a perfectly reasonable career move.

As Marlena Compton said in her 2015 essay "A Tableflip Guide: Transitioning from Tester to Developer"  "If you go to a testing conference you’ll find people talking about how you can stay in testing forever and how it is a great career path. I’ve noticed that, often, the testers who shout the loudest about staying in testing forever have carved out their own place in the power structure of the software testing industry." I urge you to read the whole essay.

I'll suggest further that those testers shouting the loudest may also depend on the minimally skilled testing ghetto for their livelihood, and may not have your best interests in mind.

If you as a software tester

are happy with your career path and prospects for growth
are happy with the skills you have and the prospects to develop them further
are respected by everyone on your development team and are treated as a peer
represent your own interests to your employer with good faith on both sides
have technical training available to you
understand technical aspects of software development other than testing

then this essay probably does not describe you. If these things are not true for you, you may be in the software testing ghetto.