Skip to main content

Posts

Showing posts from 2006

thank you Grace Hopper

You can look at wikipedia, but there are all sorts of other references.

I am a professional software tester, almost certainly because of Grace Hopper:

If she didn't invent it, she at least made popular the term "bug"
She is largely responsible for COBOL. I began my career reading COBOL. I think if I had begun with any other language, I would probably have failed.
She was one of the first to advocate adherence to standards, without which testers would not exist.
The phrase "It's easier to ask forgiveness than it is to get permission." is attributed to her, and has been a hallmark of my software testing career.
The most successful periods of my career have been with female managers. Women in IT are remarkably good managers, and without Grace Hopper's influence, there would be far fewer than than there are today.

She would have been 100 years old this month.

I am proud to note her influence.

at least look at the satellite view

Had an interesting confluence of influences this week. I sent another Better Software article to the copyeditor this week, and in it I cite (again) Harry Robinson's work. I am by no means an expert model-based tester, but understanding the basics of Harry's work is a fine tool to have in the toolbox. Harry is using Watir for testing Google Maps, as you can see if you grab the pdf and the Ruby files off of the site.

And I read that James Kim and his family may have gotten into trouble because they trusted an online map service.

I live in a pretty remote part of Colorado. A little over a year ago, my wife and I used Google Maps to navigate to another remote part of Colorado, where we were going to visit a friend. Google Maps routed us via the Mosca Pass Road.

As I mentioned to Harry, "The problem is that while Mosca Pass Road exists, it hasn't had any traffic other than hikers and horses for the last 100 years or so" Here's where Google Maps routes you tod…

Austin Workshop on Test Automation invites start

A few days ago Bret Pettichord announced the call for participation in the Austin Workshop on Test Automation. The theme this time is "Open Source Testing Frameworks".

I attended the last one of these two years ago, and it was very fine. I'm helping organize it this year.

One of the things about this conference that I think is really important is that it actively seeks apprentice/journeyman practitioners, and not just experts and beginners. This means that everyone there is actively looking for answers and questioning the state of the practice. If you think it's a good idea but you're doing something other than Ruby/Watir/Selenium, please consider attending. Diversity is good.

We've already received some really great proposals and issued some invitations to some fine people. If you're on the fence, please consider that the conference may run out of room, so the earlier you send your letter of introduction, the nicer it is for all concerned.

If you …

stuff I left out of the Fort Lewis presentation

Even before I ever heard of Edward Tufte, I had always disliked scripted presentations. My approach to speaking in public is very much like my approach to performing music used to be, back when I made my living that way: know your material thoroughly, and be prepared to interact with your audience as much as possible. It's a strangely rare approach among those who speak about software, but one or two people get it.

Martin Fowler points out one danger of this kind of presentation, being quoted, but what I find is that there are always one or two points that I really, really planned to make, but the conversation with the audience didn't go that way.

Luckily, here in the 21st century, we have blogs, so we can go back and talk about the bits we missed the first time around.

I discussed the very basic aspects of copyright and license and how they relate to open-source and proprietary software. I left out a discussion of plagiarism, which is critically important in an academic s…

linked from the soap4r wiki

One of the points I'm making to the students in my lecture tomorrow is that the most important things to contribute to open source projects are questions and answers. Documentation is also really really important to healthy open source projects. To my mind, code is the last thing most open source projects need.

Good timing that yesterday someone added to the main soap4r wiki under "Howtos" a link to my blog entry back in March dealing with "SOAP + Basic Authentication in Ruby". The link is called "Basic SOAP Authentication".

another look at soapui: load test

In our SOAP API, we have a function that I believe is intended to take care of small jobs, special cases, cleanup areas, things like that. Using this function bypasses a lot of the niftier features of our application. I have some functional tests for it, it works just fine.

Not long ago one of our customer support people asked me what kind of performance we could expect running almost a million transactions with this function in at least 30 threads. I pointed out that that was not at all what the function was designed to do. He made the excellent point that that was what thecustomer wanted to do, and we should be able to provide some information for them.

Well, crap. I am simply not set up to run that kind of a test. Neither Ruby nor Windows is very good at handling threads, and I don't have a *nix box I can appropriate immediately. (I'm working on that as we speak.) I did suggest a test that the customer could run on a test server with their own code, but I was disap…

lecture Nov. 15: "How to be a Software Expert"

The CS department at Fort Lewis College in Durango has a tradition of featuring a speaker from the community for the last lecture before the Thanksgiving break. I did the lecture two years ago on "The History of Software Testing", and I had a blast.

I was asked to do it again this year after starting the Four Corners Open Source mail list. (I hope to have a number of students (and even professors) join after the lecture.) I have a few things on my mind this time around.

For one thing, the job that I moved to Durango to take, got outsourced to Canada. Or possibly Denver. Or maybe Israel. I left before they figured out the details. But it's definitely gone.

For another thing, I spent a year working for Thoughtworks. TW itself is not gigantic (although they are spread-out) but I worked on some enormous agile projects with some really smart people from all over the world. The work was amazing, but the travel was brutal.

For another thing, I've published some pr…

mind maps for testing and other things

Better Software magazine feature article for November is about using mind maps for test cases.

I learned about mind maps at Thoughtworks, where they had been in use for test case management (and lots of other things) long before I arrived. The key advantage of mind maps, as Robert Sabourin points out in the article, is that "...misunderstandings and ambiguities are exposed and resolved in real time as you build the mind map".


That sounds too good to be true, but it really does work like that. They are also readable, easy to construct, portable, and attractive.


Here are two other ways I used mind maps that were very, very effective:


Accounting and coverage:
The project involved a re-write of an existing system. All of the (very complex) features of the existing system had to be accounted for in the new system, under the (very complex) new architecture. I used a mind map to relate existing features on the left side to new features on the right side. It was SO MUCH BETTER THAN…

"agile" "backlash"

Jonathan Kohl has a wonderful essay about the zeitgeist as it relates to A/a/gile software development.

Maybe it's because I only started making a living with software in my 30s (after being a bass player, librarian, and bartender); maybe it's because when I started, I was working on venerable systems (US 911 data-validation systems written in COBOL and C) created and maintained by smart, engaged people, but things like this don't make any sense to me:

"...companies being torn apart from within as factions rage against each other, grappling for methodology dominance..."

"...berated for not knowing the Agile lingo, and being belittled as "waterfall", and "backward" when they would ask innocent questions."

Jonathan assures me that this sort of thing goes on. Craziness.

I have to leave this business aside, though, because, while it's interesting, it doesn't affect the price of eggs in China. The meat of Jonathan's essay (for m…

FOCOS: FOur Corners Open Source

http://tech.groups.yahoo.com/group/four-corners-open-source-focos/
FOCOS is Four COrners Open Source

This is a group of software users and technology professionals based in and around the Four Corners area of the USA, where Arizona, Colorado, New Mexico, and Utah come together.

We welcome software professionals, amateurs, people from business, arts, government, academia and education, and anyone else interested in open source software and in the Four Corners area.

We encourage a wide range of discussion, but we would generally like it to revolve around these areas:

Improving our skills and broadening our experience with open source software;
Using open source software in business, academia, the arts, and government;
Improving our communities, both local and virtual, with our skill and knowledge of open source software.

#####################################################

I've been thinking about doing this for some time. Probably the original inspiration was when I got stranded in Denver…

SOAP testing the Enterprisey way

Today I found an article about SOA testing at Wells Fargo.

What a mess.

It's hard to read, but make your way to the end of the article where it describes Wells Fargo's SOA testing tools setup. Here's my un-Enterprisey answer:

Instead of SOAPScope, use soapui. Unlike proprietary SOAPScope, soapui is free and open-source, and as Mike Kelly pointed out, soapui works where SOAPScope fails. soapui is fantastic for exploring WSDLs, but I wouldn't necessarily use it as a test driver.

I use Ruby's soap4r and http-access2 libraries to drive tests. And, instead of Mercury TestDirector ($$$$$$$$$$$), I find that a combination of Ruby's test/unit features and a nice source-code repository give me all of the features that Wells Fargo gets from TestDirector. Super reliable, super simple, super understandable, low overhead and zero cost.

Then they mention "Integra Enterprise from Solstice Software" which seems to parse log files and handle HTTPS (no big deal)-- b…

Scripting Web Tests at Agile2006: great fun was had.

About 40 or so smart, engaged people attended the Scripting Web Tests (with Watir and Ruby) class. It was very interesting, this is the first time I've taught the class to students of whom the majority have a programming background. Luckily, very few had Ruby experience, so the class was still able to emphasize the exploring and experimenting that is such an important part of the presentation .

And since this was the Agile conference, the majority of the attendees were experienced pair programmers, and were eager to collaborate and share laptops and code around their tables. By the middle of exercise 2, the noise level was amazing, and the students were crazy productive. Usually we take five or six hours for this material, but this class blasted through just about everything in less than 3 hours. So many people had jumped ahead on the material that after completing exercise 3, I simply encouraged folks to try anything they wanted to try out of exercises 4, 5, and 6.

And in wh…

Another fine SOAP test tool

In a fit of coincidence and serendipity, I had a user report that the input string
è or à caused his Perl SOAP client to send crazy stuff to the server. My Ruby client also freaked out on this string, although not as badly:

XSD::ValueSpaceError: {http://www.w3.org/2001/XMLSchema}string: cannot accept 'è or à'.

but a project called "soapUI" announced a 1.6beta1 release today. I installed it, and it is not only a joy to use, but it handled the unicode input flawlessly. It's a Java app, and considering what an utter chore it is to get Java doing SOAP properly, the 1-click install was fantastic.

I'm not ready to turn in Ruby for my regression tests, but having an excellent Java SOAP client at hand is a great relief as these SOAP functions (and their users) become more sophisticated.

An interesting aspect of the soapUI project is that it seems to do load testing out of the box. I've cobbled together some nice load-profiling scripts in Ruby, but these funct…

no wonder "agile" gets a bad rap

A colleague pointed me to this blog entry from an "agile" consultant who got fired early in the project. The really telling phrase is

"The process of discovery can indeed feel open-ended as it is the very nature of discovery to explore the domain of the business opportunity in an open way. The purpose of this "openness" is to find the appropriate scope, workflow, practical boundaries, and hidden benefits in an exploratory and visual manner."

That's ridiculous. The process of discovery is to gather the minimum amount of information necessary to begin delivering working software. Nothing else.

These people wasted their client's time and money and delivered nothing. I would have fired them too.

Ruby SOAP is amazing

Back in March I grumbled about the fact that the SOAP libraries included in the standard Ruby distro on Windows are fairly broken, and to get SOAP working with Basic Authentication, one has to rummage around the internet collecting and installing a couple of libraries written by Hiroshi Nakamura.

I take it all back. No more grumbling.

I've recently been working to port about 40 API functions from one SOAP server to a different, better SOAP server. We have client code written in 6 languages besides Ruby: Java, C#, Perl, TCL, VBScript, and PHP.

Ruby was the only language that continued to function 100% with the new SOAP server and all of the tweaks we made to the new server. (PHP did pretty well, too, but I can't test in PHP, and maintenance is tricky. )

The basic issue is that there is more than one way to generate an accurate WSDL; and there is more than one way to interpret any valid WSDL. Perl's SOAP::Lite module for example will not parse our new WSDL without major …

Kevin Lawrence is still right

Back in March 2005 Kevin Lawrence wrote this great article for Better Software called "Grow Your Test Harness Naturally"

He's got a link to the PDF on his blog, luckily.

It's one of those articles that keeps being cited by various people one runs into here and there. I read it over and over until I Got It.

The basic premise is: don't write the framework first. Write some tests, then extract the common code from those tests, write more tests, abstract common code, write more tests, refactor, repeat. The refactoring generates the framework.

The thing is, I had never had enough code complex enough, running against an app stable enough, to really and truly build a framework this way.

Until now.

I have a large number things to do with Watir.

First I automated each path through the code individually. I did about 10 of these.
Each of my paths had to set up a certain set of rules, so I abstracted a method "make_rules"
Each of my paths had to set some data befo…

Internet Explorer Basic Authorization vs iFrame is just wrong and stupid

UPDATE October 2009: it seems Angrez has added support for Basic Auth to a fork and it should hit the main Watir distro RSN. News will be here. Good things come...

So I wanted to bypass the 401 Authorization Required screen that IE kicks up without resorting to actually having to manipulate the popup. So I hacked Watir's goto() method, which works great:

def goto(url)
#@ie.navigate(url)
@ie.navigate(url,nil,nil,nil,"Authorization: Basic AbdXyz46bG1ubw==\n")
wait()
sleep 0.2
return @down_load_time
end

I know I can get away with this because the user/password on the test systems are never going to change, and even if they do, it's an easy script change.

The trouble is that the main page has an iFrame, and for some reason, when called from a script, stupid IE won't use Basic Authorization credentials to request the iFrame.

So now I'm getting the authorization popup when IE tries to load the iFrame contents. …

Two things about performance testing

A couple of things to stick in your back pocket:

1) Think geometrically, not arithmetically. Think about orders of magnitude.

10 is the same as 20 is the same as 50.
2000 is the same as 5000 is the same as 7000.
100,000 is the same as 200,000 is the same as 300,000.

The interesting information comes between the orders of magnitude: is 10,000 faster or slower than 10? is 1000 faster or slower than 100,000?

2) Think about precision and accuracy.

It is precise to say "10 records per second on this OS on this hardware on this network".
It is accurate to say "a rate of between 1 and 10 records per second on any OS on any reasonable hardware".
It is accurate to say "10 records per second running locally; 1 record per second over the network; 10 seconds per record over VPN".

Accuracy is critical in the general case; precision is critical in the specific case.

SOAP + Basic Authentication in Ruby

Do you need to do SOAP in Ruby?

First of all, install the latest version of Ruby you can stand. (I'm using this one. What the heck. I'm giving the laptop back in a few hours.)

That's a fine thing, but you can't do Basic Authentication yet. You need to install a net/http that understands Basic Authentication.

Ready to go? Not quite. The version of SOAP4R you have doesn't handle arguments to web services correctly. You need to install the latest SOAP4R to get your arguments handled right.

Now you're ready. You might get an extraneous error message "ignored attr: {}mixed" but it probably won't affect your ability to do the job.

Many many thanks to Emil Marceta for walking me through the process.

Two things:
1) Perl's SOAP::Lite does this out of the box.
2) The Pickaxe dismisses SOAP pretty perfunctorily, but the beauty of SOAP is that it is (should be) very very easy for the client, using existing libraries.

What would it take to put this in the…

Unorthodox testing tools in a hostile environment

Some time ago, I built an "Enterprise" disk imaging system, the equivalent of Symantec's Ghost product, from Open Source components, hosted on FreeBSD. In the process I discovered the FreeBSD ports collection, an astoundingly huge set of applications managed as part of FreeBSD itself. I used a whole raft of stuff from the ports collection as test tools in an all-Windows environment, everything from collaboration (Twiki) to network analysis (ntop).

I was doing a lot of install/smoke testing, and the Frisbee disk imaging system was an important part of that work. I published an interview with Mike Hibler, Frisbee's lead developer, in the March 2005 issue of Better Software magazine.

I discussed the wider issues of hosting test tools on FreeBSD in a paper for the 2005 Pacific Northwest Software Quality Conference, and now that paper has been adapted as a FreeBSD White Paper.

Dru Lavigne, a noted FreeBSD evangelist and Open Source advocate not only helped me through th…

Scripting fun

I have a new-ish laptop and I haven't bothered to install unixtools or cygwin on it. (I had a bad experience with Cygwin some time ago).

I needed to grep around in the C# source code directory for stuff. Windows Search didn't do the Right Thing, and TextPad file search gave me memory errors. So I wrote a little script in Ruby to root around in the source code for stuff. Maybe it'll be useful for someone (sorry about the lack of indentation, that's left as an exercise for the reader) :

require "find"
Find.find("../interesting/directory") do |f|
# Find.prune if f =~ /Design/
# Find.prune if f =~ /UnitTest/

if f =~ /cs/
x = File.open(f)
while line = x.gets
if line =~ /stuffImLookingFor/i
puts f
puts line
end #if
end #while
end #if

end #do

Testing Web Services

There was a recent thread on the agile-testing mail list about testing Web Services. I ended up having an interesting private conversation with the original poster, during which I said:

I would immediately ask you "how are these services implemented?" Assume that your web services are implemented as pure XML-over-HTTP. I'm building a system-test framework in Ruby right now that understands how to talk to and how to listen to these XML-over-HTTP interfaces. I could just as well be using SOAP or REST, but in this case I don't have to.


> > Any "Gotchas" I need to be aware of ? Special strategies / approaches /

I think it boils down to two questions: the choice of tools, and the
choice of approach. Almost any set of tools can work with either approach. Do you know Brian Marick's distinction between "business-facing" tests and "technology-facing" tests?

The choice of approach I identified is to test with unit-like tests
that va…

Teaching the Scripting For Testers class

SQE (the stickyminds people) have announced the schedule for the STAREast conference, and I'll be teaching the Scripting for Testers tutorial on May 16, almost certainly with my colleague Dave Hoover.

I'm really looking forward to working with Dave. Not only is he a better programmer than me, but we share an interest in teaching, and in working with (and in being) beginners on the way to being experts.

Brian Marick and Bret Pettichord started the class in 2003, and this tutorial was a major contributor to the existence of the Watir test framework, to which I've contributed a bit of work. This tutorial will be the first since 2004 not taught by Bret (and the first ever not taught by one of the two of them, as far as any of us know).

It's a good class, and some great people have been a part of teaching it in the past. I'm hoping that the class will take on a life of its own from now on. Enough people have taught it, and enough people have taken it, that I think the…

Static tests and dynamic tests

A typical test expects a particular set of data to exist, right?

Say I have some software that accepts a chunk of XML and does a search for people in a database by last name and first name. It might accept

<requestroot xmlns="http://foo">
<lastname xmlns="" > MCMAHON < /lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>


and it might return

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

so I could write a test that does something like


SEND

<requestroot xmlns="http://foo">
<lastname xmlns="">MCMAHON</lastname>
<firstname xmlns="">CHRIS</firstname>
</requestroot>

RECEIVE AND PARSE

<responseroot xmlns="http://foo">
<user xmlns="">CHRIS MCMAHON</user>
</responseroot>

That is a static test, and that's how unit tests work. More than likely…
A software tester should always have the means and opportunity to read the source code for the software being tested. I've had a number of jobs as a tester, and on some them, getting to read the source code was quite an ordeal.

I'm currently working on an integration project. We had a new Business Analyst join the team, and on his first week, one of the developers gave him a guided tour of the most relevant parts of the source code for one of the applications we are integrating. I tagged along. I had surfed some of this code, but I am not a Java expert by any means, and having the design and implementation explained by one of the developers that wrote the code was a great help in understanding how and why to test the integration of that application.

Imagine that. A BA and a tester learning about the code from a happy developer. I like my job.