No excuses...

This post was originally intended to catalog a few tools to aide in viewing the internals of a system with an added chastisement to those who complain about not having enough information. In fact, I never got to CLR profiler or the array of performance counters made available in .NET. Instead, it turned into a brain dump of the ways in which I ask questions and seek out their answers. The journey took me somewhere I didn't expect. You'll see what I mean...

I've been reading a lot of posts lately talking about how testers cannot test lower level functionality, structures, etc. because they don't have access to the code.

If you're working with a .NET application, my response is "Pish-posh".

Download Reflector. Nefariously point it in the direction of those impenetrable .NET assemblies. Behold the fruit of your efforts:



This is an actual disassembly of a program I was writing using the WebAii automation framework. Though my example is incomplete, you can still see what's going on under the hood.

"But Zach", you protest, "I can't read all that code-y stuff! What kind of bugs can I find doing that?"

Well, using this tool, I can think of one possible bug that doesn't require intimate knowledge of C#, .NET, CLR, or IL disassembly techniques. How about the fact that you can go out and download a tool that can expose your stakeholder's intellectual property after a few clicks of a button? If the company is concerned about protecting its secrets, they should be concerned about the relative ease with which one could reverse-engineer their system.

There are also plugins available that allow you to view differences between two versions of the same assembly. If a bug-fix is delivered, you may be able to ascertain where that change was made. What do you find there?

"But our application is web-based! I can't just look at assemblies on my local machine!" More pish-posh! Your browser has a "view source" function. Use it to peer in behind the curtain. Want some more power than that? Download FireBug. More power than that? Download Wireshark.

But tools are not the answer to the question. In fact, they are distracting if you don't understand the question you're asking. Put another way, I'm not persuaded by the argument, "I don't know what all this code stuff is all about. I just mash buttons." It is time to take a lesson from the economy and learn something new. The internet has now made this a lot easier than it was when I was trying to learn Raster Interrupts on my Commodore 64.

Your Mission:
Make a Windows application containing one button. When you click this button, a dialog box appears that reads, "Hello, World!". I won't tell you HOW to do this, that is YOUR job. But a few points are hidden in this exercise.

Learn Developer Empathy.
Many communities like to push the idea that testing and development should be at odds. I think this is a dangerous idea. To me, developer empathy is NOT the same thing as BUG empathy because developers are not the problem. Sure, they may have coded it in wrong, but you're not going to be able to catch all the bugs in the system ANYWAY. So none of us are doing are jobs perfectly, cool?

As the saying goes, you catch more flies (which are bugs) with Honey than with Vinegar. The 'honey' here is your willingness to learn about life on their side of the table. I guarantee that if you approach a group of developers with the problems you encounter while doing this exercise, you'll receive no end of advice. You can also use your experience to frame questions about the system under test: "I notice this alert comes up in the browser when I enter in invalid information. How is that different from my little application?" These are the kinds of questions I asked developers. Just remember to ask when it is convenient for them. I always walk away with more knowledge and more questions. Which leads me to...

Mind Map your journey.
Create a mind map. In the middle, enter in the task of creating that Windows application. Before you start trying to figure out coding, try to articulate your confusion and frustration. Create nodes that encapsulate your questions: "What is a Windows application?", "Why do I hear so much about, 'Hello World'?" As you find answers, add them as nodes of the question itself. But you're not done because that answer should have created several new questions for you. Add them and repeat. Download this mind mapping program if you must.

As testers, ours is not only a journey of questioning things, but in finding out the answers to those questions. If someone plops down a huge system for you to test, how are you going to tackle it? The question is always antecedent to the approach. So if you don't know, find out and track the way(s) in which you found out. Like watering holes in the desert, there will be places you perennially come back to for answers. Where are they? You won't find them unless you go looking for them.

Brush up on your Google-Fu
Learn to ask questions in different ways, including how to ask them in a Google query. Remove extraneous words. Group the related terms together to eliminate false positive results. Reduce the results to a certain domain, a certain file type, within a certain period of time. This ability can greatly improve the signal-to-noise ratio.

Don't brush up on your Google-Fu
Send that query out without all the constraints mentioned above. Misspell a word. Ungroup terms. Expand the time frame. Select a result from page 10, not page 1. Select the LAST result. You will be surprised by how much you uncover in the journey. Bob Ross called them happy little accidents. I cannot tell you how many discoveries I made on the way to answer a completely DIFFERENT question. In fact, most of the downloads on this page are result of another train of thought.

Now, you've downloaded Microsoft C# 2008 Express and written your program. Go ahead and set up a new, small project for yourself and see it through to completion. Use TDD,BDD, NUnit or MbUnit. (What do all those terms mean? You don't seriously expect me to answer it for you, do you?) You will have gained more knowledge of how to view the internals of the system. These internals are not only the constituent bits that comprise the application, but the human processes around their creation. Knowing these processes can guide your questions later on. No, you're not developer level but you are CONVERSANT. Being conversant means being able to have a conversation, and conversations lead to questions which lead to discoveries which leads to...

Have fun playing with the tools. Just don't start/stop there.

The Gospel of Record and Playback

Microsoft unveiled its UI Automation capabilities for VSTS 2010 at Mix09. If you are interested, watch this video. Watching it was painful for me.

Some Highly Opinionated Observations
I felt truly bad for Brian Keller (Brian, if you're ever reading this, you did a great job). He was stuck practically sweeping up the ticker-tape from previous presentations. Even he found it ironic that his presentation was stuck at the end of the conference...like so many testing efforts. And like so many testers, he pleaded for feedback from a roomful of developers to help determine if there was significant interest in future talks about testing. Oh, and the technology wasn't helping either.

In 17 minutes, I saw a perfect example of the inherent difficulty with record-and-playback, even though it was capturing a lot of deep information. A functional test was recorded and the first failure was detected upon playback, only it was triggered by errors made while editing the generated code.

Recordin' Payback
Now, don't get me wrong. I am VERY impressed by what Microsoft is putting into the hands of the team - but at what premium? This is where I would like to insert my favorite argument, "Why do I need to pay $X,XXX.YY/seat for a tool that gives me the honor of pushing the button our team was tasked with creating?" How expensive is it to do any/all of the following during development:

- Define unique control IDs
- Define Active Accessibility parameters (y'know...those weird parameter fields you NEVER fill out when adding controls to GUIs in Visual Studio)
- Override control's WndProc methods with custom logic to persist user interactions to log files that can later be used for playback
- Keep logic OUT of the forms so that your application can run headlessly
- Implement Michael Hunter's Automation Stack
- Charge a junior test engineer with the task of learning how Win32 Window APIs, MsAA, Java Active Accessibility Bridge, .NET UiAutomation namespaces, etc. really works.
- Make testability hooks critical features in the application
- Make log files critical features in the application
- Report Source Code changes to the entire team

Does it take longer to implement these things? Maybe. It will definitely take longer to impress any management that merely hovers over the process with all the disinterest one could expect from a super model on a blind date with <insert NASA scientist here>. But I can say this much: if the department I'm in is considering necessary economic cutbacks, I'm glad I won't be competing with maintenance fees for COTS tool X.

My opinion is that these are more valuable for the organization because they invest in the individual(s) and their knowledge, not dumping money into 3rd party solutions to solve what are perceived as technological issues. To me, this is the same as if my parents, after being diagnosed with Diabetes, determined that the problem wasn't their diet but was actually their cooking. And then proceeded to "solve" this by eating the same foods at restaurants because...heck...they know how to cook. Sure, it costs more, but look how much healthier we are!

Hmmm. I guess it depends on the food/restaurant.

Benediction
Finally, on a more personal note: I long for the day when the ridiculous moniker of "evangelist" is purged from the tech nomenclature. I understand that it can bear secular meaning, but its use is imbued with religious connotation. An evangelist is one whose very existence bespeaks of a salvific message. As a tester and a man of faith, I find the idea that a technological solution could save us is...well...laughable; and the very reason I have a job.

Testing Tool: FileHamster

FileHamster is awesome. If it hasn't already arrived at your city - pillaging it with paws of gargantuan awesome-ness (paw-someness?) - then consider yourself both deprived AND unfortunate. Download it from here. I've been using this tool for about 1.5 years now and always come back to it.



In a nutshell, Filehamster is a glorified directory monitor that acts as a kind of localized source control. The developers originally intended it to allow creative types to work without the fear of being unable to undo their Picasso-esque changes. Anyone who has made modifications to a document, then closed the application only to realize that was in fact NOT the modification you wanted, can attest to this problem. Here's how it works from a 3048m view:

- Create a new directory/file watch
- Operate your application, saving changes will-nilly
- Observe that FileHamster captures each successive save

"No bug whoop!", you exclaim. "The .Net framework has a FileSystemWatcher class that lets me do the same thing!" That is true. Who hasn't written this kind of program before using that very class. However the ease of use, visibility of changes in real-time, and the diff plugin are benefits that keep this high on my desert island list of exploratory testing tools.

An example would be good here:
If I have a fat client application to test, one of the first things I do is pop open FileMon to observe its file system traffic for log files, artifacts, etc. Once I determine the directories where these files are written to, I create a new recursive directory watch in FileHamster. Even better, I can filter the types of files that it captures, thus reducing the potential for data overload. I then proceed to explore the app some more. Meanwhile, FileHamster is dutifully running in the background capturing all the file system changes into discrete time-stamped files and alerting me when it does so.

After a session, I can come back and observe the captured files. FileHamster honors any file associations; it will open the file in its associated application, e.g. TXT in notepad, MDB in Access, etc. This ability to observe a session's change history is valuable in-and-of itself. However, the diff plugin allows me to specify a comparison application as well. I can select two of the captured files and launch the comparison application via a context menu. Now I can see what changed in the document while I was testing away.

Experience Report...nnnNNNOW!:
I've used this approach while testing some financial applications that persisted actuarial debug values in a kind of CSV file. I would run a plain vanilla case, then run an alternate solve that should have given me the exact same answer. Both tests produced the CSV file. When I encountered a problem, I simply compared the known good (did I mention you can assign comments to the captured versions in this thing?) to the variant version and observed precisely where the diff was introduced. I don't need to be an ASA to tell you that a different max allowable loan value in year X could have significant ramifications on a Universal Life policy and its later-year cash values.

The only problem I've encountered is that I'm only able to specify one diff application. I've worked around this by creating a stub utility that maps all my diff applications by file extension. That way, my .xls files are compared in my custom Excel file utility, .PDF files are compared in another, etc. I may post my comparison stub script later, but it is simple enough to script in a language of your choice.

Download FileHamster and check it out. It runs on Windows and requires .NET 2.0. I want to say that I've also been able to run it off a thumb drive, so portability may be a benefit. Let me know what you think.

Listen Before You Send

In response to the proliferation of useless software, I created a utility to read large chunks of selected text from the clipboard. I posted it here.

I now use it in conjunction with spell-check to "sound-check" my text. It isn't perfect, but it has caught many "bugs" in my correspondences. I even use it to check text used in applications under test.