This post is edited with thanks to and appreciation for Reed Haynes, who was on our QA team for several years but has recently moved on to another adventure. I've cobbled the following article together with immense help from a draft he left me and some of his internal documents. His voice is very much alive in this article.
QA's scope of duties is wide and involves an equal dose of structure and chaos. For this article we will be focusing on the chaos.
The benefit of exploratory testing, summarized, is that it helps us understand what happens to an app in the real world where users may do the unexpected. (Let us pause here to point out that someone figured out how to cook salmon in a dishwasher.)
Sugar and spice and all things nice
The recipe for a strong exploratory tester has a few absolute requirements:
First, a genuine curiosity for interacting with the software in unintended ways.
Using an app for its intended purpose is great and all, but a tester who leans towards chaotic good can find a lot of satisfaction in uncovering unintended ways of using software. If you aren't terribly familiar with how silly things can get, we suggest spending some time reading material related to security issues in applications. A few choice resources include Google's Project Zero blog and bug tracker, the sub-reddits /r/netsec and /r/reverseengineering, Phrack, and the PoC || GTFO quarterly issues. One benefit to reading other's reports on bugs and security issues is it can help plant seeds of chaos you can use when poking a feature.
Second, a bit of understanding on how different portions of an application work in conjunction with each other. (This can be discovered by using the app heavily and noting behavior, or by having access to the codebase).
To find an entry point, it helps to understand how the puzzle pieces fit together. There are many ways you can map out the interactions of an application. There's an argument to be made that having access to the codebase is the best resource, but not everyone will have access to this information or be able to digest it. Something that anyone can do is map out the flow of an app on some scratch paper. Important things to keep note of are state transitions, user input, and points where interactions with other users (if applicable) can occur.
Third, a good rapport with your development team.
Our QA members interface heavily with multiple departments: engineering, customer success, design, ops, and sales. It's important to cultivate good relations with all members of the teams you work with, but doubly so with the engineering squad. A good rapport, above all, means that you respect that context switching for devs costs them something (generally time). While we're believers in the saying "there are no dumb questions", that comes with a caveat; the questions you are asking should be well defined and researched. Search internal documentation for keywords associated with your question, check out some posts on stack exchange, and ask your QA compatriots what their thoughts are. Interacting with the dev team is important and a good way to show that you care about their time is doing your best to make sure they aren't answering questions that you can easily find with a little bit of work. Eric Steven Raymond has a great little piece on this titled, "How To Ask Questions The Smart Way".
There's a lot of trial and error in an exploratory tester's path. For every big discovery, there are several avenues that don't result in any bugged behavior. One theme that can be gleaned from any of the sources above is that finding and reproducing weird bugs isn't always the easiest thing in the world, but tenacity will usually allow you to prevail.
What does this button do
Let's take a little dive into what exploratory testing looks like:
As an example, we'll take a look at a bug that was found in an older version of SpiderOak's Semaphor application. If you haven't used it you can think of it as an end-to-end encrypted chat and collaboration tool. The bug involved sneaking in an invalid character during new user creation which caused other accounts it interacted with to break. At the time, the sign-up process could take two different paths: account creation via username or email address.
A few things to note:
Accounts had to be unique. You could not create two users with the username
r33d, for example.
There are restrictions on the type of characters allowed. Things like periods, commas, semicolons, and other common characters used to cause mayhem when parsed were verboten.
If the username or email was valid, a 'Join Semaphor' button would change from grey (not actionable) to blue (actionable) and the process could continue.
The user could switch from email to username signup by hitting a button on the signup screen.
Our test cases for account creation included checking that:
- You could successfully create an account using either option.
- Account creation would fail if the username or email was already in use.
- A user could not proceed with account creation with an invalid entry.
- Error messages would generate if any unwanted behavior was executed.
Entry point: finding a place where a bug could exist.
In this case, a fair bit of time was spent playing with the account creation page looking for interesting interactions. Don't underestimate trial and error as an exploratory testing mainstay. Eventually, it was noticed that when switching between username and email signup options with information already added to the field, there were some inconsistencies in the 'Join Semaphor' button returning to the correct state.
Execution: triggering undesired behavior in the app.
After some further poking, we discovered that when switching between username and email signup, trailing white spaces were removed from the username/email field. When switching back to username signup, the spaces were returned and the 'Join Semaphor' button showed as valid.
Here are the steps used to execute the bug. In this example r33d is the username alias that has already been reserved and a % represents a space character.
- We are first in the username signup mode and add r33d as the username we want to take. We are able to click the 'Join Semaphor' button, but are immediately returned an error stating the the username has been taken.
- We then add a few spaces to the end of the name, the button has now been greyed out as the space character is invalid when creating a username account.
- We then switch to the email signup mode, which removes the trailing white spaces and still shows the grayed out button.
- We switch back to username mode and notice that the trailing spaces are returned and that the 'Join Semaphor' button is blue. We click the button and the account is made.
Having discovered the issue, we immediately went about reproducing it with a new account and reporting the find to the engineering team. It was later discovered that when switching between the two signup states the value being inspected for validation occurs before the new value is set. Since the input of r33d on step 3 is valid for username signup it passes the validation check. Once we press the 'Join Sempahor' button r33d%%% is sent to the server which succeeds since the username doesn't exist.
Reproduction: reproducing the issue multiple times to increase confidence that the bug is genuine.
Holy guacamole, make sure that the bug you are reporting is reproducible and that the reproduction steps can be followed by someone unfamiliar with the product. In our experience, it's not completely uncommon to witness odd behavior that we eventually chalk up to "ghosts". Network hiccups, virtual machine issues, and even human error when executing a bug are ever present possibilities. If we can't reproduce a bug at least two to three times, we'll go back to the execution step and work on finding a reliable way to cause unexpected behavior.
Documentation: clearly documenting the behavior.
The only thing left to do is document the issue. Because we find that it's generally easy for our engineering team to locate an issue once a ticket has been filed, we'll briefly cover our methodology.
Logs, pictures, and movies, oh my! We try to always include these things in tickets. When supplying logs, our aim is to make sure that we are providing a clear picture with as little noise as possible. What that means is we try to isolate the logs down to where the bug is occurring. Providing too much information isn't usually bad, but it can take longer to parse through and find the important parts. There are tons of free tools that can be found to take short videos and screen captures with. Providing images and videos of the behavior will help paint a larger picture of what the user would be doing or experiencing when encountering an issue. A picture can indeed be worth a thousand words.
Include versioning information. We're routinely using test builds with various things added or removed as we move our product towards the end of a sprint and are given a build that might make it into production. Including as much information as possible will not only assist the engineering team in locating the issue, especially if it relates to new features, but it will also help with tracking regressions as the life of the app progresses.
We also make an effort to include notes about the circumstances that may be useful and a summary of what we believe is happening from the code's perspective. If you are incorrect in your description, the engineers usually set you straight. This not only increases your knowledge of the app but also helps build that rapport mentioned earlier. In our experience, the robustness of a ticket directly correlates to the time it takes to locate and remediate discovered issues.
Testing has similar pitfalls to proofreading; it's easy to assume things will work because you think they should, especially after prodding the same application for a long period of time. We find exploratory testing to be an essential complement to test cases and regression testing. What do you know, and why do you think you know it? Try your hardest to challenge assumptions you've made about the item you're testing, and you can usually find a way to prove those assumptions wrong.