Challenges for Black Box Testing and how to solve them

Black Box Testing is always more complicated than if you have full control over the application to test. But for all these challenges, there are solutions available. Some of them are easy, some of them a little bit more complicated.

I talked a lot about these challenges in my last stream, where I wrote Integration Tests for Minecraft. Yes, I wrote a Scenario for Minecraft and automated it.

But let’s don’t waste time and jump into the challenges!

How do you control the application from the outside?

The first question, when you want to automate something from the outside, is how you can control it. Sometimes the choice of tool is easy.

For Minecraft I found a library, which lets you write bots for it, which was quite fun. But what can you use for your normal line-of-business applications?

Your setup looks like this:

Web Applications

Here is the obvious choice Selenium to automate your application. Or you can use the rather new Playwright for it.

If you want to see how to use SpecFlow with Selenium, you can watch my video series about it on YouTube. For Playwright I made a video how to migrate to it.

Mobile Applications

For that I would recommend using Appium. It is based on the same API as Selenium, so if you know one, you know how to use the other.

There are some challenges in using it, but they are solvable. For more details, watch my video series about it.

Desktop Applications

Here can I recommend another Selenium/Appium API based tool. Microsoft wrote an App Driver (like ChromeDriver, EdgeDriver) for Windows applications. It enables you to automate Universal Windows Platform (UWP), Windows Forms (WinForms), Windows Presentation Foundation (WPF), and Classic Windows (Win32) apps on Windows 10 PCs.

APIs

If you have a HTTP/Json based API, have a look at RestSharp.

Start the Application

Somehow you need to start the application you want to test. In my Minecraft session, I simply could start some processes and waited until they were ready. That’s as easy as it gets.

Web Applications

If you are lucky, you can start up an in-process webserver to host the application. But as you only get a complete application, you won’t have this luck.

On Windows you probably have to configure and start/stop an IIS. But when do you know, that the application is ready to use? If there is a status endpoint to query, you could use this. If not, you can query an endpoint, that has no sideeffect, so it doesn’t matter, that you call it multiple times.

Mobile Applications

Here it is easy, because Appium handles that for you.

Timing Issues

When you start automating an application, there are quite quickly some parts, where you simply have to wait. I am pretty sure, you will start with some waiting/sleeping calls that block everything. And that is ok, but you will quickly run into problems with it. On is, that you are always increasing the waiting duration, because of timing issues.

So try to rather early than later other techniques. One is to use polling with a lower waiting duration between calls. With that you will reduce the wait time overall.

If the APIs are providing events, when something is finished, please use them. Then you have no unnecessary wait time at all.

You can read more about this in the GivenWhenThenWithStyle Challenge #3 – How to deal with pauses and timeouts?

Resetting the Environment

This is the challenge, where the short answer is “It depends”. Again with the Minecraft automatization, it was easy, as I had just to copy files from a good copy of the world. If you have a database, you have to restore a database backup (which takes probably ages). But again, this depends on your database.

Perhaps you also can simply copy files back from a good copy.

So you see, it depends really on your applications.

Special Configuration

There is a chance, that you need to configure your application in a specific way, that it will get testable. For Minecraft it was some server configurations. But if you have to test a Micro- Service, which calls other Micro- Services, you probably want to configure it in a way, that it is calling some Stub/Mock-Microservices to have it also in control, what they are returning.

Keeping the State of the Applications twice

Because you can’t access the state of the application, you are probably need very quickly to store also some state in your test automation. This is normal. Just be cautious that the two states doesn’t run into different directions. These are bugs that are hard to find, so spend some more time in designing an API around this.

It looks like this: