Testing any software product is essential, especially when you’re offering it as a service, known as a SaaS product. It’s not just about launching a product and calling it done; you’ll be adding new features and making regular updates. For instance, if you launch a product that claims to be the best video conferencing for small business, it’s vital to consistently meet the changing communication needs of organizations. Staying updated on the latest software testing news and adopting new strategies has never been more important.
This is where smoke testing shows its value. With its quick approach to checking if your product functions correctly, it’s no surprise that the world’s leading companies consistently use smoke testing for their software. Whether your SaaS focuses on email marketing, accounting, or onboarding automation, it serves as a useful step in your QA process. If you’re planning to launch a new SaaS product, smoke testing can effectively identify any bugs or technical problems before it reaches customers—helping to improve your CSAT score and allowing you to gauge your product’s market demand.
What is smoke testing, anyway?
Simply put, smoke testing is the act of running a test to determine whether a software build is stable or unstable.
If the software is stable, the QA (quality assurance) team will step in to perform further software testing. This initial test helps spot any small bugs and can prevent further testing taking place if the software build is unstable—an important aspect as it stops resources from being wasted on unnecessary testing.
Most, if not all, SaaS businesses regularly carry out smoke tests and see it as an integral part of their business process, but at the end of the day it all comes down to your definition of business process, and what that looks like for you. After all, if an organization begins to notice that a system or a process failed, it is up to that organization to address the issue as they see fit.
You may decide to forgo smoke tests in favor of sanity testing later down the line if it suits your specific business process at that moment.
The difference between sanity tests and smoke tests
One important thing to remember, however, is that smoke tests and sanity tests are worlds apart.
Generally speaking, smoke testing happens before more detailed testing takes place, checking the overall functionality of a piece of software. It ensures the software is stable, and gives an overview of the entire application
Sanity testing, however, focuses on specific functions or bug fixes. Rather than checking the functionality of the overall software, it looks at the changes that have been made to ensure they work before pushing it through to more detailed tests.
Both can be done using automation testing and tools, and they’re usually done by the developers, before it gets sent to the QA team. Both attended and unattended automation processes could assist in your QA testing.
At this point you might be asking yourself “what is attended automation? what is QA testing, and how is it different to smoke testing?” QA testing usually involves a much more in-depth set of tests, including regression testing, integration testing, and more. Whereas smoke testing does a quick general check, the full QA process methodically ensures every aspect of your SaaS product works.
Types of smoke testing
Depending on who is talking about smoke testing, it can mean one of two things. We’ve talked about it as part of the software development process, but it can also be used during the design phase.
Smoke testing for quality assurance
Smoke testing for quality assurance purposes involves testing the overall functionality of a piece of software, flagging up major bugs or errors before a more thorough testing process is done.
Smoke testing for quality assurance is usually one of the software testing methods that are part of a much bigger QA strategy; a combination of insights and techniques used to help test the end-game of your product. Many teams use automated QA tests. What is automated QA testing? It’s almost exactly the same as manual testing, except it’s undertaken by automation tools instead of people.
While smoke testing is a great way to quickly ensure your SaaS product works, it’s important to have a detailed QA strategy in place. You also need a clear overview of the different types of testing you’ll run (like functional testing) and the risks and mitigating factors.
Smoke testing for product validation
Smoke testing for product validation tends to come up in the design phase. It’s done before you’ve built your software, to help you get an idea of your product’s market demand.
It’s commonly confused with a design sprint: an Agile process that involves using design and strategy (think methods like process mapping) to mitigate any risks that come with launching the new product to the market.
When smoke testing for product validation, it’s key you leave your ego at the door and put your customers first. This might mean putting out surveys for customers to fill in about your product or about your product’s lo-fi model or when doing crowdtesting.
When is smoke testing performed?
As mentioned above, smoke testing is typically carried out either for quality assurance or to check a product’s validation. When smoke testing your SaaS product it’s important you understand both reasons as they will guide your smoke test in a specific direction.
Since we’re concentrating on the development process, we’ll be focusing more on smoke testing for quality assurance. In this case, smoke testing is carried out whenever a new feature or function of software is developed and integrated into an existing build that is brought into a QA environment.
Simply put, smoke testing for QA purposes is done whenever a change has occurred within the software build. This might be anything from adding a CTA button to your site or adding a voice call feature to your software – so long as a change in your build’s makeup has occurred, smoke testing is applicable.
The QA team is then brought in to test the overall build against its critical functionalities, helping to uncover any flaws within the build itself. If the QA test passes without a problem, then the build is directed to further testing such as functional and sanity testing. If it doesn’t work as it should, it’s transferred back to the development team so they can continue to improve its stability before it’s passed on for further testing.
The importance of smoke testing
Let’s say you’re an SaaS SEO company that offers businesses software to help with keyword research and linking. Your company is ready to handle client success, your product is already out there and in use, and you’ve just added a new feature, so you want to make sure everything runs smoothly once it’s launched. Alongside process mining vs process discovery to ensure your organization is functioning without a hitch, you also want to ensure that you work out any issues for your service itself – hence QA testing.
This is where smoke testing comes in. Smoke testing is important because it uncovers flaws and errors in your build early on, preventing further development and testing from being carried out until the malfunctions are fixed. This prevents you from wasting resources and time on a build that is already flawed.
Whenever you’ve implemented a new feature or functionality into your build, smoke testing will help determine whether it’s stable enough to carry the software as a whole and be successful.
How to smoke test for QA purposes
When it comes to smoke-testing, there’s typically two main ways of doing it: manually and automatically. Both methods follow the same steps, the only difference is in who – or what – performs the testing; one is done via automation, possibly with the help of RPA and machine learning, and the other is done manually.
In any case, both follow the same cycle – the software build is finished, smoke testing is done to determine its stability, and if no errors are detected the build is moved on for further testing.
Checklist for a successful smoke test:
When running your smoke test, make sure it adheres to this checklist. If not, you might not be running your test as efficiently as you could. Smoke testing needs to be:
- Quick to process
- Self-scoring (if automated)
- Able to provide results on the build’s critical functionalities, as well any flaws (or lack thereof) present
- Given the green light by both the QA and developer team
1. Perform set-up tests
Once you’ve got your build ready, you’ll probably need to perform a few run-of-the-mill setup tests before moving onto smoke-testing. These tests might include setting up servers and database tables to looking into your build’s licenses.
2. Gather your test files
Now that you’ve got your setup out the way, you can move onto gathering the test files you’ll be using during the smoke-testing process.
3. Write a single script
Your next step is to write out a single script for smoke testing to ensure flexibility and stability, running the test from your build tool. Once you’ve run the smoke test, be sure to save the results with your other software build files – this step will essentially indicate whether your build is stable or unstable.
If unstable, this means the smoke test has detected errors in your build which means you’ll have to refer it back to the devops team for extra work and inspection. If stable, this means your build has been successful and can now be passed on further testing.
4. Clean up your setup
Once you’ve completed your smoke test, it’s a good idea to clean your initial setup that you organized at the start. This way you’ll have a clean environment for future QA testing, though this cleanup can also be done just before you start smoke testing.
Takeaway
As part of your product development lifecycle, smoke testing your SaaS product will help you give you a rough idea of your market demand, and iron out any issues early on. No matter which smoke testing route you choose to go down, just make sure to perform them regularly whenever you’re about to launch new software and always get your users involved in the process.