Alternatives to Postman for Exploratory and Automated API Testing
Postman is a great tool for exploratory API Testing. However, it is not the best tool for automated API system testing and there will be a
change in September which can result in higher costs for using Postman. In this blog post, I will explain why from my perspective code based open source tools are often better for automated API system testing than GUI based tools like Postman or Insomnia. I will also provide some recommendations which tool to use for which technology stack.
System Testing against the API layer is a vital part of software development, as it ensures that the APIs are functioning correctly and meeting the expected requirements. However, not all API testing tools are created equal. In this blog post, I will compare and contrast two types of API testing tools: code based open source tools and GUI based tools like Postman or Insomnia. While Postman and Insomnia have both value in the context of exploratory API Testing - I will argue that code based open source tools are better for automated API system testing, especially in terms of maintainability and integration into existing development infrastructures.
Postman change on the 15th of September 2023
On September 15th, Postman removed the legacy Scratch Pad component from its Postman Desktop App, but you can still continue using the same API testing, documentation and design functionality in the same app for free, in combination with a SaaS account that provides additional free functionality like monitors, mocking, real time collaboration and developer portal integration. Pricing is not changed, unless you have not paid for Postman before and need to adapt an Enterprise plan in order to use Postman’s SaaS offering in a company specific compliance manner. Still compliance requirements make it necessary for many companies to switch to an Enterprise plan.
Newman is OpenSource
Newman is an official open-source project associated with Postman. Its purpose is to enable the execution of any Postman collection test suite within a Continuous Integration / Continuous Delivery (CI/CD) pipeline. It can also be integrated natively into a JavaScript-based execution environment using an npm module. Moreover, Newman can successfully function with collections located within Postman’s SaaS platform, files on the local file system, and files within a version control system that follows the official collection format. In essence, collections can be run using this open-source tool. However, for convenient maintenance of collections and other artefacts, Postman is required.
Testing against the API layer on the Component Level
There are different levels of testing against the API. In addition to do API system testing through the whole stack, it is also possible to test against the API on the component level including isolation from fragile or slow dependencies (e.g. database or external services). Those are not covered by this document and usually require a different toolset fitting to the technology stack of the backend. For example, for Spring Boot there are several ways with different tradeoffs how this can be done.
Why Code Based Open Source Tools Are Better for Automated API System Testing compared to Postman, Insomnia and Bruno
System Testing against the API layer is a vital part of software development, as it ensures that the APIs are functioning correctly and meeting the expected requirements. However, not all API testing tools are created equal. In this blog post, I will compare and contrast two types of API testing tools: code based open source tools and GUI based tools like Postman, Insomnia and Bruno. While Bruno, Postman and Insomnia have value in the context of exploratory API Testing - I will argue that code based open source tools are better for automated API system testing, especially in terms of maintainability and integration into existing development infrastructures. And even for exploratory API testing many developers just use the tools for automated API system testing and do not need the special category.
Code based open source tools are API testing tools that allow the user to write and execute test scripts using a programming language, such as Java, Python, JavaScript, etc. Some examples of code based open source tools are
REST-Assured,
Karate,
supertest and
PyTest.
What are GUI based tools like Postman, Insomnia or Bruno for exploratory API Testing?
GUI based tools like
Postman,
Insomnia or
Bruno are API testing tools that provide a graphical user interface (GUI) for the user to create and run test requests using a point-and-click approach. These tools are popular among developers and testers, as they offer a user-friendly and intuitive interface. Some features of these tools include:
- Support for various authentication methods
- Ability to create and manage collections of requests
- Built-in support for API documentation
- Wide range of integrations and plugins
Due to cost and compliance issues Postman and Insomnia cannot be recommended anymore.
Bruno is quite similar to Postman and Insomnia and available as OpenSource, free to use and does need a cloud account, collections can be stored in your version control system and Bruno includes cli tool for running collections. More details can be found in the
comparison page.
What are the alternatives to Postman, Insomnia and Bruno?
There are a lot of tools available that are similar to Postman and Insomnia - for a listing of the most popular, see
Awesome HTTP Clients. There is e.g. good support for
JetBrains, a
plugin for VS Code or
the restclient package for Emacs, and a
CLI, amongst others. Only a few of those are allowed to be used at SAP, more details about the SRI entry you can find at the end of the blog. An advantage is that this allows to store these request files in your repo, version control them like the rest of your (test) code and you are not reliant on yet another tool, but just use it from your IDE of choice.
While GUI based tools like Postman or Insomnia have their advantages, they also have some limitations when it comes to automated API system testing. Automated API system testing is the process of running a set of test cases repeatedly and automatically to verify the functionality, performance, and security of the APIs. This process is essential for ensuring the quality and reliability of the software.
Code based open source tools are better for automated API system testing for the following reasons:
- Maintainability: Code based open source tools allow the user to write test scripts using a programming language, which gives them more control and flexibility over the test logic and flow. This makes it easier to maintain and update the test scripts as the APIs change or evolve. On the other hand, GUI based tools like Postman or Insomnia rely on a graphical interface, which can be cumbersome and prone to errors when dealing with complex or dynamic scenarios. Moreover, code based open source tools can leverage existing frameworks and libraries that provide reusable and modular code components, which can improve the readability and maintainability of the test scripts.
- Integration: Code based open source tools can be easily integrated with other tools and systems, such as continuous integration (CI), continuous delivery (CD), version control, reporting, etc. This can enhance the automation and collaboration capabilities of the testing process. GUI based tools like Postman or Insomnia have some extensibility features, but they are not as comprehensive or customizable as code based open source tools.
- Cost-effectiveness: Code based open source tools are free and open source, meaning that anyone can use them without paying any fees or licenses. This can save a lot of money and time for the users, as they do not have to worry about purchasing or renewing subscriptions or licenses. GUI based tools like Postman or Insomnia have free versions, but they also have paid plans that offer more features and functionalities. Sometimes these paid plans can be quite expensive or the license come with changes of terms and conditions that can be problematic for the users. Test code can be as important as productive code. You need to consider that for some products the code will need to be maintained for many decades. Sometimes the test code even outlives the productive code, since it contains the domain knowledge in the format of tests. This can remain relevant even if you rewrite the productive code. With 3rd party tools, there comes a risk and a dependency to the supplier including licensing issues. Therefore, for test code you should prefer open source tools over 3rd party tools.
The following further disadvantages of a Postman setup were mentioned by teams after migrating to code-based tools for automation:
- High complexity of test sequences and scripts in Postman / Newman.
- Poor maintainability, unstable execution
- Good Integration of Postman requires higher tier: Using newman and postman CLI, you can run Postman API tests from any Git based pipeline, also on the free plan. The higher tiers of Postman also provide a built-in GitHub, GitLab, ADO and BitBucket integration, including support for Pull Requests.
- Code Review with pull requests is not as straight forward, since collections are harder to review. The higher tiers of Postman provide better support.
- The common approach to store collections externally in the cloud can be incompliant with company policies. On the other hand, sharing code instead in a repository is transparent and compliant.
Recommendations: Alternative for Postman and Insomnia for Automated System Testing
Recommendations: Alternative for Postman and Insomnia for Automated System Testing
For Automated System Testing against the API layer, you can use the tool, which is fitting best for your technology stack. Below some code based Open Source recommendations for the respective technology stacks - the list is not exclusive and there are further similar Open Source alternatives.
- Java
- JavaScript
- Mocha or Jest NodeJS test frameworks
- Supertest HTTP client library
- If you need to test against OData APIs you can also leverage the OData libraries SAP OData library. For QMate there is a small wrapper available, which eases to make API calls for preparing transactional data for UI testing.
- Python
- ABAP
- Either leveraging one of the above tools or writing system test with ABAP unit. Especially the tools from the JavaScript stack are used by teams in Procurement. It allows the additional benefit that it can be easily combined with JavaScript UI testing frameworks (like QMate)
Conclusion
In conclusion, code based open source tools are better for automated API system testing than GUI based tools like Postman or Insomnia, especially in terms of maintainability. Code based open source tools offer more control, maintainability, flexibility, extensibility, and cost-effectiveness than GUI based tools. Therefore, I recommend using code based open source tools for your automated API system testing needs. And tools like Bruno, Postman and Insomnia only for exploratory API testing. If Postman is becoming too expensive due to the changes in September and you prefer a similar tool, you can check out Bruno for the exploratory API system testing case.
Use Case |
Stack |
Recommended Solution |
Comment |
|---|
Exploratory Testing |
General |
Bruno or the IDE plugins Visual Studio Code REST Client or JetBrains IntelliJ HTTP client |
Choice depends on preference for GUI support or better integration in IDE |
Test Automation |
Java |
Karate or RestAssured |
Choice depends on existing technology stack |
|
JavaScript |
Mocha or Jest, Supertest - HTTP client library |
Choice depends on existing technology stack |
|
ABAP |
Either leveraging JavaScript based tools or writing system test with ABAP unit. |
Choice depends on existing technology stack |
For exploratory API testing we can recommend to use
Bruno or the mentioned plugins for VSCode. It should cover the most needed features for exploratory API Testing and allows to store the collections in your repository, version control them like the rest of your code and you do not need a cloud account to use it, which dependending on the context can increase compliance and security. And so far all colleagues mentioned that the migration was straight forward.
Appendix
Do I really need a tool like Postman, Insomnia or Bruno for exploratory API Testing?
Some developers prefer most of the time code based tools for exploration. Especially if everything is more like a continuous improvement of an existing system. But there are different roles and also sometimes more explorative ways (e.g. exploring unfamiliar APIs) where a exploratory testing tool is preferred. If you and all involved roles are fine with it, there is no need for exploratory API testing tools. Still there are contexts where such tools are needed.
What are benefits of Bruno compared to Postman and Insomnia?
Bruno is OpenSource and free to use. It allows to store the collection in your repository, version control them like the rest of your code. Besides, you do not need a cloud account to use it, which increases compliance and security. More details can be found in the
comparision table.
Resources
Related Blogs on Test Automation
You can find more information about test automation in the following blogs:
