SAP released a new SAP Cloud ALM scenario in 2022 that is pretty similar to what you would find in SAP Solution Manager (User Experience Monitoring) and Focused Run (Advanced User Monitoring). This new "Synthetic User Monitoring" (SUM) is essentially a lightweight alternative to bulkier but much more powerful solutions for continuous performance monitoring, such as Tricentis Neoload.
Important fact: Cloud ALM SUM only supports web browsers testing at this time.
Load Testing vs. Stress Testing vs. Performance Testing
Performance testing is an umbrella term that looks at stability, responsiveness, scalability, speed and resource usage of your software or infrastructure. In Agile, like any other type of testing, you should be testing that performance with every change to ensure your software offers the same experience to your users.
Load Testing is a specific type of performance testing and examines how many users your system can actually handle. The test would keep adding user until the maximum threshold is attained and remain at that level for a defined amount of time. This is how most retailers anticipate peak demands such as Black Friday.
Stress testing, on the other hand, tries to find out how your servers react under intense workload. The goal here is to push your infrastructure to its limits and see how it recovers after a peak. This is usually done in waves and the test will also look for critical issues such as memory leak, data corruption or security gaps.
What can you do with Synthetic User Monitoring (SUM)?
SAP Cloud ALM "SUM" is a user performance monitoring tool for two specific types of use cases:
In order to test those two categories, SUM relies on virtual users that will test the target feature continuously based on a predefined schedule. Those virtual users are essentially short automated scripts that can be ran concurrently on several machines at different locations, depending on what you want to measure. For example, you may want to make sure your customers in Europe and North America both enjoy speedy web pages or that Chrome and Firefox users can access the same features.
Cloud ALM is not where you build the scripts. This scenario relies on Selenium IDE, which is a browser addon through which you can record web pages, add conditions (IF-ELSE, LOOP, etc.) and use data variables. You need knowledge of Selenium IDE in order to make sense of this solution.
A practical example of how to deploy Cloud ALM SUM
Disclaimer: This is only for testing and demonstration purpose. Customers should not follow this procedure for productive usage unless they know exactly why they're doing it this way.
First, it is important to understand that Selenium is doing most of the heavy lifting while Cloud ALM is more of an operational dashboard - which is great! Selenium is an open-source automation framework that is being used almost everywhere because of how big the community is and how well it blends with all sorts of programming languages.
The process works like this:
Here we go!
Deploy Selenium Grid
Selenium Grid is a proxy server that makes scheduling and running tests in parallel easy. In this example, we will use Docker to set things up because it is one of the fastest way to get the Grid running. You can find Docker images in the official Github repository.
You now have two options for the setup. The Standalone version allows you to have one execution session available for your test script. If you try running two scripts in parallel, the server will only execute the first one and queue the other. With the Hub version, you can setup as many sessions and browser types as you want, meaning your scripts can run concurrently.
There are tons of options for both, so I'd recommend you go through the recommendation and deploy the version of your choice with Docker. If you don't want to think too much, you can run the following command to get a single session running on Microsoft Edge (use the latest version):
You should then have the Grid running on localhost:4444 or the port of your choice.
Now, Cloud ALM is a secure infrastructure. This means you cannot communicate with the instance from your local network without a proxy. In the real world, you want to expose and secure a separate server and then connect it to CALM. However, a way to circumvent this for testing is to use localtunnel or a similar solution. After installing it, you can run the following command to expose your localhost to the outside (make sure to understand what this means from a security perspective):
That's it, you now have a server available for Cloud ALM to trigger executions from. The following is an example of a Hub with a total of 20 concurrent sessions.
Build the Selenium IDE script
There is not much to say here. Download the addon, start recording your steps and adjust the script until it is stable enough to run hundreds of times without failing. One important advice is to keep things short. You are NOT trying to test a complete functional requirement, only a few pieces that are critical to your core processes. If you end up with a script that is 50 steps long, I urge you to break it down into smaller components.
The official SAP Cloud ALM wiki offers a lot of insights on best practices and known limitations when it comes to building the Selenium IDE script: Synthetic User Monitoring - Selenium IDE Scripts (sap.com).
Assign the script in SAP Cloud ALM
You have a server listening for scripts to be ran against your system. You have a script that contains the steps to be performed against that system. Now, you can start thinking about running it!
Logon to SAP Cloud ALM and navigate to the "Synthetic User Monitoring" tile. From there, click the Settings icon to enter the setup:
It's alive
When you're ready, you can come back to the General tab of the previous section, assign your Runner to the Scenario and activate the scenario. Wait a few minutes and you should start seeing data on your Cloud ALM dashboard. The Runner will automatically queue or concurrently execute your scripts based on availability.
Important fact: Cloud ALM SUM only supports web browsers testing at this time.
Load Testing vs. Stress Testing vs. Performance Testing
Performance testing is an umbrella term that looks at stability, responsiveness, scalability, speed and resource usage of your software or infrastructure. In Agile, like any other type of testing, you should be testing that performance with every change to ensure your software offers the same experience to your users.
Load Testing is a specific type of performance testing and examines how many users your system can actually handle. The test would keep adding user until the maximum threshold is attained and remain at that level for a defined amount of time. This is how most retailers anticipate peak demands such as Black Friday.
Stress testing, on the other hand, tries to find out how your servers react under intense workload. The goal here is to push your infrastructure to its limits and see how it recovers after a peak. This is usually done in waves and the test will also look for critical issues such as memory leak, data corruption or security gaps.
What can you do with Synthetic User Monitoring (SUM)?
SAP Cloud ALM "SUM" is a user performance monitoring tool for two specific types of use cases:
- Unavailability: The monitored feature is not responding.
- Performance: The monitored feature is responding but is experiencing slowdowns.
In order to test those two categories, SUM relies on virtual users that will test the target feature continuously based on a predefined schedule. Those virtual users are essentially short automated scripts that can be ran concurrently on several machines at different locations, depending on what you want to measure. For example, you may want to make sure your customers in Europe and North America both enjoy speedy web pages or that Chrome and Firefox users can access the same features.
Cloud ALM is not where you build the scripts. This scenario relies on Selenium IDE, which is a browser addon through which you can record web pages, add conditions (IF-ELSE, LOOP, etc.) and use data variables. You need knowledge of Selenium IDE in order to make sense of this solution.
A practical example of how to deploy Cloud ALM SUM
Disclaimer: This is only for testing and demonstration purpose. Customers should not follow this procedure for productive usage unless they know exactly why they're doing it this way.
First, it is important to understand that Selenium is doing most of the heavy lifting while Cloud ALM is more of an operational dashboard - which is great! Selenium is an open-source automation framework that is being used almost everywhere because of how big the community is and how well it blends with all sorts of programming languages.
The process works like this:
- Deploy a Selenium Grid server for scheduling and execution.
- Build your test script with Selenium IDE (only).
- Assign your script to a scenario in SAP Cloud ALM.
- Plan the scenario execution and activate.
Here we go!
Deploy Selenium Grid
Selenium Grid is a proxy server that makes scheduling and running tests in parallel easy. In this example, we will use Docker to set things up because it is one of the fastest way to get the Grid running. You can find Docker images in the official Github repository.
You now have two options for the setup. The Standalone version allows you to have one execution session available for your test script. If you try running two scripts in parallel, the server will only execute the first one and queue the other. With the Hub version, you can setup as many sessions and browser types as you want, meaning your scripts can run concurrently.
There are tons of options for both, so I'd recommend you go through the recommendation and deploy the version of your choice with Docker. If you don't want to think too much, you can run the following command to get a single session running on Microsoft Edge (use the latest version):
docker run -d -p 4444:4444 --shm-size="2g" selenium/standalone-edge:4.5.3-20221024You should then have the Grid running on localhost:4444 or the port of your choice.
Now, Cloud ALM is a secure infrastructure. This means you cannot communicate with the instance from your local network without a proxy. In the real world, you want to expose and secure a separate server and then connect it to CALM. However, a way to circumvent this for testing is to use localtunnel or a similar solution. After installing it, you can run the following command to expose your localhost to the outside (make sure to understand what this means from a security perspective):
lt --subdomain [choose anything] --port 4444That's it, you now have a server available for Cloud ALM to trigger executions from. The following is an example of a Hub with a total of 20 concurrent sessions.
Build the Selenium IDE script
There is not much to say here. Download the addon, start recording your steps and adjust the script until it is stable enough to run hundreds of times without failing. One important advice is to keep things short. You are NOT trying to test a complete functional requirement, only a few pieces that are critical to your core processes. If you end up with a script that is 50 steps long, I urge you to break it down into smaller components.
The official SAP Cloud ALM wiki offers a lot of insights on best practices and known limitations when it comes to building the Selenium IDE script: Synthetic User Monitoring - Selenium IDE Scripts (sap.com).
Assign the script in SAP Cloud ALM
You have a server listening for scripts to be ran against your system. You have a script that contains the steps to be performed against that system. Now, you can start thinking about running it!
Logon to SAP Cloud ALM and navigate to the "Synthetic User Monitoring" tile. From there, click the Settings icon to enter the setup:
- Create a "Runner". The Runner is Selenium Grid with two key inputs: 1) the hostname is your (HTTPS) server URL, such as https://yourchoice.loca.lt/ if you are using localtunnel. Unless you tweaked the Grid server, you can enter secret as username and password for the endpoint.
- Create a "Resource". The Resource is the Selenium IDE script. Click Upload, select your file and confirm. You can update the resource with a new version by re-uploading the same script.
- Create a "Scenario". The Scenario is the script customized for Cloud ALM needs.Leave the Performance tab options as-is to have Cloud ALM automatically set the performance thresholds (e.g., orange for acceptable slowdown, red for critical slowdown). If you know your infrastructure very well, feel free to manually set those.
In the Scheduling tab, you define when to run the script (e.g., the default is every 300 seconds) - there is currently no option to set time frames, such as Monday to Friday or 8AM to 12PM.
In the Variables tab, you can set secure test data so that username and passwords are encrypted (see the SAP wiki for a lot more information about variables).
Finally, in the Selenium tab, you can adjust how the script behaves. One important option is to define the browser to be used for the execution, such as Chrome or Edge, as well as any specific argument you need (e.g., Incognito mode). If you would like the test scripts to take screenshots as well, enable the "Automatic screenshots" option - however this will use some of your instance disk space.
It's alive
When you're ready, you can come back to the General tab of the previous section, assign your Runner to the Scenario and activate the scenario. Wait a few minutes and you should start seeing data on your Cloud ALM dashboard. The Runner will automatically queue or concurrently execute your scripts based on availability.
- SAP Managed Tags:
