How to Automate Integration Testing
Friday, July 22, 2022
Reading time: 9 minutes
As we live in a rapidly changing, digitizing world, organizations need to react effectively by adapting their systems to manage their daily business activities. To make sure that Microsoft Dynamics 365 Finance and Supply Chain Management integrates smoothly with all other systems within an organization, understanding how integration testing can be done more efficiently, and what capabilities are available, is crucial.
Based on our vast experience in helping Microsoft Dynamics Partners test required integrations, we have developed a set of practices that can be used in various Dynamics projects. In this blog, we will cover the most common patterns, and explain how it is possible to automate the testing of these patterns.
In this blog, we’ll guide you through:
Integration Testing: What Are the Main Challenges?
As you probably know, Enterprise Resource Planning (ERP) software is becoming increasingly integrated, especially in the cloud ecosystem. If you have all your cloud solutions, you can connect to these various kinds of solutions by building custom integrations. Building custom integrations is one thing, but you also need to think about maintenance; and to do proper maintenance, you essentially are tasked to know more about not only Finance and Supply Chain Management, but also about your target systems. Therefore, the organization needs to maintain Finance and Supply Chain Management tests. Plus, they need to maintain their business process test, which might span across multiple services that are hosted in the different cloud or on-premises applications.
The biggest issue with this is that knowledge is mandatory. You need to know where that system is, and what it does. You need to be able to log in, check, verify, connect to it, and perform certain actions.
Finance and Supply Chain Management has a lot of toolkits available for testing inside the solution, such as RSAT or building tests by developer but currently no standard tools available for integration testing. Also, you don’t have one tool that allows you to connect toa particular service or application, so you’d have to go through it one by one. That can be challenging as you have to make a test that spans across these multiple systems.
Another challenge would be that it might require integration testing environments, which means that you need to maintain an integration test environment where all your integrations are connected. Those other services also need to be connected to their test environments, so you need to maintain a lot of connections and a lot of environments that are working together. Not to mention, you may need to do a lot of extra maintenance.
How to Overcome Integration Testing Challenges
Although there are some challenges, integration testing is a necessity as the software changes and as you get new releases and new product updates. It’s very important that these integrations work and you don’t get any interruptions or inconveniences.
But not to worry! We’re here to guide you on how to confront possible obstacles. We have identified three ways you can test out your integrations:
1. Manual Testing
A manual test is when an end-user performs a full end-to-end process to test out any kind of business process. This step involves:
This is the most straightforward way to do this kind of manual test. However, it comes with some disadvantages. First, it’s very prone to errors and if the users do a lot of testing, at some point they might start skipping steps. If they start skipping steps and they don’t do the validation and proper checks, then a lot of errors might come up. In addition, it can be time-consuming as you will need to dedicate the time of end-users so they can test the integrations regularly after new releases or after Microsoft/ISV updates. Therefore, instead of doing their normal workday, the end-users will need to actually trigger those integrations and go through this whole process manually.
2. Simulation Testing
What we usually recommend is simulation testing or end-to-end testing. Simulation testing usually involves some Microsoft tool or framework that would allow you to build an automated test in order to move away from manual tests. The main goal is to trigger the integration framework and verify that the integration returns an expected result (i.e the endpoint works, the service works, the message is transferred, or the message is sent and you get a predefined outcome). The benefit of this approach is that it is relatively straightforward and cost effective to achieve since it always takes a fixed set of parameter values.
In simulation testing, you can decide whether your test will be technical or functional. A technical test means that a developer will be using development tools to build an automated test while a functional test means that an end-user/functional analyst will be using some tools to configure a test that can be executed multiple times. In both cases you will need to define a very clear structure, and what the service is expecting. It’s a very predefined controlled environment where X++ developers can build custom code that expects a certain incoming message with a predefined outcome.
For different integrations, we have different approaches that you can take:
- Calling the service from within Finance and Supply Chain Management (X++ job)
- Postman/Excel or the third-party app that’s executed when needed
- Calling the service from within Finance and Supply Chain Management (X++ Job)
- UI Execution (Prepared by Developer)
- Data task automation setup
- Calling the service from within Finance and Supply Chain Management (X++ Job)
The downside of simulation testing is that it does not cover an end-to-end test. This means that if something changes in the other system, you wouldn’t be able to catch it with this test. That is because it works on some assumptions and if you break those assumptions, the test will not see it.
3. End-to-end testing
End-to-end testing allows you to fully automate a test from one system to another. That is when you do the trigger in one system, and you verify that the other system has a produced the expected outcome. The benefit to end-to-end testing is that if it works during the test, you can be 100% sure that it will work in production, and you also get a full end-to-end view. The downside of this approach is that it’s more difficult to automate, especially if you have a third-party system/portal that you need to sign in to. As a result, your automation is supposed to simulate a sign-in and it is also supposed to trigger an integration in the third-party application, and then come back to Finance and Supply Chain Management and verify the outcome. It becomes similar to manual testing, except you’re doing it automatically. There are not many tools that can do that directly in Finance and Supply Chain Management, therefore it’s much more difficult to achieve this out of the box.
It’s a full end-to-end business process test, the same as the user would be doing it from start to finish. The benefit of this approach is that you do a full parameter input test, test all your triggers, expected outcomes and you can sign off end-to-end business processes. This also includes a connection test which verifies that the connection the third-party application is working as expected. The real downside is that you would need to use some special tools that are able to connect to these different applications and are able to perform different kinds of actions.
Toolkit Overview: RPA vs. Selenium Testing
There are multiple third-party interface testing tools available but for the purpose of this blog, we will introduce you to some of the ones we’ve used in the past. All these tools are from the Microsoft toolkit that you can easily use today.
The Companial team has automated some tests using RPA (robotic process automation) that includes Power Automate and Power Automate Desktop. Using this approach, you can automate physical file openings, generating files, reading/writing in the file as the expected outcome, and then going back to the cloud and verifying everything works correctly. The RPA can be called within the SysTestCase framework, and it can be used for Azure DevOps solutions. SysTestCase is purely for Finance and Supply Chain Management developers, and it is the default test generation framework in X++, which can be used to create automated tests that you can run in Azure DevOps whenever you want. The main idea is to create and put a fully automated X++ test in Power Automate or Power Automate Desktop. Also, you can run this test from an Azure DevOps and it can connect to your own premises. You can connect to your own premises to verify if a correct file was created, moved to the correct directory and it produced an expected result.
Another way is to use C# NUnit framework that can be used for user session automation by selenium testing and WebDrivers for a browser-based application such as Finance and Supply Chain Management. Selenium testing framework can be codable, so that means that a developer can in one scope create a browser session that navigates to your desired third-party web application (customer portal, sales screen, HR system), signs in and performs a series of steps such as input values to text controls, mouse click actions on the web application, reading values from the browser to mimic user actions. After the steps are completed, in the same session the X++ developer can verify that Finance and Supply Chain Management has received any incoming messages from those connections. In addition, this can be done directly in SysTestCase framework.
That means you can put it in Azure DevOps and as a result, run this test every night. So, this could be even done when developers are building modifications and they can always verify that the integrations work all the time. This ensures there’s a very limited chance that something could go wrong during creating new changes. The downside of this approach is that it required X++ development to build and record the steps with the C# framework. There are suppliers that offer applications which work in the same principle without X++ coding (similar to a Power Automate development) but these would need to be evaluated how they work with Finance and Supply Chain Management.
Toolkit Overview: Microsoft Toolbox
The downside of both the above-mentioned tools (RPA and Selenium Testing) is that these are pretty technical. Therefore, it might require some technical effort to implement, and you might need to spend some time digging into it. So, what does Microsoft think about this?
The current Microsoft toolbox (see Figure 1) has been developed for various kinds of testing. However, if you take a closer look, there are no dedicated toolkit integration tests. The good thing is that there is one coming up soon.
Toolkit Overview: RSAT Integration Testing
RSAT (regression suite automation tool) will get a new improvement where our RSAT tool should be able to support an end-to-end integration scenario. It will allow run sessions for third-party applications that run before or after Finance and Supply Chain Management is run. Essentially, you will be able to trigger an integration in some portal and then verify the outcomes in Finance and Supply Chain Management. It’s the same as if you were doing with the C# NUnit framework and selenium WebDrivers as discussed above, although, in this case, you likely won’t require coding. The main idea is to be able to do integration scenario testing and be able to do that without developer help. Since development can be expensive it’s not something that customers really aim to do. We will get this feature in general availability and in public preview within the next few months, so stay tuned and follow the latest Microsoft improvements so you don’t miss out!
Manual or Automated, That’s the Question
At the end of the day, you will need to make the final choice about which method serves you best. Each integration that is built always has some end-to-end scenario with underlying technology. Once the design and implementation is made, you need to start thinking about how to maintain it. Which approach to choose really depends on the actual integration scenario. To help decide, you can ask yourself the following questions:
By considering each of those questions, you will be able to evaluate which testing approach would be the most suitable for your system. For example, if the system is changing a lot, you should either use manual or end-to-end testing as simulation won’t work. It can be that for one integration you will do a manual test, for other you will do a full end-to-end test, and for another it might be that you will simply do a simulation test. And that’s fine as each integration is unique with unique challenges.
We hope this blog provides valuable insights for you and your organization.
If you need further help or advice from our product experts on other development areas, we provide testing services, technical Dynamics 365 Finance and Supply Chain Management consulting for Partners on their Dynamics 365 Finance and Supply Chain Management customer implementation or ISV solution development projects.