How to Use ERP Regression Testing to Prevent Downstream Disaster

Changes—even small ones—in your Microsoft Dynamics ERP can have major downstream impacts, because your systems are integrated. Bringing all your business functions under one umbrella can do wonders for efficiency, but the flip side is that multiple departments could be affected by a minor alteration in your ERP solution. That's why you need to be familiar with regression testing - the process of running tests on software applications to ensure that programming changes or additions haven't broken existing functionality or introduced bugs into the system. Dynamics 365 for Finance & Operations can be affected by custom code, updated settings, configurations or new data. By creating pre-determined test scenarios to simulate how data moves through your system and the actions that happen as it does, you can identify and correct potentially costly errors before pushing your changes live.

Best Practices for Regression Testing

Regression testing most often occurs during the initial implementation of your ERP system, when the largest number of changes are being made. If you have a good ERP implementation partner, testing is a critical part of your implementation roadmap. However, you should test any time you change or update your ERP system, even if you've been using it for years.

Regression testing may be time consuming, but not doing it can be disastrous. For example, we worked with a customer that specialized in drop shipment orders to the U.S. from China. The inventory went straight from the vendor to the customer without going through a warehouse. The company neglected to test all possible scenarios, so it wasn't until they were already up and running that they realized purchase and sales orders were being entered incorrectly, with significant impact to their general ledger. They had to revert back to manual processes, trace back the mistakes, and then retrain their department to enter the information correctly in the new ERP. It cost a lot of time and effort and caused a lot of pain that could have been avoided by testing prior to launch.

Below are some best practices to consider as you prepare to test:

1. Everything in the scope of your D365 F&O project should be reflected in the test scenarios you create. This should include ancillary systems that feed data into your ERP system. For example, if you are using Concur to manage your organization's expenses and travel, be sure to test the upload from Concur into D365, to see if you need to make changes in Concur before continuing with your testing.
2. Use actual orders you've executed in the past as test cases, to simulate real-world conditions.
3. Don't test just one record. Make sure tests reflect the real-world volume of data you expect in journal entries, orders, invoices, etc.
4. Test from end to end, verifying results every step of the way. If you are testing a customer purchase scenario, make sure the test order makes its way properly through sales, the warehouse and on to accounts receivable for payment. Don't run tests for each department in silos, because you want to make sure that the data and outcomes are consistent through each department.
5. Compare the outcomes of test scenarios in your new ERP system with results from your legacy systems. This way, you can identify where outcomes are different before they adversely affect your business.
6. Identify all possible business case scenarios. For example, there may be three possible paths for a customer order to get entered into your ERP: by telephone, by email or via EDI. You need to test all three, because you don't want to find out after your system has gone live that entering orders via EDI doesn't work, for example.
7. Remember to test exception cases. Every company has them, and they've probably caused you headaches in the past. If you don't test those outliers now, they will inevitably emerge after you've gone live, so it's better to be prepared upfront
8. Test data from various sources. If you consistently test data from the same customer or vendor, you may get correct results. But you probably have multiple data sources, such as international customers or vendors, along with existing customers or vendors that have been migrated from your legacy systems.
9. Test data that has been manually added to your new system, as well as data that was migrated from your old system. You'll want to make sure there's a consistent result, no matter where your data originated.
10. Verify that legacy data is migrated correctly, and test that it works all the way through the process. For example, you don't want to miss an address line when you print your invoices and end up sending products to the wrong address.

For all of the above, you want to define expected or assumed outcomes prior to testing. Then compare your expected results to your actual results.

Your expected outcomes can also be positive or negative. The expected positive outcome for a purchase order test would be one where the order ships successfully, and an invoice is sent to the customer. If the test scenario calls for you to select an item that cannot be sold in the selected locale, your expected negative outcome would be an error message. If your actual outcomes don't match your expected outcomes, be prepared to investigate why, make necessary changes, and then run the test again.

Optimizing Your Testing

To get the most out of regression testing, you'll want the people who will be executing the tasks on a day-to-day basis to input the data into D365 F&O. You get two tests for the price of one: You can test your ERP setup, and your employees get trained and tested on using the system in a scenario that doesn't (yet) have real-world consequences.

Running the tests with your normal user base also functions as a stress/load test, to make sure your configuration can handle having a normal number of users logged in and using the system, with a standard number of transactions.

Additionally, some of this testing can be automated. Our development team has built an automated testing framework, so we can automate and repeat the testing of a single order going all the way through the system multiple times with a large amount of data.

By testing changes to D365 F&O early and often, you can help ensure that when those changes go live, business will proceed smoothly.