The Importance of Feedback

When designing a integration that is going to export data to a separate application, do not underestimate the importance of having a way to validate the data that is being sent. It might not seem critical to the feature but it is.

You must have some capability to inspect what you are sending and what you are getting back. This capability does not need to be present for every request made but it must toggleable in the sense that you can inspect the Request and the Response if you needed to.

The other must have are configurable controls for the rate of export. It is invaluable, during testing, that the flow rate be easily controllable. In this way you can: stop an export job, release a change, tune the flow rate way down, start the export, and inspect a small number of records for validity.

Keeping the rate low for the first run or two gives you the opportunity to inspect the results and make any changes before a large set of data is effected. If there is an issue you only have to deal with a small number of records. This removes a lot of risk from the first job run which is almost always the most risky anyway since that run will have the least testing done to it and it will encounter scenarios that you possibly haven't thought of yet.

Do not scope an integration without these capabilities. You need a test harness if one is not available through any other means and you need controls to tune the job easily. Not including these in the original estimate will only lead to more time fixing and troublingshooting spent later.