We experienced an API issue yesterday, after making a small change to our platform in preparation for migrating to our revamped infrastructure next week. This change resulted in an unanticipated permissions conflict, meaning that about 400 clients were unable to access our "latest.json" API endpoint between approximately 03:00 and 06:20 UTC.
Instead of their regular up-to-date exchange rate information, affected clients received an error message stating that their App ID was invalid.
The change we made was in a section of our production environment data pipeline that lies outside of our regular deployment path, and was not picked up by our independent monitoring due to the specific conditions under which it occurred, so we were unable to catch the error ahead of time.
As soon as we received notice that clients were unable to access our latest API endpoint at 06:00 UTC, we investigated, detected and deployed a fix for the error. From 06:20 UTC, affected clients were once again able to access our latest API. Any client who had cached a response locally during that time may have continued to experience errors in their integration until their cache was renewed.
This incident was caused by human error. We have tightened up our pre-deployment review procedures in response, to ensure that it can not happen again.
We apologise that this occurred and sincerely hope it didn't cause severe issues for you or your business.
Please don't hesitate to contact firstname.lastname@example.org with any questions or comments.
William Crowcroft - Founder & CEO, Open Exchange Rates