Identified
We're implementing fixes to prevent processing issues with large files. Due to the complexity of the work, we expect these fixes to be released tomorrow by midday (CEST)
The next update can be expected by tomorrow morning.
Investigating
We have restored the service to it's nominal level of performance and identified what triggers the issue. However we still don't have a permanent fix for it.
The next incident update should be expected by the end of the day (around 5PM CEST).
The request to all customers for now is to not upload large files for processing (for example, excel files with hundreds of thousands of rows).
Identified
We are working on a fix to avoid large files for being requeued for processing while they are still being processed. This should provide relief to the situation and allow us to identify next steps to address all root causes
Investigating
We observed that the performance of the system post hot fix is still degrade. We continue to investigate where the cause of the issue is to identify the quickest path to restore the service completely.
Monitoring
We've deployed the fix and are monitoring systems in case the incident reoccurs
Identified
The incident reoccurred before the root cause could be completly erradicated. We have identified the issue and the permanent fix will soon be released
Resolved
All systems are operation again.
Resolved
The incident has been resolved.
Monitoring
The outage has been resolved and the AI features work again. We're still monitoring the features to make sure they are fully performant again.
Identified
We've identified the issue and are applying fixes to resolve the outage.
Investigating
We're currently investigating a full outage of AI features on the platform.