Elevated response times in the US region for a small percentage of the requests
Incident Report for Svix
Resolved
There were elevated response times in the US region for a small percentage of the requests. The issue was due to a memory leak in a new third party library we integrated with, which caused some nodes to operate with constrained resources (heavy CPU/MEM usage). This fixed itself automatically by killing the affected nodes (each one as it exhausted its resources over the span of a few minutes) and starting new ones instead. The team has also deployed a fix to revert the inclusion of the new library pending further investigation on why this was happening, and why it wasn't detected in the staging environment.
Posted Aug 24, 2023 - 16:00 UTC