Hypercare Status - United States

Hypercare Status - United States

Back
Oct 18, 2022 2:07 PM EDT

Intermittent Delays

Impacted:
Code Teams - Canadian Region

Updates

  • Nov 8, 2022 6:33 PM EST

    We sincerely apologize for any impact of the degraded performance incident may have caused your organization of the last few weeks. After monitoring the performance of our servers we've deduced that this incident has been resolved. Please be advised that a post mortem will be sent out by the end of this week to explain the technical issues our servers experienced along with the fix that was deployed.

    Resolved
  • Oct 20, 2022 7:26 PM EDT (2 weeks, 5 days, and 7 minutes earlier)

    ----------- Canadian Region Only ----------We have put in an updated version for the web platform that was causing additional stress to the server which reduces the number of unnecessary network requests sent. We believe this should prevent any slowdowns from occurring while allowing us time to resolve the root issue. We will continue to monitor the situation. The fix for the root issue will be coming out early next week but with the mitigation (additional servers) as well as the updated version for web platform we believe slowdowns should not occur anymore.

    Monitoring
  • Oct 19, 2022 3:05 PM EDT (1 day, 4 hours, and 21 minutes earlier)

    --- Relevant to Canadian Regions and Customers --- Our system has been reacting very positively since we upgraded our infrastructure at 19:00 last night to support the additional load generated from our web platform. The root cause has yet to be resolved however you should see little to no slowdowns given this upgrade in infrastructure. We hope to have this fully resolved soon and will continue posting updates.

    Monitoring
  • Oct 18, 2022 5:56 PM EDT (21 hours and 9 minutes earlier)

    Our team will be running some stress tests around 2100 EST to ensure our infrastructure can absorb any potential high traffic load as a mitigation until the problem is fully resolved. Once our team confirms this stress test passes, our system will be marked as fully operational in the Canadian region.

    Monitoring
  • Oct 18, 2022 5:25 PM EDT (31 minutes earlier)

    The issue seems to be identified and traced back to the web browser platform. We are currently investigating all the possible reasons why this issue is occurring on the web platform and will update you as soon as we have any updates.

    Identified
  • Oct 18, 2022 3:47 PM EDT (1 hour and 38 minutes earlier)

    The engineering team remains engaged to determine the trigger of the issue and continues to explore multiple paths to resolution. So far we know that our servers are experiencing a larger than usual spike in resource consumption and is only affecting users in the Canadian region.We’ll provide another update when additional information becomes available.

    Monitoring
  • Oct 18, 2022 2:07 PM EDT (1 hour and 40 minutes earlier)

    ---- Affecting only the Canadian Region ----There is a Hypercare service disruption that is occurring intermittently on the Canadian servers. During a service disruption, end users will have performance issues that affect their experience with Hypercare. We’re investigating the issue and an update will be provided in 30 minutes or sooner if we have more information.

    Investigating