Scheduled maintenance is currently in progress. We will provide updates as necessary.
Posted May 02, 2016 - 10:00 UTC
Scheduled
Spring cleaning is well under way here at Packet and it’s time for us to rid our network of some bugs that we’ve uncovered in the past few months. As such, we’ve scheduled a maintenance event that will impact your servers while we work on select upstream network devices.
Maintenance Window
Start Time: Monday, May 2, 2016 at 6:00 AM (EDT)
End Time: Monday, May 2, 2016 at 8:00 AM (EDT)
Location: New York City Metro Datacenter in Parsippany, NJ, US (EWR1)
Impact & Scope of Work
During this window, Packet will upgrade software on switches delivering service to your compute instance(s). Each server instance will be inaccessible on the network for a maximum period of approximately (15) minutes while we reload upstream switch pairs.
We will also suspend server provisioning in all sites during this outage window, as our customer portal and API will experience limited availability.
Please note that server power will not be interrupted; there is no need to log into machines following this event to verify operations or restore services.
Reason for Maintenance
In the past year, we’ve provisioned hundreds of thousands of server instances on the Packet platform. Although nearly all of our provisions succeed (99%+), the most common failures have been network configuration related -- due in part to the sheer volume of network state changes we make as part of our automated provisioning process. More severely, some customers have also seen connectivity issues caused by reloads or failure modes in our top-of-rack switch infrastructure.
We’ve worked closely with our equipment partner, Juniper Networks, on troubleshooting these issues. Our demanding use of these switches - and the bugs we have identified when the switches are pushed to this scale - has led to the development of a new software release. After extensive testing in our Lab, we are now ready to deploy this software across our infrastructure.
While our network architecture offers “n+1” redundancy all the way down to our customer-facing switch infrastructure (a defense against a single switch physically failing) all servers will still be down for a (15) minute period during this maintenance. Given the failure modes we’ve seen on the current switch software, we have decided to carry out cold reboots of redundant switch pairs rather than utilizing our “in service” software update abilities.
Thank you for your support and understanding regarding this issue - we know that maintenances are never fun! However, we’re confident that this work will result in a better performing and more stable experience for your compute instance at Packet.
If you have any questions, or wish to discussion your individual services and redundancy plans, please do not hesitate to contact our support team during business hours.