Start Date: October 9th, 2018 @ Morning EDT
End Date: October 10th, 2018 @ 1:30 AM EDT
Internal Ticket: #40
Location: EWR1
Description: IPv6 Reachability Issues and Emergency DSR Maintenance
Packet experienced loss of IPv6 reachability in its EWR1 (New Jersey) facility from the morning of October 9th, 2018 until October 10th, 2018 around 1:30 AM EST. This impacted customers in EWR1 to varying degrees until it was resolved through a configuration change applied to our routing infrastructure. IPv4 connectivity was not affected during the incident.
During investigation, Packet discovered that IPv6 routes were being dropped incorrectly from the spine layer in our network forwarding table (the spine is a pair of aggregate switches that connect our top of rack switching to our border routers).
To correct the issue, Packet had to modify the Packet Forwarding Engine (pfe) on the affected switches. This modification required traffic to be drained from the redundant switches one at a time and then allowing the switches’ PFE to fully reload. As load was shifted, customers may have seen increased latency as one switch was forced to handle all load.
All times are in EDT.
Monday, October 8, 2018
Tuesday, October 9th, 2018