Alerts and Maintenance
09-30-2016 We have been notified by Canwood property management that during October 2-3 2016 there will be concrete repairs around one of the other buildings on the property (29229 Canwood). There will be no effect on the datacenter operations but please use caution in the parking lot area.
09-26-2016 We have been notified by SCE that there will be a 24-hour outage affecting the Canwood datacenter on October 14, starting 5pm through Octoer 15, 5pm. We will switch to generator power for the duration of the outage and we have scheduled fuel deliveries to ensure no interruption of service. You can find information from SCE at https://www.sce.com/wps/portal/home/outage-center/check-outage-status/?ecid=web_quicklink-home_outage, referencing the SCE outage number.
08-26-2016 2:00pm all repairs have been completed by AT&T and main phone services are restored.
08-23-2016 5:00pm PDT Widespread phone service outage; datacenter unaffected. Due to a road-works construction accident a significant number of phone lines in the Agoura Hills area were damaged. There is no effect on our datacenter operations.
06-11-16 07:00am PDT All systems transitioned to utility power cleanly. Everything appears healthy. We will continue to monitor as normal.
06-11-16 06:55am PDT Utility power has just been restored. We are monitoring systems for proper transition off generator power.
06-11-16 06:10am PDT The generator is being refueled in case the outage is extended again. All systems are still working normally.
06-11-16 04:00am PDT SCEdison has extended the outage to 7am.
06-10-16 09;21pm PDT Utility power was shut down. Canwood main generator activated normally. All systems are WORKING WELL. Our team is on-site monitoring all aspects of the datacenter. If you encounter any problems please call our Support line at 818-735-3000. Thank you!
05-08-16 05;30pm PDT ALL power is back ON at Canwood datacenter, all systems are functioning correctly. Please call Support at 818-735-3000 if you are having any ongoing problems. Thank you for your cooperation and once again, our apologies for the outage.
05-08-16 03:10pm PDT Power is back ON at Canwood datacenter except for a single subpanel which is still being worked on. Cabinets connected to that subpanel are still down due to a failed main breaker in the panel. Engineers are working on a repair strategy.
05-08-16 02:30pm PDT Diagnosis of power issue at Canwood datacenter ongoing. Engineers have been onsite for the last several hours working on pinpointing the failed components. We apologize for this outage and will update when new information is available
05-08-16 07:30am PDT We are experiencing a power failure at the Canwood data center. Engineers are onsite working on the issue. More updates will be posted here as they come in.
04-05 16 10:17am PDT Our Support Team has contacted all clients that contacted us and all are reporting service is restored. If you are currently experiencing any issues please contact our Support Team at 818-735-3000 for immediate assistance. Our apologies for any impact to you. Thank you.
04-05 16 09:48am PDT Time Warner is reporting a core router failure in the Los Angeles (Compton) area. We have terminated our connection to Time Warner and all traffic has been rerouted to our alternate uplink connections.
04-05 16 09:36am PDT We are experiencing packet loss for traffic originating or traversing the Time Warner network destined for our datacenters. We are taking immediate steps to route around the outage/issue.
03-20-16 12:45am PDT Planned network maintenance completed. All systems appear to be healthy. We will continue to monitor for anomalies.
03-19-16 11:00pm PDT Planned network maintenance started.
02-26-16 02:23pm Services are restored for all affected clients. A failed optical cable assembly was replaced and the circuit was restored successfully. Our apologies for the downtime. If you have any continuing issues please contact our Support department.
02-26-16 1:30pm Our technician has arrived at the datacenter and is working with the facility engineer to diagnose and repair.
02-26-16 12:18pm Diagnosis indicates likely a failed optical connector. A technician has been dispatched to replace it, ETA is 1:30pm with a repair time estimated at 15 mins.
02-26-16 11:55am Our connectivity provider California Internet is experiencing an outage. A subset of clients using their service to connect to our datacenter will be down, all other clients are unaffected. We are working with CA Internet tech support right now to resolve the issue. Check back on this site for further updates.
02-09-16 02:45am Connectivity to the T1/EoC border router was restored. All services are working as expected. Please contact KeyInfo support if you are a client experiencing any issues. Our apologies for the downtime.
02-09-16 12:15am a border router servicing a subset of T1 clients has lost connectivity to our backbone. Tentative diagnosis is a failed fiber optic interface. A technician has been dispatched to the datacenter to diagnose and replace if necessary. Expected repair is estimated by 3am.
We have a planned network maintenance window scheduled for Feb. 27, 2016. A notice will be emailed to all clients with details approx. 1 week prior to the window.
12-01-15 12:02pm Cogent is experiencing packet loss and high latency upstream from our datacenters. We have taken routing actions to avoid the Cogent network for outbound traffic and have an open trouble ticket with Cogent in-progress. This is not related to a DDoS attack on our datacenters or clients.
11-17-15 9:40am We have confirmed a DDoS attack and it is actively being mitigated. The attack has ceased but we we continue to monitor closely in conjunction with our DDoS service team. All services have returned to normal.
11-17-15 8:55am we experienced 27Gbps burst of traffic for 12 minutes. Our DDoS mitigation service did not initially identify this as an attack. We are investigating the situation and will provide more information as soon as we have it. Check back here for updates.
08-23-15 11:31am: Our border router servicing Covad T1, Ethernet over Copper (EoC) and DSL clients has rebooted twice after the firmware upgrades completed last night. We are currently diagnosing the issue and working on stabilizing the router. Our apologies for the repeated outages. We have set the router to boot into the older version of firmware should it reboot again. Only clients directly connected to this router are affected by this issue. All other upgraded devices are stable and operating properly. Further updates will be posted here as they occur.
08-23-15 01:15am: Our network maintenance was completed successfully. All devices were upgraded. We will continue to monitor systems for any problems.