Network Status

Alerts and Maintenance

11-15-2018 1:29PM PST - All systems are normal, except for the Spectrum Circuit.

The Spectrum circuit continues to show intermittent challenges as they perform repairs on the environment.  We continue to be in communications with their support team, but have been given no ETA at this time to a permanent solution.

Spectrum has provided a few details as follows:

  • They are in the process of repairing over 12 miles of fiber that was completely melted.
  • In some cases, they have had lay fiber directly on the ground as a bypass fix because there are no telephone poles to hang cable on because they were destroyed in the fire.
  • They do expect further interruptions, but are doing everything they can to limit the outages.

Their Twitter site is the only published source of information and this is the relevant thread:

https://twitter.com/Ask_Spectrum/status/1061356644875923456

We will continue to monitor the service until it is repaired.  We will continue to prefer routing of traffic on our other circuits until Spectrum provides confirmation that a permanent solution is in place.

If you need any help at all, including technical support of any kind or assistance in restoring your environment that may have been affected by the fires, please do not hesitate to contact our support team.


11-14-2018 12:24PM PST - All systems are normal, except for the Spectrum Circuit, and no fire activity threatens any operations at this time.  Our offices are now open and all clearance for all road traffic was given yesterday evening.

The Spectrum circuit started to show intermittent challenges as they perform repairs on the environment.  We have been in active communications with their support organization this morning, but have been given no ETA at this time to a permanent solution.

We will continue to prefer routing of traffic on our other circuits until Spectrum provides confirmation that a permanent solution is in place.

If you need any help at all, including technical support of any kind or assistance in restoring your environment that may have been affected by the fires, please do not hesitate to contact our support team.


11-12-2018 7:00PM PST - We continue to monitor the Woolsey Fire and new Santa Ana wind conditions closely.

All systems are normal and no fire activity threatens any operations at this time.  Our offices will be closed tomorrow to keep the Agoura Road clear for official fire, technical and other official traffic.

The Spectrum circuit continues to perform without service disruption.  We will continue to prefer routing of traffic on our other circuits until Spectrum provides confirmation that a permanent solution is in place.

If you need any help at all, including technical support of any kind or assistance in restoring your environment that may have been affected by the fires, please do not hesitate to contact our support team.

Lastly, and most importantly, today is Veteran’s Day.  We offer our most sincere gratitude for those who have served our country defending our freedom.  We stand united in support of our troops and the spirit of this great country.


11-11-2018 1:05PM PST - We continue to monitor the Woolsey Fire and new Santa Ana wind conditions closely.

All systems are normal and no fire activity threatens any operations at this time.

The Spectrum circuit continues to perform without service disruption.  It is still possible that there will be future service interruptions over the next few days as crews work to finalize and harden their repairs and we will work with Spectrum to minimize the impact to our clients.  We will continue to prefer routing of traffic on our other circuits until Spectrum provides confirmation that a permanent solution is in place.

As before, and it cannot be repeated enough, we offer our sincere appreciation to the City of Agoura Hills and, most importantly, all of the first responders who continue to work to save lives and defend property across the region.

If you need any help at all, including technical support of any kind or assistance in restoring your environment that may have been affected by the fires, please do not hesitate to contact our support team.

Lief Morin


11-10-2018 10:40PM PST - We are pleased to report that Spectrum has restored service with a temporary fiber patch.  We were concerned that this might lead to flaps of those affected circuits in the first few hours after the repair, but those network circuits have been stable since the patch was applied.

All Spectrum clients (point-to-point and those who have IP addresses assigned from Spectrum) should now have restored access.

It is very possible and likely that there will be future (although probably short) service interruptions over the next few days as crews work to finalize and harden their repairs.

As a result, we will continue to prefer routing of traffic on our other circuits until Spectrum provides confirmation that a permanent solution is in place.

Please don’t hesitate to contact us if we can be of service or support in any way.

Lief


11-10-2018 4:30PM PST

To all of our clients and community:

First and foremost, we send all of our love to those families who have lost loved ones or their homes over the last 72 hours.  It has been one of the most turbulent times in this area in long memory.

We have continued operations throughout the extreme fire event known as the Woolsey Fire.  One of our upstream communications providers, Spectrum, has suffered an outage that is affecting all of Ventura County.  This came as a result of the destruction of one of their fiber hubs in an area known as Bell Canyon.

They have fiber crews ready to engage to repair that fiber hub as soon as the Fire Marshall declares that it is safe to do so and grants access.  At this time there is no ETA and is completely dependent on the conditions on the ground.  Several of our point-to-point clients are affected by this outage as a result.  We will be in regular contact with Spectrum and, more specifically, monitoring the regional events until all operations return to normal.

All other operations regarding our business, facilities and communications are normal and being continuously monitored.  We are grateful to the extended team for their long hours to keep operations during this extraordinary event to continue normal operations.

As before, we offer our sincere appreciation to the City of Agoura Hills and, most importantly, all of the first responders who continue to work to save lives and defend property across the region.

If you need any help at all, including technical support of any kind or assistance in restoring your environment that may have been affected by the fires, please do not hesitate to contact our support team.

Lief Morin, CEO


11-09-2018 5:10PM PST Spectrum/Time Warner/Charter is experiencing an outage in the Simi/Agoura Hills/Thousand Oaks areas.  We have reports of Point-to-Point data circuits, residential internet and cable TV outages.  Spectrum confirms multiple outages as a result of fire.  Our upstream connection to Spectrum is up, but only one route is available indicating routing issues upstream from our datacenter.  We are directing traffic away from this circuit and out our alternate upstream providers. 

On a positive note the fire is not approaching the Agoura Court datacenter.  We have confirmation that the City of Agoura Hills will provide structure defense for the Community Center, Library and the Sheraton Hotel. Our datacenter as well as the AT&T central office on Agoura Road are recognized as critical regional telecommunications infrastructure facilities and will receive similar treatment, should that become necessary. At this time the datacenter remains out of danger.

11-09-2018 11:20AM PST The #woolseyfire continues to burn in surrounding areas but has not affected the Agoura Court (AGC) datacenter and our team continues to monitor the situation closely.  We believe  that the situation will remain stable.  Winds are currently blowing in a southerly direction, past/away from the datacenter.  There are evacuation orders in the area to keep roads clear for emergency vehicles and personnel.  Utility power is on and our emergency backup generators are fully-fueled to bridge any outages.  Our redundant internet connections are in underground ducts, safe from fire damage in surrounding areas.  Please avoid the area for your safety and that of our first-responders.  There are freeway closures and police present. If you need any assistance please contact us.


11-08-2018 10:15PM PST  The Agoura Court(AGC) office will be closed tomorrow Friday Nov 9 due to the #Woolseyfire.  Mandatory evacuations have been issued in the Agoura Hills area primarily to keep roads clear for emergency personnel.  Our Support teams will be monitoring the AGC datacenter remotely and our on-call technicians will be available if needed.  We are monitoring the situation closely but do not anticipate any adverse effects for the datacenter at this time.  Updates will be posted here as the situation changes.  Please call our Support line 818-735-3000 with any urgent questions or concerns, or email our Support mailbox support@keyinfo.com for non-urgent needs. 

History

10-05-2018 12:25am PDT Service was restored and all circuits should be fully functional. This notice will be updated when more information regarding the RFO is received.

10-04-2018 10:47PM PDT Vendor equipment replacement at our Los Angeles datacenter caused an outage to 16 clients using Ethernet-over-Copper type circuits.  We were not notified of any planned maintenance and have a ticket open with the vendor to get a Reason for Outage report.  We apologize for any impact to your business.

09-27-2018 At approximately 1:30am the fiber break was located and repaired, restoring the private fiber ring to fully-redundant operation.

09-26-2018 A network outage has occurred for some clients in the AGC datacenter and the AGC KeyCloud. The outage was at ~4:24PM PDT and lasted 9 minutes. The initiating event was a break in one segment of our private fiber ring, which did not disrupt network operations, but a vendor's repair technician unplugged the incorrect fiber during diagnostic efforts. Our apologies for this outage; a preliminary RFO is available from our Support team or your Account Manager. We are still investigating all the aspects of the outage with Cisco Technical Services due to a failure of backup GRE tunnels to carry the network traffic as designed.  We will update the RFO when the investigation is complete.

09-30-2016 We have been notified by Canwood property management that during October 2-3 2016 there will be concrete repairs around one of the other buildings on the property (29229 Canwood).  There will be no effect on the datacenter operations but please use caution in the parking lot area.

09-26-2016 We have been notified by SCE that there will be a 24-hour outage affecting the Canwood datacenter on October 14, starting 5pm through Octoer 15, 5pm.  We will switch to generator power for the duration of the outage and we have scheduled fuel deliveries to ensure no interruption of service.  You can find information from SCE at https://www.sce.com/wps/portal/home/outage-center/check-outage-status/?ecid=web_quicklink-home_outage, referencing the SCE outage number.

08-26-2016 2:00pm all repairs have been completed by AT&T and main phone services are restored.

08-23-2016 5:00pm PDT Widespread phone service outage; datacenter unaffected. Due to a road-works construction accident a significant number of phone lines in the Agoura Hills area were damaged. There is no effect on our datacenter operations.

06-11-16 07:00am PDT All systems transitioned to utility power cleanly. Everything appears healthy. We will continue to monitor as normal.

06-11-16 06:55am PDT Utility power has just been restored. We are monitoring systems for proper transition off generator power.

06-11-16 06:10am PDT The generator is being refueled in case the outage is extended again. All systems are still working normally.

06-11-16 04:00am PDT SCEdison has extended the outage to 7am.

06-10-16 09;21pm PDT Utility power was shut down. Canwood main generator activated normally. All systems are WORKING WELL. Our team is on-site monitoring all aspects of the datacenter. If you encounter any problems please call our Support line at 818-735-3000. Thank you!

05-08-16 05;30pm PDT ALL power is back ON at Canwood datacenter, all systems are functioning correctly. Please call Support at 818-735-3000 if you are having any ongoing problems. Thank you for your cooperation and once again, our apologies for the outage.

05-08-16 03:10pm PDT Power is back ON at Canwood datacenter except for a single subpanel which is still being worked on. Cabinets connected to that subpanel are still down due to a failed main breaker in the panel. Engineers are working on a repair strategy.

05-08-16 02:30pm PDT Diagnosis of power issue at Canwood datacenter ongoing. Engineers have been onsite for the last several hours working on pinpointing the failed components. We apologize for this outage and will update when new information is available

05-08-16 07:30am PDT We are experiencing a power failure at the Canwood data center. Engineers are onsite working on the issue. More updates will be posted here as they come in.

04-05 16 10:17am PDT Our Support Team has contacted all clients that contacted us and all are reporting service is restored. If you are currently experiencing any issues please contact our Support Team at 818-735-3000 for immediate assistance. Our apologies for any impact to you. Thank you.

04-05 16 09:48am PDT Time Warner is reporting a core router failure in the Los Angeles (Compton) area. We have terminated our connection to Time Warner and all traffic has been rerouted to our alternate uplink connections.

04-05 16 09:36am PDT We are experiencing packet loss for traffic originating or traversing the Time Warner network destined for our datacenters. We are taking immediate steps to route around the outage/issue.

03-20-16 12:45am PDT Planned network maintenance completed. All systems appear to be healthy. We will continue to monitor for anomalies.

03-19-16 11:00pm PDT Planned network maintenance started.

02-26-16 02:23pm Services are restored for all affected clients. A failed optical cable assembly was replaced and the circuit was restored successfully. Our apologies for the downtime. If you have any continuing issues please contact our Support department.

02-26-16 1:30pm Our technician has arrived at the datacenter and is working with the facility engineer to diagnose and repair.

02-26-16 12:18pm Diagnosis indicates likely a failed optical connector. A technician has been dispatched to replace it, ETA is 1:30pm with a repair time estimated at 15 mins.

02-26-16 11:55am Our connectivity provider California Internet is experiencing an outage. A subset of clients using their service to connect to our datacenter will be down, all other clients are unaffected. We are working with CA Internet tech support right now to resolve the issue. Check back on this site for further updates.

02-09-16 02:45am Connectivity to the T1/EoC border router was restored. All services are working as expected. Please contact KeyInfo support if you are a client experiencing any issues. Our apologies for the downtime.

02-09-16 12:15am a border router servicing a subset of T1 clients has lost connectivity to our backbone. Tentative diagnosis is a failed fiber optic interface. A technician has been dispatched to the datacenter to diagnose and replace if necessary. Expected repair is estimated by 3am.

We have a planned network maintenance window scheduled for Feb. 27, 2016. A notice will be emailed to all clients with details approx. 1 week prior to the window.

12-01-15 12:02pm Cogent is experiencing packet loss and high latency upstream from our datacenters. We have taken routing actions to avoid the Cogent network for outbound traffic and have an open trouble ticket with Cogent in-progress. This is not related to a DDoS attack on our datacenters or clients.

11-17-15 9:40am We have confirmed a DDoS attack and it is actively being mitigated. The attack has ceased but we we continue to monitor closely in conjunction with our DDoS service team. All services have returned to normal.

11-17-15 8:55am we experienced 27Gbps burst of traffic for 12 minutes. Our DDoS mitigation service did not initially identify this as an attack. We are investigating the situation and will provide more information as soon as we have it. Check back here for updates.

08-23-15 11:31am: Our border router servicing Covad T1, Ethernet over Copper (EoC) and DSL clients has rebooted twice after the firmware upgrades completed last night. We are currently diagnosing the issue and working on stabilizing the router. Our apologies for the repeated outages. We have set the router to boot into the older version of firmware should it reboot again. Only clients directly connected to this router are affected by this issue. All other upgraded devices are stable and operating properly. Further updates will be posted here as they occur.

08-23-15 01:15am: Our network maintenance was completed successfully. All devices were upgraded. We will continue to monitor systems for any problems.

Customer Support

Tech Support
Toll Free : 877-735-1500
Local : 818-735-3000
For non-emergencies only
support@keyinfo.com

Accounting
billing@keyinfo.com

Sales
Toll Free : 877-442-3249
Local : 818-992-8950
sales@keyinfo.com

Documents
Acceptable Use Policy
SLA
Privacy Policy