IS INTERNATIONAL EXPERIENCES CONNECTIVITY OUTAGES OVER LAST THREE WEEKS
Internet Solutions (IS) first experienced an outage on three of our five international fibre circuits on Monday the 26th of July. These three 45 Mbps circuits operate between New York City in the US, and Cape Town and Rosebank (2 circuits) in SA respectively. The other two fibre circuits are via the SAT2 fibre and SAT3 fibre cables respectively, and these have not been affected. IS also operates a sixth circuit, via Satellite. IS is adding a sixth fibre circuit on 1 September 2004, which will result in 7 circuits in total.
The total duration of this initial outage was just over three hours and occurred just before 09:00. The outage was as a result of a cable fault on a segment of the cable system in Spain, where this portion of the cable system is managed by the Spanish telco, Telefonica. The net result of this outage was that approximately half of IS' international capacity was unavailable.
On Thursday the 29th of July, at 03:00 in the morning, the same three circuits were taken out of commission for what was to be emergency repair work performed by Telefonica. It was believed after this work that the service returned to stability.
On Wednesday the 11th of August, the same three international fibre circuits failed again just before 09:00 and only returned to operation just before 13:00. It was confirmed this morning in review with Telkom SA that the fault causing this outage occurred in a telco exchange again in Spain, when the backplane of a telco switch failed.
On Thursday 12 August at 13:00, the same three international fibre circuits failed again. At this stage these circuits are still not functional. The feedback at this stage is that there is a cable break in Spain (again) but in a different location than previously experienced.
IS contracted with Telkom SA and AT&T respectively for the provisioning of these three circuits (an STM1). These circuits are mapped across different cable systems managed by different telcos. From South Africa, the circuits operate on the SAT3 undersea fibre cable, and are connected in the Canary Islands with the PenCan6 undersea fibre cable and some terrestrial cable components to Conil on the Spanish mainland, where the last connection en route to the US, to the Columbus 3 undersea fibre cable (crossing the north Atlantic), is made. The Spanish component, where all three outages over the last three weeks have occurred, is under the management of Telefonica. Telkom SA and AT&T have contracted with Telefonica for the connecting service between SAT3 and Columbus 3.
These Telefonica services have been in commission for just under a year, and barring the last three catastrophes, have been exceptionally stable and robust, with these the first critical failures to have occurred on these services.
The outages obviously had, and are still having, a massive impact on services to customers. Though IS still has three other large bandwidth services available, the nature of the outages is such that services to all customers are affected.
During and after the previous outage yesterday, IS had two extensive engagements with firstly executives and secondly a senior technical planning team, from Telkom SA and AT&T. Good progress was made in evaluating potentially weak points in the service and developing alternatives. It is unfortunate that a third outage occurred before we were able to action the changes we agreed on.
IS has assured customers that it "is aggressively pursuing the stabilisation of this service, which involves the mapping of this service onto alternative underlying infrastructure. The focus now is on returning the service to production as soon as possible, after which we will make changes to the underlying infrastructure to ensure a robust service our customers expect from us. We separated the links to build in redundancy, now to be let down on numerous occasions by the international carriers. It is this redundancy that we are continually looking to improve upon".