A Rare Look Inside the Q9 Data Centre

Editor’s note: Heather is a Customer Communications Representative at Tucows. Her many responsibilities include handling customer and emergency communications, drafting incident reports and taking care of special technical communications. If you’re a Tucows customer, Heather plays an integral role in keeping you informed about the status of Tucows various services.

Last week, I was given the opportunity to visit Q9, one of Tucows’ data centres. Every day I write about our products and service in status updates, maintenance windows and customer reports. Tucows infrastructure hardware — including our servers, routers, storage device and data centres — have become so familiar in my day-to-day tasks that I find myself thinking about them like actual live staff members.

q9_logo.gifSeeing the strength of the Tucows network, from the fibre drop to the servers, is a rare treat. Tours of Q9 are infrequent for security and logistical reasons. Access to the site is strictly controlled with only highly technical staff members granted security clearance to the data centre and our cages. Our Network Operations Centre (NOC) and Operations teams at Tucows headquarters conduct monitoring with a number of sophisticated remote systems. We also have staff who stay onsite at each of our data centres to coordinate with multiple sites and the team back at headquarters.

Q9’s data centre is built with the highest caliber of technical security: bullet resistant glass and biometric entrances with access cards are among the authentication and verification steps that must be complete before entering the areas of the cages. Q9 has strict protocols for site usage and visits. Surveillance includes cameras covering every angle and extensive onsite and offsite monitoring. There are mantraps that use biometrics to protect access to designated areas. The chamber has a biometric fingerprint unit that (whether imagined or not) felt like a current went through me when I touched it for validation scanning. Let’s face it: my inner geek loved the chance to see the hardware of the layer 3!

One of our many racks at Q9Our cage is large, with rows and racks of servers, massive storage devices, routers and cables. We use the best of the best equipment, including NetApp, for storage devices. Being a trained Library and Information Technician, I was extremely impressed with the labeling of every device with Dewy-Decimal-like meticulousness. Every machine has redundant power backups for testing and security. The hum and beeps of all the machines represent millions of domains, mailboxes, products and services.

Customers depend on security, reliability, accuracy and professionalism. Every day at work our teams ensure that every aspect, from security to equipment organization to functionality to maintenance, is seamlessly conducted. Q9 helps us out with multiple service points, generators and a contract of 100% service availability. When the blackout struck the Eastern Seaboard circuits in August 2003, Q9 remained online with their downtown NOC fully redundant and emergency plans enacted. Last year we moved to Q9 because of the reliable and consistent service it enables Tucows to pass onto its customers.

Tucows employees love the Internet. We are passionate about our work and almost daily there are internal challenges to see who has the best new tech toy or website reference. The abstract part about working in the Internet industry is that unless you are logged into the physical devices or visiting the data centre cages, you reside in the top layer and your own imagination. Now when I write about our devices I will have a much better visual of some of our infrastructure. Unlike a former coworker, I did not hug the machines! Instead, I just smiled and secretly waved.

Service Outage Update

Update Wednesday, November 21, 2007 09:54a.m. ET: Parked Pages has been online since the early morning. We are happy to report that all services are online. Thank you for your patience.


Update at 4:50 p.m. ET:The operations team report that Parked Pages is now intermittently unavailable; they continue to investigate this. We can, however, advise that all other services remain fully online. We’ll keep you updated via System Status regarding Parked Pages.


Update at 1:50 p.m. ET: Our Operations team has got the traffic hitting our load balancers under control. We’re back online and service has been restored to OpenSRS Domain Provisioning and Management, OpenHRS, CGPRO OpenSRS Email, Blogware, Website Builder, Digital Certificates, Managed DNS Service and WHOIS. Parked Pages remains offline. We’re continuing to monitor our services, analyze and block traffic as required. We’ll keep you updated via System Status and here on the blog.


UPDATE at 12:57 p.m. ET: Our network operations team has established that our load balancers are experiencing significantly higher than normal traffic levels. Working with our data colocation vendor, the team is currently searching for the source of this traffic. We have put some blocks in place and are continuing to analyze and monitor this traffic so that we can get our services back online.


Unfortunately, a number of our services are offline including: OpenSRS Domain Provisioning and Management, OpenHRS, CGPRO OpenSRS Email, Blogware, Website Builder, Digital Certificates, Managed DNS Service, Parked Pages, and WHOIS.

Services not affected are the Email Defense Service, the new Tucows Email Service and our ticketing system.

Domain names continue to resolve and all email messages are being queued. No mail will be lost - we’ll deliver it as soon as we resolve this issue.

While we are experiencing higher than normal call volumes, our Support teams are available to take your calls and emails. The ticket number for this issue is 10794.

As our Operations team works to resolve this issue, we’ll keep our System Status page updated, and also keep you posted with updates here on our blog.

We apologize for any inconvenience this outage is causing to you and your customers.