Downtime & Outages

[Ralle]

Here will be posted when the site suffers from downtime or to announce if there is maintenance scheduled.

 

[Ralle]

2012-08-06 18:56:20 - 2012-08-06 20:16:20 (GMT)
Unknown network issues resulted in the site being inaccessible. The server was rebooted which fixed the issue.

 

[Ralle]

2012-08-17 06:36:20 - 2012-08-17 07:24:20 (GMT)
The server crashed. We've had this happen before. It is likely due to the virtualization software running on the system.

 

[Ralle]

2012-08-19 20:29:21 - 2012-08-19 21:38:20 (GMT)
There was a network issue in the datacenter. I have not been told what the cause was.
However, I have requested a new server, which will be set up within a month so Hive will migrate over there and not use virtualization from then on. This will help stability as we've had many issues with that.

 

[Ralle]

2012-09-06 12:28:21 - 2012-09-06 15:29:20 (GMT)
The server crashed and I was at school. It took me over an hour to look at my phone and notice the site being down. I immediately emailed support and it took some time to get things right.
On Tuesday I am ordering a new server.

 

[Ralle]

2012-10-10 03:00:00 - 2012-10-10 06:22:58 (GMT)
The site was inaccessible. I contacted support and they updated the network driver.

 

[Ralle]

2012-03-03
The site is being moved in 4-5 hours onto a different server. This one is bigger and faster.

UPDATE:
2012-03-04 10:50:00 (GMT)
The site is now moved to the new server. Please report any problems in the bug report forum.

 

[Ralle]

2013-03-11 08:02:58 - 2013-03-11 08:25:58 (GMT)
Software updates were applied which turned out to require a reboot.

 

[Ralle]

2013-03-23 10:09:58 - 2013-03-23 12:46:58 (GMT)
The host server (the one that hosts Hive's VM) was rebooted to optimize how the network is routed. An insignificant partition was not being able to be mounted on the system which caused it to not boot.

 

[Ralle]

2013-05-24 19:37:58 - 2013-05-24 20:13:58 (GMT)
The server was down and I requested for support to take a look. It is now up. I don't know what did it.

UPDATE: After coming back it shows that no server was rebooted. Therefore it must have been a network issue.

 

[Ralle]

2013-06-22 07:59:58 - 2013-06-22 09:41:58 (GMT)
Scheduled downtime by our sysadmin. I forgot to post it on the site, sorry about that.

 

[Ralle]

This had a long way coming. I kept forgetting to post it.

2013-07-18 23:21:58 - 2013-07-19 00:03:58 (GMT)
2013-07-18 22:05:00 - 2013-07-18 22:32:58 (GMT)

The server disappeared from the net. We got support to get it back, then it disappeared again. They ended up switching a network switch and it came back to normal.

Now, this weekend we have something else:

I wanted to let you know that I am planning to do some server maintenance this coming weekend starting at 2 AM CDT or 7 AM UTC. This is a version upgrade for our hypervisor, so VMs will need to be shutdown while the upgrade takes place. I am scheduling a window of two hours for this upgrade, though hopefully the maintenance will take less time than this window.

 

[Ralle]

2013-08-10 08:03:58 - 2013-08-10 08:51:58 (GMT)
Downtime progressed as planned. Good job Ghan_04 .

 

[Ralle]

2013-10-07 06:08:58 - 2013-10-07 06:30:58 (GMT)
I forgot to add this, but the perfectionist in me required me to add it. This was the server move.

 

[Ralle]

2014-02-01 05:19:59 - 2014-02-01 05:33:58 (GMT)
The server crashed and was rebooted. The database needed a repair but I was asleep for a few hours.

 

[Ralle]

2014-04-11 21:56:58 - 2014-04-11 22:46:00 (GMT)
After upgrading stuff on the server, the web server wouldn't start. Thanks to our dear Samuraid, it was quickly fixed.

 

[Ralle]

2014-04-19 13:03:58 - 2014-04-19 15:50:58 (GMT)
I just got access to the server after some significant downtime. It appears that it has just been turned on. I will edit this post when I find out more.

EDIT: Just had the web server down for a few minutes to do a database repair. Some tables were not in a good state.

EDIT: I have been in touch with Ghan_04. Apparently the server rebooted and the virtual machines (Yes, Hive is a virtual machine) were not configured to start up automatically. This is now fixed.

 

[Ralle]

2015-08-22 06:49:58 - 2015-08-22 06:54:58 (gmt)

we had a server pdu failure in the datacenter this evening which caused downtime while the techs fixed the problem.

 

[Ralle]

2015-10-14

dear quadranet customer(s),

planned network maintenance notification

location: Los angeles
start: 10/14/2015 @ 23:59 pdt (gmt-7)
end time: 10/15/2015 @ 02:00 pdt (gmt-7)
expected downtime: None

description:
Our network engineers will be performing maintenance on the router that you are connected to. During this maintenance additional line cards will be added to our routers.

If you experience an outage, have problems with your connectivity after the maintenance window, or if you have any questions regarding the maintenance at any point, please contact our support department at [email protected] or 1-888-5-quadra.

We appreciate your understanding and patience during this maintenance and welcome all feedback.

 

[Ralle]

2015-11-11 14:17:58 - 2015-11-11 14:32:58 (GMT)

Network problems.

 

[Ralle]

Planned maintenance from our host:

Dear QuadraNet Customer(s),

Date: Wednesday, November 18 2015
Start Time: 11:00PM PST
End Time: 03:00AM PST

Summary: The ___ floor distribution router is being replaced.
Possible Impact: You may briefly lose connectivity during this maintenance.

Description:
QuadraNet is replacing the distribution router that provides your service. During this maintenance every effort will be made to maintain connectivity and limit any possible outage to no more than 60 seconds.

 

[Ralle]

2016-04-22 11:45:00 - 2016-04-22 12:02:00 (GMT)

Our host Linode reports network issues:

We're currently investigating reports of poor connectivity and packet loss in Frankfurt.

EDIT: (7 hours later) This ended up being intermittent throughout the day but it seems like it has settled now.

 

[Ralle]

2017-02-13 01:05:00 - 2017-02-13 06:58 (CET)

The database was down, so I put it back up. Nothing in the logs. Will keep an eye on things.

 

[Ralle]

2017-04-07 09:10:44 - 2017-04-07 10:04:15 (GMT)

Moved the site.

 

[Ralle]

2017-05-10 04:56:57 - 2017-05-10 06:03:19 (UTC)

Outage at provider. More info

 

[Ralle]

2018-01-13 00:19:49 - 2018-01-13 06:56:34 (GMT+1)

Database stopped. Restarted it.

 

[Ralle]

2018-02-14 00:14:00 - 2018-02-15 07:00:00 (UTC+1)

The server was migrated to a different plan saving me some money. The plan had the same specs as the former one but might have been hosted on a server with issues. The support team took too long to respond, so I decided to move the site back to a known safe place in the middle of all of it. Everything should be working now.

 

[Ralle]

2018-02-23 19:16:00 - 2018-02-23 21:11:00 (UTC+1)

Connectivity issues affecting our the data center caused the site to be down for a few hours. The connectivity issues have been identified as being the result of a power failure. The problem was for once not our fault. You can read more about it here.

 

[Ralle]

2018-05-10 23:08:38 - 2018-05-11 04:59:55 (UTC)

An unoptimised area of the site was getting hit hard causing the rest of the site to stop responding. I didn't write the code for it and I am working with XenForo to optimise this piece of code. So technically the site was just really active.

Also, I was asleep. If you find the site to be down, please message @StoPCampinGn00b, @Kam, @Archian or @KILLCIDE as they all have my phone number and know to wake me up if the site is down.

 

[Ralle]

2018-06-15 09:43:42 - 2018-06-15 11:15:49 (UTC)

Site was down and I was out visiting people, so I only got to this after more than an hour of downtime. Our host said this:

Hello,

We have detected an issue affecting the physical host your Linode resides on.

We're working to resolve this as quickly as possible and will update this ticket as soon as we have more information for you.

Your patience and understanding is greatly appreciated.

Thanks,
David S.
Linode Support Team

After this the server could not startup by itself and I had to repair the filesystem to make it startup.

 

[Ralle]

2018-07-31 16:48:19 - 2018-07-31 19:56:20 (UTC)

Might have been too many visitors... After a tweak in the configuration things started to run better, but I am somewhat at a loss. Anyway, seems pretty stable...

 

[Ralle]

2021-01-08 23:00:22 - 2021-01-09 07:08:24 (UTC)

The session table was crashed and needed repairing. Not sure why that happened, could be because we are close to the storage limit on this server (it's planned to switch to one with more space). This happened just after I went to bed of course.

 

[Ralle]

2021-01-20 16:48:08 - 2021-01-20 18:21:48 (UTC)

Dear Client,

Thank you for your message. Due to Power Distribution problems at your server location your service was temporarily unavailable. This has now been restored and your device should be now back online. We apologise for the inconvenience this has caused.

 

[Ralle]

Today.

Dear diary. We've been nearly out of disk space on the server and it responded by dragging its feet making it seem like the world was going at a glacial pace. I backed up and removed HIVE 2 and we have 400 GB of free space once more.

 

[Ralle]

Dear diary.

Last night the site went down because a disk not containing the site got full of server lint. I have cleaned it out and the site runs nicely once more.

2021-11-17 23:58:22 UTC+0 - 2021-11-18 05:25:00 UTC+0

 

[Ralle]

The search engine was down the past hour. Issues with Docker, not sure why. A restart kicked the dust out of the system.