Hi guys,

Having had a few hours of sleep for the first time in days, here is an update for you on our server issues this morning in Europe. It is something that I (Brian) have to handle myself, and the car is about where it can drive itself to the data center.

Rest assured that we are doing everything we can 24/7 to get back up. There is NO risk of data loss - your collections are safe, but the process of checking our disc set takes time.

Status as of now is that we have a secondary server ready with the latest data, but our priority is still to get our primary server up and running, to avoid down-time later on.

We have set the system to operate on the second server in a read-only mode, to ensure we do not have data changes that needs to get back to the primary server, but there are many configuration parts, and we are not done with configuring everything, so there can be issues, also in read-only mode.

We cannot say for sure when the disc check of the primary server finalises. If the process is not soon ending, we will be considering opening full up on the secondary server.

This is absolutely not a normal situation for us - we know how important operation is for all of you, and we are doing all we can - historically, this is by far the biggest issue we have had time wise, but in the end, the safety of your collection, and the data is the top priority, even if it means some down-time.

Timing wise, this is for us also very critical, being more or less ready to release version 6 of our apps on iOS, and (surprise), a big 5.50 update for Android, this is likely going to delay this time. Our team on development is unable to do much without an operating service, and, our team of moderators are sitting on their hands and waiting.

Thank you for trusting us with your collection data, and we hope that you will continue future support, despite issues such as this.

The time frame of what happened:

- On Monday morning, we had our primary webserver crash out of the blue three times in a row with 10 minutes between them. It can seriously slow, and then crashed. Unplugging the network cable and it ran fine. The issue was identified to an internal networking error on the server, which had a DNS mis-configuration.

- During Monday we began to get reports of IO errors when writing to the file system, primarily by our own team handling missing titles. A read-only check of the discs showed some NTFS corruption errors, without a doubt due to the crashes.

- We have our backups in place, and before doing any sort of disc repair, we spend Tuesday on getting a new server up, obtaining new discs, and running a backup restore on that. We switched completely to enterprise SSDs a few years ago, so in case we needed to not only ensure backup, but also operate on this server, we inserted a new SSD disc array.

- Late Tuesday evening we had the server running, and the backup successfully confirmed and restored on the secondary server, and we then took a copy of the files changed from the short time of the backup to the time of the live data, which due to millions and millions of files is also a lengthy process. We could then start the chkdsk process, and this is when you could no longer use the service.

- Yesterday (Wednesday), we unfortunately could see that the process halted half way. It was restarted and is still at this time running.

- Yesterday, we restored the files copied from the time of the backup to current live data, and the data on server 2 is now complete and consistent with the live data.

- Yesterday we could see that our API code would not operate on the second server, which was set up using a newer version of the OS. This was resolved during the night, and we can now host on the secondary server, should we choose to.

- The check disc operation is at this time still in operation, and we have the data in two places, besides on a backup set, so while our down-time for now remains, there is no risk of data loss.

We have changed the DNS pointers, and lowered the time DNS pointers can be cached, as the secondary server won't take the IP of the primary. The primary server should not be holding onto its IP address while running disc check, but some network is holding it reserved for the server (if you are a network guru and know why, please write us).