It seems <b>we had our first unexpected downtime yesterday</b>. We've had some offline times previously but most of them have been for a shorter period of time and on purpose due to maintenance and updates.

Access to our services were failing on <i>Thursday from about 4:00 to 20:30</i> when I finally brought things back online. Those times are on my local EEST+3 timezone, which means from 1:00 to 17:30 UTC and from Wed 18:00 to 10:30 PDT. This is <i>about 16 and half hours of lost time</i>.

Looking through logs, this seems to have been caused by the <i>server running out of memory</i> and subsequently OOM-killing itself to death. I have few things I can do to prevent same problem bringing us down in the future. (Like moving the DB to a different server as the OOM-killer chose the poor DB to die on first round.)

I did notice the problems in the morning but due to unrelated complications (non-project ones) I didn't manage to get the server back online until that evening. I do apologize for this extending our downtime.

<b>Everything should be back to normal now</b> but due let me know if there are still problems around. Thanks and now let's crunch hard to make up for the lost time!