Wikipedia experienced some downtime starting 5pm Eastern Time on Saturday. The problem has been attributed to a central image server that seems to be having NFS and Squid related difficulties, and the Wikipedia developers gave more of an insight to the problem in their IRC channel on freenode, and at the time of this writing are hard at work correcting the problem.

Starting at about 5pm Eastern time, every Wikipedia page was replaced with an error mostly depicting “ERR_CANNOT_FORWARD, errno (11)”, a squid error also indicated by the sq* subdomain generating the error. Wikipedia developers said the error is caused by a faulty image server, which has causing problems ever since an upgrade was made to it. Shortly after the downtime, the IRC channel linked to in the error page was flooded with chaotic messages by users already missing wikipedia.

Wikipedia relies on NFS and Squid to distribute the website content and cache it to deal with many requests, along with PHP/FastCGI driving the web application behind it all. The problem is mostly due to NFS errors with the image server (thats .jpg, not HDD images), which causes squid to halt processing. While we don’t have specific details of the upgrade made to the image server and related software, it has been causing problems ever since, according to developers, and this is the peak of the issues.

Other sources claim there is also an SQL problem, as some error messages also show query errors as well as SQL server connection errors, which may also be symptoms of the same NFS-related issue at hand.

The downtime is being worked on, but nobody at the moment is sure how long the downtime can be expected to last. In the meantime, static.wikipedia.org and answers.com will have to suffice for last minute reports needing research for the hundreds thousands that rely on Wikipedia as a source of information.