We've now crunched over 200,000 HADSM models including Classic slabs and several types on the main CPDN project. We've processed over 10,000 HADAM models on the Seasonal Attribution Project and CPDN, and are fast approaching 23,000 HADCM models from BBC and CPDN members. Project totals can be seen on the CPDN index page where the display is now clearer. We can select the type of model we wish to crunch on the project preferences page of our account.

Many thanks to all members for taking such good care of your computers and models. For members not quite certain how to take care of them, Mike has advice and suggestions in the project READMEs here and here.

Several recent presentations by the research team have been added to the publications section of the CPDN website.

Some of the CPDN main project servers are down for maintenance.

http://climateapps2.oucs.ox.ac.uk/cpdnboinc/server_status.php

The CPDN-BOINC forum and our CPDN account and model web pages are not accessible. Trickles and zip file uploads may fail and produce messages like

26/02/2008 00:50:30|climateprediction.net|Message from server: Project is temporarily shut down for maintenance

The best idea is to suspend BOINC network activity until the CPDN servers are up again, though we realise that this is not always possible for multi-project crunchers. The BBC, SAP and Beta2 servers are all functioning normally.

Tolu worked all Monday and a good part of the night on unscheduled server maintenance, mostly moving large volumes of data. All the CPDN servers are again fully functional.

BBC and CPDN crunchers together have now completed over 23,000 HADCM models. Thank you, everyone! The current 80-year HADCMs are 160-year models that didn't complete on the first computer that crunched them. Many are specially selected to be completed on a different computer because they produced interesting results during their first 80 years.

Following the recent CPDN main project server outage there are still delays in successful CPDN trickles appearing on our models' web pages and our credits being reported to the stats sites. We cannot at the moment say exactly how long these delays will last.

These delays do not affect the running of our models.

Thanks to all crunchers for your continued patience.

Following the recent maintenance earlier this week, the trickle, credit and statistics export daemons have now been restarted on the climateprediction.net servers (note that the beta, BBC/CCE and SAP climate projects have not been affected). Accordingly you should see the lagged trickles appear on the website today, credit should appear tomorrow, and the external statistics sites should be updated with the new credit on the day after tomorrow.

CPDN upload server climateapps3 is not running. This means that some computers running CPDN cannot upload trickles and zip files. The best advice is

* For CPDN-only computers, suspend BOINC network activity until the server is running again. Your models can continue to crunch.

* For multi-project computers whose network activity can't be suspended, if possible suspend your HADCM model in your BOINC manager Tasks window before the model reaches its next trickle creation point at the beginning of December or zip file creation point at the beginning of December at the end of a decade. HADSM and HADAM models trickle more frequently and can be suspended immediately. This will help avoid multiple failed upload attempts and messages.

We realise that this is inconvenient for multi-project crunchers.

* Check CPDN server status here.

BBC, SAP and CPDN Beta uploads are not affected.

The CPDN upload servers are all now functional and accepting trickles and zip files. Upload server climateapps3 had filled up so Tolu had to move data to another machine. This other disk is now almost full so Milo will deal with it this week.

So there will probably be at least one further CPDN outage during the next few days. If and when this happens the advice in my last post will again be useful. Some CPDN-only crunchers may prefer to suspend BOINC network activity all week and only enable it for short periods after checking the server status page.

At least none of the servers are broken and there are plans to provide CPDN with much more storage capacity.

Some successful trickles have not yet appeared on our models' web pages which means some members probably still have a credit delay. Many thanks to all CPDN members for your patience.

Milo is moving a lot of data around on the CPDN servers this week and will probably continue during the week beginning Mon 10 March. He hopes to do this without closing any CPDN servers down. Connections to the servers may at times be slow.

Several CPDN members have reported today that their model zip files have failed to upload even though CPDN server status shows as all green ie up. There is still heavy load on the CPDN servers. The best idea is (as usual when this happens) to suspend BOINC network activity if you can and try again a few hours later or tomorrrow. Suspending network activity avoids multiple failed upload attempts and multiple BOINC messages.

We understand that multiproject crunchers may not be able suspend BOINC network activity.

Thanks to all crunchers for your patience.

Several CPDN Beta2 servers are down:

http://cpdnbeta.oerc.ox.ac.uk/server_status.php

Although the Beta2 server status page currently shows the upload server as running, some (perhaps all) zip file uploads are failing.

There will be a planned power outage in the Oxford Atmospheric, Oceanic and Planetary Physics department on Tuesday 18 March from 07.00 UTC for about half a day. The servers will need to be powered down on Monday 17 from about 17.00 UTC. Because Milo will be away on Tuesday, he may not be able to reactivate the servers until Wed 19 at approx 09.00 UTC.

The BBC servers will not be affected.
The SAP project website, forum and some uploads will be affected.
The CPDN website, forum and some computers' uploads (trickles and zip file uploads to upload server uploadatm) will be affected.
CPDN stats exports of credit for trickles received should not be affected.
The CPDN independent forum will not be affected. It's a good idea for all members to register there.

The usual advice for outages applies. CPDN and SAP members should if possible suspend BOINC network activity before the outage begins. Some multi-project crunchers may wish to fetch extra work from other projects before the outage begins so they can also suspend network activity.

The server status of ClimatePrediction projects can be checked here:

CPDN
http://climateapps2.oucs.ox.ac.uk/cpdnboinc/server_status.php
BBC
http://bbc.cpdn.org/server_status.php
SAP
http://attribution.cpdn.org/server_status.php

Update on CPDN HADSM slab models.

A small but significant proportion of these models develop into slow-processing 'iceballs' that also have abnormal graphics. Everyone crunching a HADSM should keep an eye on whether its progress is normal. One can check the graphics display to see whether the globe is still normal, the speed of progress or sec/TS as shown in the graphics window, and trickle frequency. Trickle and sec/TS history are shown on the model's web page. Anyone who thinks their HADSM is not developing normally should look at Geophi's post here. He links to a thread where members can ask for advice.

REMINDER

CPDN models, especially HADSMs, do not like to be interrupted while post-processing data and preparing a file for upload. HADSMs do this at the end of each phase, HADAMs at the end of the whole model, and HADCMs after every 10 years. The graphics display shows when this post-processing is in progress. Avoid exiting from BOINC or suspending models while they are preparing these files and if possible wait until the trickles and files have been uploaded.

The SAP server is down. Tolu is working on it.

The SAP server is now fully functional again.

The SAP server is again down for maintenance.

CPDN model availability

Although the CPDN server status page shows that plenty of seasonal attribution HADAM models are available, these HADAM models in fact come from the Seasonal Attribution Project server which is still down for maintenance. Members who want a HADAM model will have to wait two or three days to get one, or can select a different type in the CPDN preferences of their account.

CPDN upload server climateapps3 is down, which means that some computers won't be able to upload trickles and zip files. At the moment we don't know what has caused the problem.

If you encounter this problem, the best idea is to suspend BOINC network activity in the BOINC manager Activity menu. Multi-project crunchers who can't suspend network activity may prefer to suspend their climate models before they reach the next trickle of zip file creation point until the server problem is resolved. (If you do this, set CPDN to No New Tasks first to avoid getting extra unwanted models.)

CPDN server status page

SAP server

This server is, to quote Milo, in 'a bit of a mess' and needs to have its operating system reinstalled (not by one of CPDN's programmers). Milo says that if this is done later today as planned, the SAP server should be up and running again tomorrow.

SAP members still crunching models may have final zip files waiting in their Transfers window to be uploaded. After the first upload attempt is made, these zip files can safely stay in the Transfers window for up to two weeks. Then they time out. At the moment no-one's SAP zip files should be in danger of timing out.

CPDN servers

Upload server climateapps3 is again up and running, accepting trickles and zip files.

SAP HADAM models will not be available for CPDN crunchers until the SAP server is fixed (see the above post).

Server update

SAP

The SAP server is running again but is under great pressure and may or may not allow trickle and zip file uploads, and may or may not provide HADAM seasonal models for CPDN members who have selected this model option in their CPDN preferences.

At some point this week the SAP server will be shut down for a system upgrade. When this will be Milo cannot say because it is entirely up to the IT person in AOPP (a different Oxford department). He was planning to do it last Friday...

Until the SAP server is back to normal, SAP members may want to suspend their models and crunch something else. CPDN members unable to get a SAP model could consider selecting a different type of model.

CPDN

* There's a big job running on the database server. It has been running since Saturday morning. This is to do with urgent firefighting that relates to processing data for physicists. This may cause slow response times on the CPDN-BOINC website and forum and slow access to CPDN project web pages (eg those showing model details).

* One of the CPDN upload servers, uploadcomlab, is down. Milo shut it down to check what disks are in it (ready for an upgrade) and when he re-connected it one of the power supplies blew. It will return when Milo can order some new disks and also a power supply. So this upload server problem could last for several days.

This means that some CPDN members will not be able to upload trickle and zip files. If you encounter this problem, please consider suspending network activity (BOINC manager Activity menu) to avoid repeated failed upload attempts and multiple BOINC messages. We realise that this may be impossible for multi-project crunchers.

You can check CPDN server status here.

Many thanks to all members for your patience, cooperation and good humour.

With regard to CPDN upload server uploadcomlab, Milo says

Because there are several CPDN upload servers, this uploading problem only affected some CPDN crunchers. If you turned off BOINC network activity because your trickles and zip files couldn't upload, please enable network activity again now.

Milo also says he now expects the SAP server to remain problem-free.

CPDN main project

The HADAM seasonal models now being sent out to CPDN members are a new type. Tolu says

These new HADAM models are named hadam3h_c_*. The previous workunits were called hadam3h_n_* and hadam3h_a_*.

The server only sends HADAM models to computers with at least 1.5Gb RAM. However, that isn't enough for computers with Vista, and dual or quad cores also need more.

Reminder: we can select the type of model we want to download next time by editing the ClimatePrediction preferences in our CPDN account.

CPDN main project

As a result of the CPDN's BOINC server version upgrade, on Friday there was no credits export to the stats sites. We hope Saturday's credits export will take place normally.

CPDN main project

CPDN upload server phkup is down. Milo says

This will affect the upload of some computers' trickles and zip files. If you can, suspend BOINC network activity while the problem lasts.

CPDN main project

The phkup upload server is up and running again, but upload server uploadcomlab is down. Milo is backing up its data today so it can have its new disk installed tomorrow. This means that some different CPDN computers are now unable to upload trickles and zip files. If you find that your computer can't upload to CPDN, if possible please suspend BOINC network activity for the time being.

SAP project

The SAP server has been problematic for several days. SAP crunchers have not been able to upload trickles and files. As SAP models are provided for CPDN crunchers from this server, there may also be difficulty in downloading new SAP HADAM models. Milo will investigate this server tomorrow.