Login.ci.uchicago.edu and monitor.ci.uchicago.edu just had an abrupt reboot. Apparently the power cables were a little to taut when I slid a new server in the rack.I apologize to the interruption.  But machines are online again.

After a long fight, tape backups are now running again. First we had a drive failure. Then after a few weeks of testing, we had a tape jam that caused both the library and the drive to fail. We have verified that the home filesystem has been backed up and both tape drives are functioning normally.

Over the last week you may have noticed extremely poor performance on CI machines, especially with respect to home directories. The short explanation is that we still have some performance tuning to do to the CI home NFS file server to be handle hundreds of active connections. We made some changes a few weeks back, which greatly helped, but there is still some work to be done. I’ll explain at the end, in more detail, the filesystems in use at the CI for those who are curious, but I wanted to first give some pointers of how to get the best performance when using your data and have minimal impact on the other CI users.

Before getting into the tips, I want to remind everyone of the CI Resource wiki. There is a FAQ, explanations of the various resources the CI provides, and the CI policies. Everyone should take a moment to browse through that and know to look there in the future, as we’re continually updating it.

For those looking for quick, general rules of thumb, here they are:

• If you only need to access the data for a short period of time (15 minutes) and you only need access to that data from one node, you’ll get much better performance by copying the data locally and operating on it there and copying it back once you’re done.
• In a cluster, if you’re doing lots of small operations (file creations/deletions, reads, writes, etc), it’s best to stage your data locally, do your operations, and stage it back to a networked filesystem.
• In a cluster, if you need multiple nodes accessing the same data, it’s much better to stage that data to GPFS or PVFS than it is to operate on it in NFS.
• In a cluster, If you need multiple writes to the same data at the same time, you should think about using MPI-IO to allow concurrent writes to the same data without file locking.
• It’s best to not store 100s of items in a single directory on GPFS and PVFS. Instead look to create a deeper organization and less wide

Read the rest of this entry »

The tape backup system is currently offline since the barcode reader had to be cleaned. It is performing an verification audit of its resources which could take some time. No data should be in jeopardy, but no backups or restores can be processed until the audit is completed. If you have any questions please email support at ci.uchicago.edu. Thanks and have a great weekend

At approximately 10:40am central time, we had to reboot the web server since it had become unresponsive. It was back online by approximately 10:43am. We are continuing to monitor its usage to determine what caused the outage.

Because things come in threes, one last weekend announcement. We have spent the last few weeks working to update documentation and help resources as well as some policies. While there is still more to do I wanted to take the time to remind everyone where to find answers to common questions: http://www.ci.uchicago.edu/wiki/bin/view/Resources. And as always, do not forget about the support blog at http://blog.ci.uchicago.edu/support which also has an RSS feed to get your fix.

We are proud to announce that we now provide server side mail filtering. This will allow the mail server to filter your mail, and remove the need for your mail client to be online to do so. With this we also now have vacation message capabilities and filter out SPAM before it hits your inbox. We have written a small tutorial to get you started on using this new, powerful service at http://www.ci.uchicago.edu/wiki/bin/view/Resources/MailFiltering. Please let us know if you have any problems or questions.

We are proud to announce that we have a new database server online. It runs both a MySQL 5.0 server and a PostgreSQL 8.3 server and will replace the server currently running on the web server. This should greatly reduce the load on the web server and provide quicker web responses and at the same time provide much more capable hardware for database operations. Along with the new server there is now a web form to request databases at http://www.ci.uchicago.edu/support/databases. Currently we will only be creating PostgreSQL databases on the new server until we migrate the current MySQL databases from the web server. On March 15th while we are migrating the home filesystems to the new NFS server, we will also be migrating the existing databases from the web server to the new database server. During this time, the MySQL server on the web server will be taken offline. We will make every effort to ensure that existing services that depend on the MySQL databases are updated to use the new server, but please let us know if you have special consisderations. Currently the following web services will need to be updated:

• CI Blog
• Flash Gallery
• AccessGrid Drupal Site

If you have any scripts that point to the web server for database service, you will need to update them to point at db.ci.uchicago.edu. We will also be deploying web interfaces as a management interface for any databases you might have. As always if you have any questions or comments, please email support at ci.uchicago.edu.

There is yet another additional network outage.

On March 4 between 7:00am and 7:30am NSIT will perform software upgrade on a switch in the Research Institutes server room. We expect less than 10 minutes outage.

NOTE: This maintenance is in addition to the previously announced outage at Midnight.

Beginning at approximately 6am on Saturday and continuing through late Sunday evening, there will be service provider maintenance on the I-Wire fiber ring. Due to this work, the connection to the research and education networks (Internet2) will have slight (less than 1 minute) interruptions in service as the work progresses. This is primarily connectivity to other schools, universities and research laboratories. NSIT anticipates there will be about a dozen such events that will occur during the course of the work. Normal commodity Internet traffic should not be impacted, e.g traffic to ISPs, Google, commercial web sites, etc. This will not affect network connectivity within campus or to various directly connected campus remote sites such as the Gleacher Center, Prudential Plaza and Press Langley.