Merrilee Proffitt had to announce to our partners a potential slide of one week due to data loss. This was pretty embarassing, and we already had a librarian comment back that weekly backups are too far apart. Well, yes. So, in one of my less fun compositional efforts, i wrote the following to be shared.
The RedLightGreen project has hit a series of cascading decisions that meant moving faster than with our usual caution. Unfortunately, we had to switch to the latest version of Websphere 5 in order to use the latest version of DB2. This was a surprise, as IBM notes that the DB2 clients are one version forward- and back-compatible. However, in the step between versions 7 and 8, the behavior for communicating between 32 and 64 bit architectures has been added/changed and is not back-compatible. This meant rapidly deploying to a version of Websphere that supports the version 8 client -- and other applications in our usual development environment were not ready for this promotion. We installed this new version of Websphere on an machine usually used for network monitoring. It will be both the development and production machine while the other applications are migrated to the new version. I did not remember to ask our operations staff to back up this machine as if it were a development machine. Thus, the conditions were set for our data loss.
I reply because I want to assure you that in normal operating and development situations RLG does conform to standard industry practices. In this situation, we were doing too many things at once. Even so, I'm proud of the work our folks have done in the strain of upgrading to two new application systems. I was warned at the onset of the database upgrade that there would be hiccups. Nontheless, the performance enhancements of going to the next version were well worth the struggle.