By Tay Kratzer, Lead Developer for GWAVA Reload

Reload is a GroupWise Hot Backup, Quick Restore, and Disaster Recovery solution for GroupWise post offices and domains. Reload, as a Novell GroupWise backup solution, backs up GroupWise post offices and domains on the NetWare, Linux^{Wikipedia.">Wikipedia." class="glossary-indicator">i} or Windows platform and post offices and domains on any of these platforms that are clustered.

(WARNING: As you read this article you'll probably feel validated in your struggles in trying to manage your GroupWise backup and restore procedures. So print out this article, and have some tissues close by as you read it. You are likely to laugh or cry as you do so!)

As the lead developer for Reload, let me explain in my own words the goal and story behind Reload.

The Problems with Restoring from Backup
Many times as a GroupWise specialist at Novell I was asked to assist in getting GroupWise user objects or a GroupWise message store back from backup. In fact, I actually wrote a chapter in the Novell Press GroupWise Administrator's guide specifically on the subject. (See Chapter 28 of the Novell Press GroupWise 6.5 or GroupWise 7 Administrator's guide, or read it online: Novell's GroupWise 6.5 Administrator's Guide - Chapter 28 - Restoring Deleted GroupWise Users)

Although the methods explained in that chapter explain how to restore a user and their e-mail, the problem still remained that actually restoring a GroupWise post office from tape is such a time-consuming process. The problem is further compounded by the fact that GroupWise message stores continue to swell in size.

There are several good explanations for why the GroupWise message stores are swelling:

• Increased bandwidth options are rapidly increasing, and so more solutions are being digitized, and then transported through e-mail
• We live in an "Information Age", and GroupWise is the transport for much of the information in an organization
• The GroupWise client, and third-party offerings for GroupWise, continues to grow in sophistication and features
• Increased Bandwidth and storage solutions (SANs and Clusters), are allowing GroupWise customers to consolidate into much larger GroupWise post offices

As an example of the kind of message store growth we are talking about, the other day I was working with one of my customers who has had Reload in place for about a year. They thought that Reload was malfunctioning somehow, because the Reload full backup sets for one of their GroupWise post offices were well over 200 gigabytes. This didn't seem right as their GroupWise post office was only about 150 gigabytes . . . or so they thought.

The last time they calculated the disk space of this particular GroupWise post office was about 6 months ago. At that time it was indeed 150 gigabtyes. When we investigated the size of their GroupWise post office, the contents of the OFFILES directory alone was 212 Gigabytes. When my customer saw this, their jaw dropped! In my customer's final analysis, they realized that their GroupWise post office message store size had grown by over 50% in 6 months.

The Reload Story - The Fateful Three Days
Background
At Novell I was assigned to provide GroupWise support to a few of Novell's larger customers. I played this role for several years, and the story that I'm about to tell was one I saw over and over again with my customers. Furthermore, I could see a steadily increasing trend in the scenarios similar to the one explained in this story.

For simplicity, I'm going to refer to the organization I am talking about as Company ABC. At Company ABC, the IT staff had consolidated many of the departments into large post offices, on NetWare clusters. And why not?

Here are the technology improvements that allow for this:

• Increased bandwidth abilities
• Caching-mode client
• Server side GWCHECK
• Multithreaded GWCHECK
• 10 times as many Message Databases (in GroupWise 7)

I even remember when a GroupWise product manager was espousing 10,000 user post offices (as long as users were in caching mode). I must take responsibility for my customer's actions also; I too was beating the post office consolidation drum, the notion of a 1000 user post office was totally feasible, and I had led Company ABC in this direction.

Naturally once their post offices were consolidated, Company ABC needed to do a better job of managing disk space, so that their post offices didn't exceed the capacity of their SANs. It seemed as though they were always riding the edge of their disk space. And now that these larger post offices housed users with all kinds of job descriptions, creating a cookie cutter mailbox size restriction was not feasible. So they GroupWise Administrator decided to put a reasonably high mailbox restriction on all users, so that at least they might see the "Mailbox Size xx%" value that is displayed in the GroupWise client when mailbox restrictions are enabled. Then users would be able to pull up that slick wizard that helps them to know where the space in their mailbox is being used.

Sure enough, just enabling this feature helped. Users became aware of the disk space being used in their mailbox, and they started to manage their mailboxes a little bit better. Now the post offices were not riding so close to the edge of the limits of their disk space.

Crisis

One day, a user who worked in the Purchasing Department that we'll identify as "The Purchasing Agent", pulled up the slick "Mailbox Size" tool. Somehow, the user totally misunderstood this feature, and WHACKED THE ENTIRE CONTENTS OF THEIR MAILBOX.

( . . . . Are you laughing, or crying?)

When the user realized what they did, of course they contacted their GroupWise
Administrator. Now remember this is The Purchasing Agent we are talking about, we're talking about legal contracts, bids, stuff that affects the bottom line at Company ABC.

You know the drill, or for those of you lucky enough to have a department specialist that handles backups, let me explain. Company ABC is a big outfit, with numbers of users in the tens of thousands. So when the GroupWise Administrator needs data from backup, it's a cross-departmental function. The Backup Administrator was using a very sophisticated tape backup solution from IBM called Tivoli. As I explain this process, I am in no way criticizing Tivoli, as all tape backup solutions have the same constraints. May I also be quick to add, that even with Reload, tape backup solutions are still a necessary component.

Recovery from Tape Backup
So here's the drill with this tape backup solution:

1. Go into the Tivoli software, indicate that you want a data set back from the most recent backup, let's say it was Wednesday night
2. With its mechanical arm system, Tivoli then pulls the tape(s) that contain last full backup, which in Company ABC's scenario, was on Sunday.
3. Tivoli commits the data from tape(s) to the hard drive of the Unix server^{Wikipedia.">Wikipedia." class="glossary-indicator">i} that is running Tivoli. We'll call this the "restore data set".
4. With its mechanical arm system, Tivoli then pulls the tapes that contain EACH incremental backup since Sunday, and adds this data to the growing restore data set.
5. In total, approximately 150 gigabytes is restored from several tapes. Fortunately, because of the high end Tivoli system they are using, no one needs to feed tapes into the tape drive. However, the restoration process must share time with the backup process that happens each night. And so the total time to recover is about 1 1/2 days. Now, we have 150 gigabytes on the Tivoli Unix server, now the Tivoli system needs to move the data to some place.
6. A Tivoli Agent must be running on the server where the data will be moved to. Naturally, the data needs to be moved to a NetWare server.
7. Now the GroupWise Administrator is scrambling to find disk space on a server that also houses a Tivoli Agent. Sufficient space is not readily available, so the GroupWise Administrator and the IT Administration group are scrambling to find a NetWare server with close to enough space, and then they hurriedly look for data they can delete to make 150 gigabytes of disk space available.
8. After getting sufficient disk space, the Tivoli server is instructed to send the data set to the Tivoli Agent on the NetWare server with sufficient disk space. The speed between the Tivoli Server and the NetWare server is fast, but transferring 150 Gigabytes takes several more hours to transfer from the Tivoli server to the NetWare server.
9. Approximately two days have passed, and we are into the third day. On the third day the GroupWise Administrator finds that not all of the files that should be at the post office got backed up, because of open file issues with the backup software, and NetWare.
10. The GroupWise administrator then follows the methods of getting the post office to accept a direct connection to the post office, so they can get into the user's GroupWise mailbox with a UNC connection to the post office.
11. Finally they were able to archive the message items off. Of course some of the stuff could not
    be restored, because a few of the message databases were not retrievable. So the end result was that after 3 days passed, about 80 % of the user's mailbox was restored. Fortunately the end user was very nice about the issue, and the IT staff didn't get a lot of grief over the issue.

But what about next time?

Shortly after this event, I had a meeting with the boss of the GroupWise Administrator. The conversation went something like this. "Tay, thanks for your help during the three-day process of getting this person's e-mail back." But then he explained that this same post office that The Purchasing Agent was on, also housed the CIO's mailbox.
What if the CIO were to ask for one e-mail to be restored from yesterday? Were they going to have tell the CIO that it would take three days to restore one simple e-mail? If it takes three days to restore and e-mail from yesterday, then it isn't an e-mail from yesterday, it's an e-mail from four days ago! He saw that the likelihood for this three-day nightmare was only going to increase. Here's why:

• Some of the users have data that is vital to the organization, which if deleted must be restored, no questions asked.
• Many users who were accustomed in the past to getting their mail back when they were on smaller post offices were going to expect the same service levels they had before their mailbox was moved to a consolidated post office.
• E-mail is quickly becoming people's information filing cabinet. Although most customers aren't using GroupWise Document Management per se, still because of enhanced collaboration in GroupWise with things such as shared folders, GroupWise is becoming a document management system in its own right. An GroupWise Administrator can try to fight this fact, with technologies such as Novell iFolder for example, but the reality is that iFolder is not nearly as collaborative as GroupWise and users will use GroupWise as their data repository.

The conversation proceeded with the GroupWise Administrator and her boss, and proceeded into an exploration of what technology might solve the customer's problem.

Here are the technologies that often come to mind:

SNAPSHOT Technology

Pros

• The SNAPSHOT is relatively easy to access; most likely within a 15 to 30 minute period of time a GroupWise client could be accessing a SNAPSHOT. 15 to 30 minutes is perfectly acceptable, and much better than 3 days.
• Some SNAPSHOT technologies do integrate with the GWTSA functionality, and thus provide critical features such as SmartPurge technology that assures that no message items are purgable until they are backed up. As such, no e-mail communication is lost.

Cons

• A SNAPSHOT is 100% of the size of the post office. Although 25% of the time the most recent night's backups are what's needed, a remaining 74% of the time backups from the last two or three weeks are needed. Finally 1% of the time data is needed from say 6 months ago. So if a post office is 150 Gigabytes, to keep it around for three weeks would cost about 3 Terabytes on a SAN.
• If you can afford the SAN space for the SNAPSHOTS, how about the Tape drive space? Each nightly backup will cost on tape the same amount as 100% of the size of the post office. Furthermore; the equivalent of a full backup needs to traverse the network to the backup server each night, instead of once a week. If you access data on the SNAPSHOT server in order to archive it off for example, the data is deleteable. So if the data isn't committed to tape immediately; then there isn't a 100% assurance that all of the e-mail is backed up.

DBCOPY/GWBACKUP to a Secondary Server or a SAN

Batch jobs running DBCOPY and GWBACKUP solutions have the same pros and cons of the SNAPSHOT technology. However, they are inferior to SNAPSHOT technology in that they do not integrate with the GWTSA technology. For GroupWise systems that are relatively small in size however, a batch solution may be sufficient for their backup needs. For this customer though, such a solution was insufficient and lacking certain features.

It was going through this analysis process and realizing that no solution existed that the epiphany for Reload came to me.

Reload - the Answer

First off, Reload is the only backup solution for GroupWise that is designed around GroupWise’s uniquely efficient architecture. Since Reload is designed by GroupWise specialists, it is far more efficient than other backup solutions. Furthermore, Reload is unique when compared to most backup solutions, in that it is designed to make backed-up GroupWise data accessible for restore purposes in a matter of two minutes.

Reload is a solution that allows an administrator to make a GroupWise post office accessible via a TCP/IP connection. No rights need to be granted, no need to set up a Restore area, etc. in ConsoleOne. The user uses their native GroupWise client, without any special client plug-ins, to access their e-mail.

Furthermore, Reload only needs to pull 12% of the size of a GroupWise post office (on average) in order to represent 100% of the post office. It's this kind of efficiency that allows the State of Utah to be backing up 6 post offices ranging in size from 50 Gigabytes to 250 gigabytes in size, to one Reload server, retaining two weeks worth of mail, with about 2.4 Terabytes available on a SAN.

And just as soon as an incremental backup job is run on a Reload post office profile, the backup is immediately usable and accessible. There's no complicated method of combining a full backup set with the incremental backup sets.

Reload at the State of Utah Governor's office
The State of Utah Governor's office is backing their GroupWise post office to a surplus Gateway brand PC with some big hard drives in it. They are retaining three weeks worth of data on their Reload server. The Governor's office is using almost all of the features of Reload. For example, they are using the "Auto-Reload" feature that automatically loads a GroupWise POA against the most recent backup of their post office. Combine this with the fact that they are going to rollout an additional icon with ZENworks that says something like "GroupWise - Backup" and the users will be able to easily self-service themselves into the most recent backup without the need for their GroupWise Administrator to lift a finger.

Reload how GWAVA Reload and Arnold Schwarzenegger Pay(ed) a Visit On The Same Day to the State of Utah Governor's office:

http://www.novell.com/coolsolutions/feature/19170.html

They are also using the advanced Tape Archive feature of Reload, and thus having the Reload server pushing Tape Archives directly to their Veritas server once a week. The net effect is they did not have to buy anything additional such as a Veritas Agent for the Reload server in order to get their backups to tape.

Reload also integrates with the SmartPurge technology in GroupWise, in that messages cannot be purged from the GroupWise post office until they are backed up with Reload. What's particularly unique to Reload is that once a post office is backed up to the Reload server, the message store on the Reload server cannot be purged, ever!

Reload not only acts as a Hot Backup/Quick Restore solution, but through a few menu clicks, the Reload server can actually become the live GroupWise post office. This is particularly attractive to those customers wanting a disaster recovery solution.

To further appeal to customers interested in using Reload for disaster recovery, Reload can even be configured to pull incremental backup jobs intra-day. So it is possible then to have the most recent backup be just a few hours old, rather than a day old.

Dual-site Disaster Recovery with Reload

One customer that has two sites, one in Chicago and another in New York, is designing a disaster recovery solution with Reload as follows:

Both Chicago and New York will have Reload servers, backing up the live GroupWise post office on the Local Area Network. Then Chicago will define a second Reload profile representing New York. However, the Chicago Reload server will be pulling its data from the New York Reload server, and not the live GroupWise post office in New York. In like manner New York will define a second Reload profile representing Chicago. And the New York Reload server will be pulling its data from the Chicago Reload server, and not the live GroupWise post office in Chicago.

The reason the customer is going to use this model is with the following idea in mind. Reload does have GroupWise message store databases open for Write during some very short windows of time. If the WAN connection between New York and Chicago were to go down during a write to a database, then a database may get corrupted. It’s better that this happen to the GroupWise message store on the Reload server, versus the GroupWise message store on the live GroupWise post office.

You may be wondering how Chicago and New York can handle the full backup process each week. Isn’t that a lot of data to be pulling across the wire to generate the full backups? The answer is no. When a Reload creates a Full Backup (also called a “Portable Backup" in Reload), it does not need to pull the post office again across the network as do traditional backup solutions. Reload only pulls the entire post office across the network when it originally establishes the profile. Reload Incremental Backups are what pulls the GroupWise message store data across the wire, a full backup simply capitalizes on the information that has already been pulled across the wire each night when the Incremental Backup jobs were performed on the post offices.

More Cool things about Reload

1. No software runs the servers that are being backed up with Reload
2. Reload is installed with one easy command
3. Creating a Reload profile is a simple wizard-driven process
4. Reload is integrated with GWAVA's Reveal product. As such, it's a very simple to use Reveal and Reload together to do historical auditing of e-mail. These two solutions combined are a boon for Human Resources and Legal Department auditing purposes.
5. Reload is integrated with GWAVA's Redline product, and can be monitored by Redline.
6. The Reload administration menu has a version called "Reload Mobile", so backups can be loaded up via a mobile device such as a BlackBerry device, or any other mobile device with an SSH client.
7. Updating the Reload software is a simple one word command, and Reload updates itself over the wire. The command is: reloadu
8. Reload is Novell YES Tested and Approved
9. The Reload server can be monitored from a web browser
10. Because a Reload post office is accessed like any other standard GroupWise post office, integrations with third-party products, such as the Advansys product called Archive to Gowork immediately with Reload.

For more information

• If you have any questions about this, go post them in the GWAVA Nation forum and I'll see what I can do to answer them for you. I'll be blogging^{Wikipedia. ">Wikipedia. " class="glossary-indicator">i} about this as well, so check back often for more information about this critical topic.
• May I invite you to take a look at the Reload overview page at GWAVA's web site.