Adventures in upgrade, then restore, then trying to fix the restore, followed by migration to 2008...
(forewarning - this is rant about a failed upgrade from 7913 to 7916, a restore to recover from the failed upgrade, followed by a migration from 2003 to 2008R2 - all with some lessons learned in the process thrown in. I wrote this in stages over the past 24 hours so grammatically it may be literally fubar... sorry - I just need to vent)
I was running 7913 on Windows 2003 x86 and figured it was time to upgrade to 7916, in preparation to go to 7916 on 2008R2. As is usual with SCP patches (at least on my system) - the upgrade from 7913 to 7914 failed to install on the first try - thank goodness I always shutdown Windows and take a VM snapshot before attempting to install a SCP patch. Of course this basically fubar's SCP and renders it unstartable (hmm - the spell checker wants to change "unstartable" to "unstable" - that might be a good word to describe it too)
Well, after the upgrade failed to install on the second try (can't tell you why it failed - don't really care why either at this point - the VM snapshot is my ace in the hole to get back to my original working config), I figured what the heck - lets uninstall, reinstall, and restore the backup - I never have had to restore a backup for as long as we've been running SCP (snapshots are great!). So I uninstall SCP while downloading the full version of 7913 from ManageEngine's archives.
Now you've got to keep in mind we've been running SCP for a long time - long enough that the original installer installed SCP into C:\AdventNet\ME\SupportCenter by default. Now today's installers defaults to C:\ManageEngine\SupportCenter. No biggie though - it's not like I am really attached to the folder name it gets installed to. So I let the installer do it's thing (that mistake number one). Following the instructions posted at the bottom of the patches download page (where it documents the restore / recovery procedure in the event an update fails), I log into the SCP once as is advised - yeah - the basic install worked.
Ok - time to restore... Launch restoreData.bat, point it at my backup that I previously created, and let 'er rip... Well - that was mistake number two...
So what I've found is a couple of major gotchas when restoring.
1.) The license keys are not restored by restoreData.bat
2.) Any SSL certificates are not backed up by backUpData.bat
3.) Any configuration changes to the database or ports are not backed up by backUpData.bat
4.) Any passwords set for the root user in MYSQL are not backed up by backUpData.bat
5.) Any file system path changes are NOT retained in the database during the restore - in my case the restores executes the SQL scripts from the backup that points at C:\AdventNet\ME\SupportCenter (specifically in the scripts application.sql, conffile.sql, and moduleinstance.sql)
6.) Any customization for the database connector are not backed up by backUpData.bat
7.) We had a few other customizations we had done in the past - i.e. Rebranding (rebrandInfo.xml) - none of which are backed up by backUpData.bat
**Narrative... Now at this point, thankfully - I have the VM snapshot that I could restore to because SCP just wasn't all there or working the way it should. Not wanting to mess with the production server any longer, I restored the snapshot to 7913, then took a 7-Zip copy of the working SCP 7913 installation folder. Then I cloned our 2003R2 VM template, copied the 7-Zip working copy of SCP 7913 from the production server to a temp folder on the new VM, isolated the VM from the production network - giving the new VM the same hostname and IP as the production server, and started over as if I had just uninstalled SCP after the failed 7914 upgrade. The install and restore didn't go any better than it did on the production server... I think I started over in this manner 6 times just to get all the steps right and working - gotta love snapshots.
For the missing files, I simply copied from the 7-Zip what I needed to fix this mess. As for the path changes - I discovered the backup file easily opens in 7-zip - I extracted application.sql, connfile.sql, and moduleinstance.sql and used Notepad++ to search and replace with the correct path. Then I simply re-inserted them in the backup and used it to restore with again.
Actually - if you open up the backup files created by backUpData.bat using 7-Zip, you'll find it only backups the following folders:
C:\ManageEngine\SupportCenter\custom
C:\ManageEngine\SupportCenter\fileAttachments
C:\ManageEngine\SupportCenter\index
C:\ManageEngine\SupportCenter\inlineimages
As for the database, backUpData.bat simply exports the tables out to scripts. During the restore, it simply executes these scripts to re-populate the database
Long story short - don't rely on just the SCP backup. Here are the additional files I would recommend you keep safe so if you ever have to do a restore... There may be more than this - but these are ones I found so far.
C:\ManageEngine\SupportCenter\Bin\license\supportcenterplus_license.xml
C:\ManageEngine\SupportCenter\server\default\conf\rebrandInfo.xml
C:\ManageEngine\SupportCenter\server\default\conf\scp.keystore <-- this may not exist on your system if you are not using ssl
C:\ManageEngine\SupportCenter\server\default\conf\fqdn_sslcert.pfx <-- this may not exist on your system if you are not using ssl
C:\ManageEngine\SupportCenter\server\default\deploy\mysql-ds.xml
C:\ManageEngine\SupportCenter\server\default\deploy\jbossweb-tomcat50.sar\server.xml
Also - take note and document any password changes to the root user for MySQL - you'll want to put that back after the restore.
To make things simpler for me (and to remove the mystery if SCP was actually running or not), when I was prompted if I wanted to start the SCP service after the fresh install, I selected no. I also set the ManageEngine service to manual at that point. I opened a command prompt in C:\ManageEngine\SupportCenter\Bin\ and ran startDB.bat. Then I ran run.bat - once SCP finished initializing I logged in the one time, logged out, and hit CTRL+C to shutdown SCP. After every change (i.e. restore, https, MYSQL password change, patch installation, etc) I would run run.bat once login once, the stop SCP. When I finally finished the entire restore process, I changed the ManageEngine service to automatic and rebooted.
**Narrative - after documenting and fixing things in my new temporary 2003 VM so it worked correctly after a restore to a fresh system, I went back to the production server, created another backup using backUpData.bat, took another VM snapshot, uninstalled 7913, and restored using the notes I had wrote working with the temporary 2003 VM - the entire process took about 15 minutes at this point, including the upgrade to 7916.
So now that I was on 7916, and have removed most of the mystery for myself around the restore process, I decided to move the installation from 2003 to 2008R2. It should have been a lot smoother process this time around - but it wasn't really. Well - maybe...
So I clone my standard Windows 2008 R2 template into a new VM. I copy all the necessary files to it to install SCP, including the full install of 7916, and a backup of 7916 from my 2003 server. I move the 2008R2 VM off onto it's own, unroutable network, and configure it with the same name and IP as the 2003 server. I login as a local administrator (since it is not yet added to the domain), and proceed to launch the 7916 installer.
Problem one appears - MySQL is no longer bundled with the full installer as of 7915. What the heck - I guess I can migrate to Postgres if I really have too. Ooohhh - good news - the default path is still C:\ManageEngine\SupportCenter though. So I push through the initial install mostly with defaults, then change the ManageEngine service to manual. I run StartDB.bat, and image my surprise when Postgres won't start because I am logged in as an Administrator. Ok - "net user Postgres Password /add" followed by "runas /env /user:postgres StartDB.bat". Well - at least Postgres is running now. Launch run.bat, followed by IE and login once as per the restore / recovery procedure, then kill run.bat with a CTRL+C.
Launch restoreData.bat, point it at my 2003 backup and let her rip. Much to my surprise - it appears to have completed successfully - keyword here being "appears". Run run.bat and IE again and log using http. It looks ok - but wait - our company logo doesn't appear any more. But is present in C:\ManageEngine\SupportCenter\custom\customimages. Maybe it was just an IE issue - nope - Firefox, Chrome and Safari are do not show it. Well - its only the company logo - I can re-import it - no big deal. Uh oh... better check preferences to see if my signature is still there - along with the images in it. Nope - they are all gone too, for all our users btw - great...
Well - I guess it is ok - there are not that many of us - we can recreate them if we have too. Times wasting - I need to get HTTPS active so I can put this thing into production before customers start hitting the 2003 box for the day. "changeWebServerPort.bat 8443 https". Copy my sslcert in conf, and edit server.xml to include keystorePass="sslcert_password" keystoreType="pkcs12" - just like it was on 2003. run.bat and launch IE. Page not found - where did SCP go? Netstat -ano | find ":8443" Nope - nothing is listening on 8443 - but right there in the run.bat console window - "Server Started. Please connect your client at https://localhost:8443". I'm afraid not.
Times ticking... I don't really want to keep screwing around with this - so enough troubleshooting. Back to the production 2003 server. I stopped the ManageEngine service - manually stopping the MySQL database afterwards with mysqladmin (for some reason - MySQL doesn't always stop when the service does). Right click C:\ManageEngine and create a new 7-Zip file afterwhich I transferred that over to my isolated 2008R2 server. After verifying nothing to do with SCP or Postgres was running, I renamed C:\ManageEngine to C:\ManageEngine.PG, and then extracted the 7-Zip file to the root of the C:\. After the extracting was complete, I run startDB.bat and image my joy when MySQL starts up. run.bat... Launch IE and whoa... There is SCP as it was on 2003 - with logos, signatures, and correct SSL certificates.
I did a couple of quick tests, then shut down 2008R2 and moved the VM over the production network, shutting down the 2003 server. All that was left was to delete C:\ManageEngine.PG, add the 2008R2 server to the domain, install Symantec Endpoint Protection and BackupExec agents, and configure a few custom scheduled tasks (i.e. backUpData.bat, daily reboot, and few other maintenence tasks). I now have a shiny new 2008R2 server running 7916 using MySQL.
That was this morning. Since then, I've been busy putting some of what I learn from this adventure to work. I now have 4 separate backups configured. One with BackupExec to backup C:\ManageEngine to our dedup folder every night. One with BackupExec to do full VMDK backups nightly to tape. One utilizing backupData.bat (but edited change the location to "C:\SCP Backups\Integrated Backup" instead of "C:\ManageEngine\SupportCenter\backup") that runs nightly at 9:30. And finally I wrote a VBS script that stops the ManageEngine Support Center service with a net stop /y, then 7-Zips "C:\ManageEngine" to "C:\SCP Backups\7z Backups" and restarts the service afterwards - this runs at 3:45 am. This way I now have complete backups of the C:\ManageEngine folder on the server - just incase...
I've also setup a couple of other maintenance scripts that deletes backups older than 2 weeks from these two backup folders, and removes the host access logs that are more than 2 weeks old to keep disk utilization to a minimum.
Looking back - the migration from 2003R2 to 2008 could have been simple had I known then what I know now. Build a 2008R2 box, install the same version SCP as was on the 2003 box, into the same folder structure (just to create the ManageEngine Support Center service on 2008R2). Shutdown SCP on 2003, shutdown SCP on 2008R2, rename the install folder on 2008R2, and copy the folder structure over from 2003. Restart ManageEngine Support Center service, and you are done. The fact that the installer uses Postgres versus MySQL didn't matter in the end, because neither of them are installed as Windows services, and both database engines live in and are started from inside C:\ManageEngine, with no external hooks.
Anyways - hopefully someone will find this information useful and timely - before they experience a disaster or failed patch install. Besides - I feel better after having vented about this entire process - but really - really - should it be this difficult?
dcc
I was running 7913 on Windows 2003 x86 and figured it was time to upgrade to 7916, in preparation to go to 7916 on 2008R2. As is usual with SCP patches (at least on my system) - the upgrade from 7913 to 7914 failed to install on the first try - thank goodness I always shutdown Windows and take a VM snapshot before attempting to install a SCP patch. Of course this basically fubar's SCP and renders it unstartable (hmm - the spell checker wants to change "unstartable" to "unstable" - that might be a good word to describe it too)
Well, after the upgrade failed to install on the second try (can't tell you why it failed - don't really care why either at this point - the VM snapshot is my ace in the hole to get back to my original working config), I figured what the heck - lets uninstall, reinstall, and restore the backup - I never have had to restore a backup for as long as we've been running SCP (snapshots are great!). So I uninstall SCP while downloading the full version of 7913 from ManageEngine's archives.
Now you've got to keep in mind we've been running SCP for a long time - long enough that the original installer installed SCP into C:\AdventNet\ME\SupportCenter by default. Now today's installers defaults to C:\ManageEngine\SupportCenter. No biggie though - it's not like I am really attached to the folder name it gets installed to. So I let the installer do it's thing (that mistake number one). Following the instructions posted at the bottom of the patches download page (where it documents the restore / recovery procedure in the event an update fails), I log into the SCP once as is advised - yeah - the basic install worked.
Ok - time to restore... Launch restoreData.bat, point it at my backup that I previously created, and let 'er rip... Well - that was mistake number two...
So what I've found is a couple of major gotchas when restoring.
1.) The license keys are not restored by restoreData.bat
2.) Any SSL certificates are not backed up by backUpData.bat
3.) Any configuration changes to the database or ports are not backed up by backUpData.bat
4.) Any passwords set for the root user in MYSQL are not backed up by backUpData.bat
5.) Any file system path changes are NOT retained in the database during the restore - in my case the restores executes the SQL scripts from the backup that points at C:\AdventNet\ME\SupportCenter (specifically in the scripts application.sql, conffile.sql, and moduleinstance.sql)
6.) Any customization for the database connector are not backed up by backUpData.bat
7.) We had a few other customizations we had done in the past - i.e. Rebranding (rebrandInfo.xml) - none of which are backed up by backUpData.bat
**Narrative... Now at this point, thankfully - I have the VM snapshot that I could restore to because SCP just wasn't all there or working the way it should. Not wanting to mess with the production server any longer, I restored the snapshot to 7913, then took a 7-Zip copy of the working SCP 7913 installation folder. Then I cloned our 2003R2 VM template, copied the 7-Zip working copy of SCP 7913 from the production server to a temp folder on the new VM, isolated the VM from the production network - giving the new VM the same hostname and IP as the production server, and started over as if I had just uninstalled SCP after the failed 7914 upgrade. The install and restore didn't go any better than it did on the production server... I think I started over in this manner 6 times just to get all the steps right and working - gotta love snapshots.
For the missing files, I simply copied from the 7-Zip what I needed to fix this mess. As for the path changes - I discovered the backup file easily opens in 7-zip - I extracted application.sql, connfile.sql, and moduleinstance.sql and used Notepad++ to search and replace with the correct path. Then I simply re-inserted them in the backup and used it to restore with again.
Actually - if you open up the backup files created by backUpData.bat using 7-Zip, you'll find it only backups the following folders:
C:\ManageEngine\SupportCenter\custom
C:\ManageEngine\SupportCenter\fileAttachments
C:\ManageEngine\SupportCenter\index
C:\ManageEngine\SupportCenter\inlineimages
As for the database, backUpData.bat simply exports the tables out to scripts. During the restore, it simply executes these scripts to re-populate the database
Long story short - don't rely on just the SCP backup. Here are the additional files I would recommend you keep safe so if you ever have to do a restore... There may be more than this - but these are ones I found so far.
C:\ManageEngine\SupportCenter\Bin\license\supportcenterplus_license.xml
C:\ManageEngine\SupportCenter\server\default\conf\rebrandInfo.xml
C:\ManageEngine\SupportCenter\server\default\conf\scp.keystore <-- this may not exist on your system if you are not using ssl
C:\ManageEngine\SupportCenter\server\default\conf\fqdn_sslcert.pfx <-- this may not exist on your system if you are not using ssl
C:\ManageEngine\SupportCenter\server\default\deploy\mysql-ds.xml
C:\ManageEngine\SupportCenter\server\default\deploy\jbossweb-tomcat50.sar\server.xml
Also - take note and document any password changes to the root user for MySQL - you'll want to put that back after the restore.
To make things simpler for me (and to remove the mystery if SCP was actually running or not), when I was prompted if I wanted to start the SCP service after the fresh install, I selected no. I also set the ManageEngine service to manual at that point. I opened a command prompt in C:\ManageEngine\SupportCenter\Bin\ and ran startDB.bat. Then I ran run.bat - once SCP finished initializing I logged in the one time, logged out, and hit CTRL+C to shutdown SCP. After every change (i.e. restore, https, MYSQL password change, patch installation, etc) I would run run.bat once login once, the stop SCP. When I finally finished the entire restore process, I changed the ManageEngine service to automatic and rebooted.
**Narrative - after documenting and fixing things in my new temporary 2003 VM so it worked correctly after a restore to a fresh system, I went back to the production server, created another backup using backUpData.bat, took another VM snapshot, uninstalled 7913, and restored using the notes I had wrote working with the temporary 2003 VM - the entire process took about 15 minutes at this point, including the upgrade to 7916.
So now that I was on 7916, and have removed most of the mystery for myself around the restore process, I decided to move the installation from 2003 to 2008R2. It should have been a lot smoother process this time around - but it wasn't really. Well - maybe...
So I clone my standard Windows 2008 R2 template into a new VM. I copy all the necessary files to it to install SCP, including the full install of 7916, and a backup of 7916 from my 2003 server. I move the 2008R2 VM off onto it's own, unroutable network, and configure it with the same name and IP as the 2003 server. I login as a local administrator (since it is not yet added to the domain), and proceed to launch the 7916 installer.
Problem one appears - MySQL is no longer bundled with the full installer as of 7915. What the heck - I guess I can migrate to Postgres if I really have too. Ooohhh - good news - the default path is still C:\ManageEngine\SupportCenter though. So I push through the initial install mostly with defaults, then change the ManageEngine service to manual. I run StartDB.bat, and image my surprise when Postgres won't start because I am logged in as an Administrator. Ok - "net user Postgres Password /add" followed by "runas /env /user:postgres StartDB.bat". Well - at least Postgres is running now. Launch run.bat, followed by IE and login once as per the restore / recovery procedure, then kill run.bat with a CTRL+C.
Launch restoreData.bat, point it at my 2003 backup and let her rip. Much to my surprise - it appears to have completed successfully - keyword here being "appears". Run run.bat and IE again and log using http. It looks ok - but wait - our company logo doesn't appear any more. But is present in C:\ManageEngine\SupportCenter\custom\customimages. Maybe it was just an IE issue - nope - Firefox, Chrome and Safari are do not show it. Well - its only the company logo - I can re-import it - no big deal. Uh oh... better check preferences to see if my signature is still there - along with the images in it. Nope - they are all gone too, for all our users btw - great...
Well - I guess it is ok - there are not that many of us - we can recreate them if we have too. Times wasting - I need to get HTTPS active so I can put this thing into production before customers start hitting the 2003 box for the day. "changeWebServerPort.bat 8443 https". Copy my sslcert in conf, and edit server.xml to include keystorePass="sslcert_password" keystoreType="pkcs12" - just like it was on 2003. run.bat and launch IE. Page not found - where did SCP go? Netstat -ano | find ":8443" Nope - nothing is listening on 8443 - but right there in the run.bat console window - "Server Started. Please connect your client at https://localhost:8443". I'm afraid not.
Times ticking... I don't really want to keep screwing around with this - so enough troubleshooting. Back to the production 2003 server. I stopped the ManageEngine service - manually stopping the MySQL database afterwards with mysqladmin (for some reason - MySQL doesn't always stop when the service does). Right click C:\ManageEngine and create a new 7-Zip file afterwhich I transferred that over to my isolated 2008R2 server. After verifying nothing to do with SCP or Postgres was running, I renamed C:\ManageEngine to C:\ManageEngine.PG, and then extracted the 7-Zip file to the root of the C:\. After the extracting was complete, I run startDB.bat and image my joy when MySQL starts up. run.bat... Launch IE and whoa... There is SCP as it was on 2003 - with logos, signatures, and correct SSL certificates.
I did a couple of quick tests, then shut down 2008R2 and moved the VM over the production network, shutting down the 2003 server. All that was left was to delete C:\ManageEngine.PG, add the 2008R2 server to the domain, install Symantec Endpoint Protection and BackupExec agents, and configure a few custom scheduled tasks (i.e. backUpData.bat, daily reboot, and few other maintenence tasks). I now have a shiny new 2008R2 server running 7916 using MySQL.
That was this morning. Since then, I've been busy putting some of what I learn from this adventure to work. I now have 4 separate backups configured. One with BackupExec to backup C:\ManageEngine to our dedup folder every night. One with BackupExec to do full VMDK backups nightly to tape. One utilizing backupData.bat (but edited change the location to "C:\SCP Backups\Integrated Backup" instead of "C:\ManageEngine\SupportCenter\backup") that runs nightly at 9:30. And finally I wrote a VBS script that stops the ManageEngine Support Center service with a net stop /y, then 7-Zips "C:\ManageEngine" to "C:\SCP Backups\7z Backups" and restarts the service afterwards - this runs at 3:45 am. This way I now have complete backups of the C:\ManageEngine folder on the server - just incase...
I've also setup a couple of other maintenance scripts that deletes backups older than 2 weeks from these two backup folders, and removes the host access logs that are more than 2 weeks old to keep disk utilization to a minimum.
Looking back - the migration from 2003R2 to 2008 could have been simple had I known then what I know now. Build a 2008R2 box, install the same version SCP as was on the 2003 box, into the same folder structure (just to create the ManageEngine Support Center service on 2008R2). Shutdown SCP on 2003, shutdown SCP on 2008R2, rename the install folder on 2008R2, and copy the folder structure over from 2003. Restart ManageEngine Support Center service, and you are done. The fact that the installer uses Postgres versus MySQL didn't matter in the end, because neither of them are installed as Windows services, and both database engines live in and are started from inside C:\ManageEngine, with no external hooks.
Anyways - hopefully someone will find this information useful and timely - before they experience a disaster or failed patch install. Besides - I feel better after having vented about this entire process - but really - really - should it be this difficult?
dcc
Topic Participants
dean.colpitts
Renjith Chembakarayil
SupportCenter Support