Cisco Prime Infrastructure Inline Upgrade Failure 3.1 to 3.4

Prime Infrastructure

 

Experience with upgrading a Cisco Prime Infrastructure instance from version 3.1 to 3.4.

 

Current Prime Infrastructure Instance:

!!!! Hardware
VMware 6.5
16 CPU
24G Memory



!!!! show version
Cisco Application Deployment Engine OS Release: 3.1
ADE-OS Build Version: 3.1.0.001
ADE-OS System Architecture: x86_64

Copyright (c) 2009-2016 by Cisco Systems, Inc.
All rights reserved.
Hostname: tnetprime


Version information of installed applications
---------------------------------------------

Cisco Prime Infrastructure
********************************************************
Version : 3.1.0
Build : 3.1.0.0.132
Critical Fixes:
        PI 3.1.6 Maintenance Release ( 6.0.0 )
Device Support:
        Prime Infrastructure 3.1 Device Pack 10 ( 10.0 )
        Prime Infrastructure 3.1 Device Pack 11 ( 11.0 )

 

Upgrading to Version: 3.4

 

Inline Upgrade 3.1 to 3.4

 

The Prime instance is backed up to a remote repository. Upgrade bundle is copied locally into Prime and upgrade process initiated.

tnetprime/admin# backup pi3.1-before-upgrade repository tnetbackup application NCS
...
  
tnetprime/admin# show repository tnetbackup
...

tnetprime/admin# copy ftp://tnetbackup.teranetics.net/prime/PI_BUNDLE-3.4.0.0.348-Upgrade.tar.gz disk:/defaultRepo
...

tnetprime/admin# show repository defaultRepo
...

tnetprime/admin# application upgrade PI_BUNDLE-3.4.0.0.348-Upgrade.tar.gz defaultRepo
Save the current ADE-OS running configuration? (yes/no) [yes] ? yes
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Please ensure you have a backup of the system before proceeding.
Proceed with the application upgrade ? (yes/no) [yes] ? yes

DO NOT press ^C while the upgrade is in progress
Aborting upgrade with a ^C may leave the system in a unrecoverable state

Initiating Application Upgrade...
  Stage 1 of 7: Transferring file ...
  -- complete.
  Stage 2 of 7: Unpacking file ...
  -- complete.

*** System will reboot after a successful installation of this package ***
After reboot, please login again into the server to check status

  Stage 3 of 7: Executing pre-install ...
  -- complete.
  Stage 4 of 7: Upgrading binaries ...
  -- complete.
grep: /opt/CSCOlumos/LumosInstall.log: No such file or directory
          Install Operation Failed

% This application Install or Upgrade requires reboot, rebooting now...

Broadcast message from admin@tnetprime
        (/dev/pts/4) at 11:50 ...

The system is going down for reboot NOW!

Application upgrade successful

tnetprime/admin# 

 

The prime instance is pretty much stoned at this point. The command “ncs status” reported all services down. “show version” reports old version. Server reboot didn’t help. Running the upgrade again fails,┬ástating that the upgrade version is already installed.

Trying to restore and it failed as well.

tnetprime/admin# restore pi3.1-before-upgrade-180514-1120__VER3.1.0.98.15_BKSZ31G_CPU16_MEM4G_RAM23G_SWAP15G_APP_CK2007684786.tar.gpg repository tnetbackup application NCS
* NOTE *
If the system console is disconnected or got cleared on session timeout
run 'show restore log' to see the output of the last restore session.

Restore will restart the application services. Continue? (yes/no) [yes] ?

DO NOT press ^C while the restoration is in progress
Aborting restore with a ^C may leave the system in a unrecoverable state

Initiating restore.  Please wait...
ERROR: The last upgrade operation was unsuccessful. See dbadmin_StdOut.log for details. To recover, reinstall the server and restore the saved backup data.
% Application restore failed

tnetprime/admin#

 

Instead of troubleshooting it further, we decided to create a new Prime instance and restore from backup.

 

Migrate 3.1 to 3.4

 

New Prime Infrastructure VM deployed from OVA file:

 

Restored from backup data and it worked this time.

tnetprime/admin# copy ftp://tnetbackup.teranetics.net/prime/pi3.1-before-upgrade-180514-1120__VER3.1.0.98.15_BKSZ31G_CPU16_MEM4G_RAM23G_SWAP15G_APP_CK2007684786.tar.gpg disk:/defaultRepo/
...

tnetprime/admin# show repository defaultRepo
...

tnetprime/admin# restore pi3.1-before-upgrade-180514-1120__VER3.1.0.98.15_BKSZ31G_CPU16_MEM4G_RAM23G_SWAP15G_APP_CK2007684786.tar.gpg repository defaultRepo application NCS

* NOTE *
If the system console is disconnected or got cleared on session timeout
run 'show restore log' to see the output of the last restore session.

Restore will restart the application services. Continue? (yes/no) [yes] ?

DO NOT press ^C while the restoration is in progress
Aborting restore with a ^C may leave the system in a unrecoverable state

Enter the backup password, if your backup is password protected. Otherwise, press Enter to continue the data restoration.

Password :
Initiating restore.  Please wait...
  Restore Started at 05/15/18 14:08:00
  Stage 1 of 9: Transferring backup file ...
  -- completed at 05/15/18 14:08:08
  Stage 2 of 9: Decrypting backup file ...
  -- completed at  05/15/18 14:11:31
  Stage 3 of 9: Unpacking backup file ...
  -- completed at  05/15/18 14:11:33
  Stopping PI server ...
  Stage 4 of 9: Decompressing backup ...
  -- completed at  05/15/18 14:13:28
  Stage 5 of 9: Restoring Support Files ...
  -- completed at  05/15/18 14:13:44
  Stage 6 of 9: Restoring Database Files ...
   -- completed at  05/15/18 14:14:51
  Stage 7 of 9: Recovering Database ...
  -- completed at  05/15/18 14:36:06
  Stage 8 of 9: Updating Database Schema ...
    This could take long time based on the existing data size.
                  Stage 1 of 5: Pre Migration Schema Upgrade ...
                                        -- completed at: 2018-05-15 14:42:30.766, Time Taken : 0 hr, 6 min, 17 sec
                  Stage 2 of 5: Schema Upgrade ...
                                        -- completed at: 2018-05-15 15:17:52.647, Time Taken : 0 hr, 35 min, 20 sec
                  Stage 3 of 5: Post Migration Schema Upgrade ...
                                        -- completed at: 2018-05-15 15:22:23.182, Time Taken : 0 hr, 4 min, 28 sec
                  Stage 4 of 5: Enabling DB Constraints ...
                                        -- completed at: 2018-05-15 15:23:59.862, Time Taken : 0 hr, 1 min, 28 sec
                  Stage 5 of 5: Finishing Up ...
                                        -- completed at: 2018-05-15 15:24:46.311, Time Taken : 0 hr, 0 min, 45 sec
  -- completed at  05/15/18 15:25:21

  Stage 9 of 9: Re-enabling Database Settings ...
   -- completed at  05/15/18 16:16:51
   Total Restore duration is: 02h:08m:51s
INFO: Restore completed successfully.

Starting Prime Infrastructure...

This may take a while (10 minutes or more) ...

Prime Infrastructure started successfully.

Completed in 1117 seconds

tnetprime/admin# ncs status
Health Monitor Server is running. ( [Role] Primary [State] HA not Configured )
Database server is running
FTP Service is running
TFTP Service is running
Matlab Server is running
Matlab Server Instance 1 is running
Matlab Server Instance 2 is running
Matlab Server Instance 3 is running
NMS Server is running.
WSA Service is running.
SAM Daemon is running ...
DA Daemon is running ...
Compliance engine is running

tnetprime/admin# show ver

Cisco Application Deployment Engine OS Release: 3.1
ADE-OS Build Version: 3.1.0.001
ADE-OS System Architecture: x86_64

Copyright (c) 2009-2018 by Cisco Systems, Inc.
All rights reserved.
Hostname: tnetprime


Version information of installed applications
---------------------------------------------

Cisco Prime Infrastructure
********************************************************
Version : 3.4.0
Build : 3.4.0.0.348

tnetprime/admin#


 

Prime Infrastructure successfully upgraded to version 3.4 using the data migration method.

 

Share this:


Tagged with:
| |

Leave a Comment

1 Comment on "Cisco Prime Infrastructure Inline Upgrade Failure 3.1 to 3.4"

  Subscribe  
newest oldest
Notify of

Thanks for this article. I had the same problem and this allowed me to get to 3.4.