Cisco Identity Services Engine Upgrade Failure 2.0 to 2.3

Cisco ISE

 

Experience with upgrading a two-node Cisco Identity Services Engine (ISE) deployment from version 2.0 to 2.3.

 

Current Cisco ISE deployment:

!!!! Hardware

VMware 6.5

Node 1: Primary Admin, Secondary Monitoring, PSN
  4 CPU
  24G Memory

Node 2: Secondary Admin, Primary Monitoring, PSN
  4 CPU
  24G Memory



!!!! Show version

Cisco Application Deployment Engine OS Release: 2.3
ADE-OS Build Version: 2.3.0.187
ADE-OS System Architecture: x86_64

Copyright (c) 2005-2014 by Cisco Systems, Inc.
All rights reserved.
Hostname: tnetise01

Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version      : 2.0.0.306
Build Date   : Thu Oct  8 18:55:23 2015
Install Date : Mon Oct 16 13:14:57 2017

Cisco Identity Services Engine Patch
---------------------------------------------
Version      : 5
Install Date : Tue Oct 17 14:42:24 2017

 

Upgrading to version: 2.3

Cisco ISE Upgrade Bundle 2.3

 

Running the Upgrade Readiness Tool (URT) first. URT is run on the ISE Secondary Admin Node (SAN). It performs a mock database upgrade (clones the actual and upgrades it) which validates how the actual upgrade will go when initiated. A more recent version of URT tool was available which wasn’t noticed at that time. The recommendation is to run the latest version of URT when available.

 

tnetise02/admin# application install ise-urtbundle-2.3.0.298-1.1.0.SPA.x86_64.tar.gz local
Save the current ADE-OS running configuration? (yes/no) [yes] ? yes
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...

###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################

Checking ISE version compatibility
- Successful

Checking ISE persona
- Successful

Along with Administration, other services (MNT,SESSION) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<45 days old)
- Note: URT is 244 days old and its version is 1.1.0. There might be a recent URT bundle on CCO, please verify on CCO
Do you want to proceed with this version which is 244 days old (y/n):y
Proceeding with this version of URT itself

Installing URT bundle
- Successful

########################################
# Running Upgrade Readiness Tool (URT) #
########################################
This tool will perform following tasks:
1. Pre-requisite checks
2. Clone config database
3. Copy upgrade files
4. Data upgrade on cloned database
5. Time estimate for upgrade

Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Successful
Appliance/VM compatibility
- Successful
Trust Cert Validation
- Successful
System Cert Validation
- Successful
5 out of 5 pre-requisite checks passed

Clone config database
=====================
 [########################################] 100%  Successful

Copy upgrade files
==================
- N/A

Data upgrade on cloned database
===============================
Modifying upgrade scripts to run on cloned database
- Successful

Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
- Successful

Running sanity after schema upgrade on cloned database
- Successful

Running data upgrade on cloned database
- Data upgrade step 1/100, AuthzUpgradeService(2.0.0.308)... ..Done in 169 seconds.
- Data upgrade step 2/100, NSFUpgradeService(2.1.0.102)... Done in 1 seconds.
- Data upgrade step 3/100, UPSUpgradeHandler(2.1.0.105)... ..Done in 131 seconds.
- Data upgrade step 4/100, UPSUpgradeHandler(2.1.0.107)... Done in 2 seconds.
- Data upgrade step 5/100, NSFUpgradeService(2.1.0.109)... Done in 0 seconds.
- Data upgrade step 6/100, NSFUpgradeService(2.1.0.126)... Done in 0 seconds.
- Data upgrade step 7/100, NetworkAccessUpgrade(2.1.0.127)... Done in 1 seconds.
- Data upgrade step 8/100, ProfilerUpgradeService(2.1.0.134)... Done in 0 seconds.
- Data upgrade step 9/100, ProfilerUpgradeService(2.1.0.139)... Done in 0 seconds.
- Data upgrade step 10/100, ProfilerUpgradeService(2.1.0.166)... ................- Failed
Final cleanup before exiting...

Collecting log files ...
- Encrypting logs bundle...
Please enter encryption password:
Please enter encryption password again to verify:
Encrypted URT logs(urt_logs.tar.gpg) are available in localdisk. Please reach out to Cisco TAC to debug
% Post-install step failed. Please check the logs for more details.

tnetise02/admin#

 

Researching online brought to our attention that this is a known bugĀ CSCvh13873. A hotfix patch is available that can be obtained by opening a case with Cisco TAC. The patch was obtained and applied.

 

tnetise02/admin# application install ise-apply-CSCvh13873_2.0_2.1_2.2_2.3_common_11-SPA.tar.gz local 
Save the current ADE-OS running configuration? (yes/no) [yes] ? 
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...
 
Checking if CSCvh13873_2.0_2.1_2.2_2.3_common_1 is already applied
  - Successful
 
Checking ISE version compatibility
  - Successful
 
Applying hot patch CSCvh13873_2.0_2.1_2.2_2.3_common_1
Please take a config backup before proceeding with below steps. Do you want to proceed (y/n):y
- Stopping monit and ise application server
Stopping ISE Application Server...
- Running hotpatch wrapper script to delete policies, checks and rules
Deleting policies - 1st set
Deleting policies - 2nd set
Deleting checks and rules
- Starting ise application server and monit
Starting ISE Application Server...
 
Hot patch applied successfully
job 1 at 2018-05-15 14:35

Application successfully installed
tnetise02/admin#

 

The URT ran successfully this time. Another good thing about this tool is it gives an estimate of the actual upgrade times on each node. This is useful in planning the actual upgrade change.

 

tnetise02/admin# application install ise-urtbundle-2.3.0.298-1.1.0.SPA.x86_64.tar.gz local
Save the current ADE-OS running configuration? (yes/no) [yes] ?
Generating configuration...
Saved the ADE-OS running configuration to startup successfully

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...
Initiating Application Install...

###########################################
# Installing Upgrade Readiness Tool (URT) #
###########################################

Checking ISE version compatibility
- Successful

Checking ISE persona
- Successful

Along with Administration, other services (MNT,SESSION) are enabled on this node. Installing and running URT might consume additional resources.
Do you want to proceed with installing and running URT now (y/n):y

Checking if URT is recent(<45 days old)
- Note: URT is 245 days old and its version is 1.1.0. There might be a recent URT bundle on CCO, please verify on CCO
Do you want to proceed with this version which is 245 days old (y/n):y
Proceeding with this version of URT itself

Installing URT bundle
- Successful

########################################
# Running Upgrade Readiness Tool (URT) #
########################################
This tool will perform following tasks:
1. Pre-requisite checks
2. Clone config database
3. Copy upgrade files
4. Data upgrade on cloned database
5. Time estimate for upgrade

Pre-requisite checks
====================
Disk Space sanity check
- Successful
NTP sanity
- Successful
Appliance/VM compatibility
- Successful
Trust Cert Validation
- Successful
System Cert Validation
- Successful
5 out of 5 pre-requisite checks passed

Clone config database
=====================
 [##--------------------------------------] 5% Validating connection to ISE data 
 [####------------------------------------] 10% Validating available disk space  
 [######----------------------------------] 15% Extracting base database files   
 [##########------------------------------] 25% Cloning database                 
 [####################--------------------] 50% Exporting data from ISE database 
 [##############################----------] 75% Importing data into cloned datab 
 [########################################] 100%  Successful                    

Copy upgrade files
==================
- N/A

Data upgrade on cloned database
===============================
Modifying upgrade scripts to run on cloned database
- Successful

Running schema upgrade on cloned database
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
- Successful

Running sanity after schema upgrade on cloned database
- Successful

Running data upgrade on cloned database
- Data upgrade step 1/100, AuthzUpgradeService(2.0.0.308)... Done in 48 seconds.
- Data upgrade step 2/100, NSFUpgradeService(2.1.0.102)... Done in 1 seconds.
- Data upgrade step 3/100, UPSUpgradeHandler(2.1.0.105)... .Done in 63 seconds.
- Data upgrade step 4/100, UPSUpgradeHandler(2.1.0.107)... Done in 1 seconds.
- Data upgrade step 5/100, NSFUpgradeService(2.1.0.109)... Done in 0 seconds.
- Data upgrade step 6/100, NSFUpgradeService(2.1.0.126)... Done in 0 seconds.
- Data upgrade step 7/100, NetworkAccessUpgrade(2.1.0.127)... Done in 1 seconds.
- Data upgrade step 8/100, ProfilerUpgradeService(2.1.0.134)... Done in 0 seconds.
- Data upgrade step 9/100, ProfilerUpgradeService(2.1.0.139)... Done in 0 seconds.
- Data upgrade step 10/100, ProfilerUpgradeService(2.1.0.166)... ....Done in 294 seconds.
- Data upgrade step 11/100, NSFUpgradeService(2.1.0.168)... Done in 0 seconds.
- Data upgrade step 12/100, AlarmsUpgradeHandler(2.1.0.169)... Done in 2 seconds.
- Data upgrade step 13/100, RegisterPostureTypes(2.1.0.180)... Done in 15 seconds.
- Data upgrade step 14/100, RegisterPostureTypes(2.1.0.189)... Done in 0 seconds.
- Data upgrade step 15/100, UPSUpgradeHandler(2.1.0.194)... Done in 0 seconds.
- Data upgrade step 16/100, TrustsecWorkflowRegistration(2.1.0.203)... Done in 0 seconds.
- Data upgrade step 17/100, NSFUpgradeService(2.1.0.205)... Done in 0 seconds.
- Data upgrade step 18/100, NetworkAccessUpgrade(2.1.0.207)... Done in 0 seconds.
- Data upgrade step 19/100, NSFUpgradeService(2.1.0.212)... Done in 0 seconds.
- Data upgrade step 20/100, NetworkAccessUpgrade(2.1.0.241)... Done in 0 seconds.
- Data upgrade step 21/100, NetworkAccessUpgrade(2.1.0.242)... Done in 1 seconds.
- Data upgrade step 22/100, UPSUpgradeHandler(2.1.0.244)... Done in 0 seconds.
- Data upgrade step 23/100, ProfilerUpgradeService(2.1.0.248)... Done in 0 seconds.
- Data upgrade step 24/100, NetworkAccessUpgrade(2.1.0.254)... Done in 0 seconds.
- Data upgrade step 25/100, UPSUpgradeHandler(2.1.0.255)... Done in 17 seconds.
- Data upgrade step 26/100, MDMPartnerUpgradeService(2.1.0.257)... Done in 0 seconds.
- Data upgrade step 27/100, NetworkAccessUpgrade(2.1.0.258)... Done in 0 seconds.
- Data upgrade step 28/100, ProfilerUpgradeService(2.1.0.258)... Done in 18 seconds.
- Data upgrade step 29/100, MDMPartnerUpgradeService(2.1.0.258)... Done in 0 seconds.
- Data upgrade step 30/100, UPSUpgradeHandler(2.1.0.279)... Done in 1 seconds.
- Data upgrade step 31/100, NSFUpgradeService(2.1.0.282)... Done in 0 seconds.
- Data upgrade step 32/100, NetworkAccessUpgrade(2.1.0.288)... Done in 0 seconds.
- Data upgrade step 33/100, NetworkAccessUpgrade(2.1.0.295)... Done in 1 seconds.
- Data upgrade step 34/100, CertMgmtUpgradeService(2.1.0.296)... Done in 0 seconds.
- Data upgrade step 35/100, NetworkAccessUpgrade(2.1.0.299)... Done in 0 seconds.
- Data upgrade step 36/100, NetworkAccessUpgrade(2.1.0.322)... Done in 0 seconds.
- Data upgrade step 37/100, NetworkAccessUpgrade(2.1.0.330)... Done in 0 seconds.
- Data upgrade step 38/100, NSFUpgradeService(2.1.0.353)... Done in 0 seconds.
- Data upgrade step 39/100, ProfilerUpgradeService(2.1.0.354)... Done in 0 seconds.
- Data upgrade step 40/100, NSFUpgradeService(2.1.0.427)... Done in 0 seconds.
- Data upgrade step 41/100, NSFUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 42/100, ProfilerUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 43/100, UPSUpgradeHandler(2.1.101.188)... Done in 0 seconds.
- Data upgrade step 44/100, NetworkAccessUpgrade(2.2.0.007)... Done in 0 seconds.
- Data upgrade step 45/100, UPSUpgradeHandler(2.2.0.118)... Done in 3 seconds.
- Data upgrade step 46/100, GuestAccessUpgradeService(2.2.0.124)... Done in 36 seconds.
- Data upgrade step 47/100, NSFUpgradeService(2.2.0.135)... Done in 0 seconds.
- Data upgrade step 48/100, NSFUpgradeService(2.2.0.136)... Done in 0 seconds.
- Data upgrade step 49/100, NetworkAccessUpgrade(2.2.0.137)... Done in 0 seconds.
- Data upgrade step 50/100, NetworkAccessUpgrade(2.2.0.143)... Done in 10 seconds.
- Data upgrade step 51/100, NSFUpgradeService(2.2.0.145)... Done in 1 seconds.
- Data upgrade step 52/100, NSFUpgradeService(2.2.0.146)... Done in 1 seconds.
- Data upgrade step 53/100, NetworkAccessUpgrade(2.2.0.155)... Done in 0 seconds.
- Data upgrade step 54/100, CdaRegistration(2.2.0.156)... Done in 4 seconds.
- Data upgrade step 55/100, NetworkAccessUpgrade(2.2.0.161)... Done in 0 seconds.
- Data upgrade step 56/100, UPSUpgradeHandler(2.2.0.166)... Done in 0 seconds.
- Data upgrade step 57/100, NetworkAccessUpgrade(2.2.0.169)... Done in 0 seconds.
- Data upgrade step 58/100, UPSUpgradeHandler(2.2.0.169)... Done in 0 seconds.
- Data upgrade step 59/100, NetworkAccessUpgrade(2.2.0.180)... Done in 0 seconds.
- Data upgrade step 60/100, CertMgmtUpgradeService(2.2.0.200)... Done in 0 seconds.
- Data upgrade step 61/100, NetworkAccessUpgrade(2.2.0.208)... Done in 0 seconds.
- Data upgrade step 62/100, RegisterPostureTypes(2.2.0.218)... Done in 1 seconds.
- Data upgrade step 63/100, NetworkAccessUpgrade(2.2.0.218)... Done in 2 seconds.
- Data upgrade step 64/100, NetworkAccessUpgrade(2.2.0.222)... Done in 0 seconds.
- Data upgrade step 65/100, NetworkAccessUpgrade(2.2.0.223)... Done in 0 seconds.
- Data upgrade step 66/100, NetworkAccessUpgrade(2.2.0.224)... Done in 2 seconds.
- Data upgrade step 67/100, SyslogTemplatesRegistration(2.2.0.224)... Done in 0 seconds.
- Data upgrade step 68/100, ReportUpgradeHandler(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 69/100, IRFUpgradeService(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 70/100, LocalHostNADRegistrationService(2.2.0.261)... Done in 0 seconds.
- Data upgrade step 71/100, NetworkAccessUpgrade(2.2.0.300)... Done in 0 seconds.
- Data upgrade step 72/100, CertMgmtUpgradeService(2.2.0.300)... Done in 0 seconds.
- Data upgrade step 73/100, NSFUpgradeService(2.2.0.323)... Done in 0 seconds.
- Data upgrade step 74/100, NetworkAccessUpgrade(2.2.0.330)... Done in 0 seconds.
- Data upgrade step 75/100, NSFUpgradeService(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 76/100, NetworkAccessUpgrade(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 77/100, NetworkAccessUpgrade(2.2.0.342)... Done in 0 seconds.
- Data upgrade step 78/100, AuthzUpgradeService(2.2.0.344)... Done in 0 seconds.
- Data upgrade step 79/100, RegisterPostureTypes(2.2.0.350)... Done in 0 seconds.
- Data upgrade step 80/100, DictionaryUpgradeRegistration(2.2.0.374)... Done in 37 seconds.
- Data upgrade step 81/100, UPSUpgradeHandler(2.2.0.403)... Done in 0 seconds.
- Data upgrade step 82/100, DictionaryUpgradeRegistration(2.2.0.410)... Done in 0 seconds.
- Data upgrade step 83/100, UPSUpgradeHandler(2.3.0.100)... Done in 36 seconds.
- Data upgrade step 84/100, UPSUpgradeHandler(2.3.0.110)... Done in 1 seconds.
- Data upgrade step 85/100, NetworkAccessUpgrade(2.3.0.145)... Done in 0 seconds.
- Data upgrade step 86/100, NodeGroupUpgradeService(2.3.0.155)... Done in 1 seconds.
- Data upgrade step 87/100, IRFUpgradeService(2.3.0.155)... Done in 0 seconds.
- Data upgrade step 88/100, UPSUpgradeHandler(2.3.0.158)... Done in 0 seconds.
- Data upgrade step 89/100, NetworkAccessUpgrade(2.3.0.178)... Done in 0 seconds.
- Data upgrade step 90/100, NetworkAccessUpgrade(2.3.0.182)... Done in 0 seconds.
- Data upgrade step 91/100, RegisterPostureTypes(2.3.0.189)... Done in 0 seconds.
- Data upgrade step 92/100, CertMgmtUpgradeService(2.3.0.194)... Done in 2 seconds.
- Data upgrade step 93/100, UPSUpgradeHandler(2.3.0.201)... Done in 0 seconds.
- Data upgrade step 94/100, ProfilerUpgradeService(2.3.0.232)... Done in 0 seconds.
- Data upgrade step 95/100, GuestAccessUpgradeService(2.3.0.243)... Done in 0 seconds.
- Data upgrade step 96/100, NSFUpgradeService(2.3.0.249)... Done in 0 seconds.
- Data upgrade step 97/100, NSFUpgradeService(2.3.0.273)... Done in 0 seconds.
- Data upgrade step 98/100, NSFUpgradeService(2.3.0.298)... Done in 0 seconds.
- Data upgrade step 99/100, ProfilerUpgradeService(2.3.0.298)... Done in 2 seconds.
- Data upgrade step 100/100, GuestAccessUpgradeService(2.3.0.298)... Done in 14 seconds.
- Successful

Running data upgrade for node specific data on cloned database
- Successful

Time estimate for upgrade
=========================
(Estimates are calculated based on size of config and mnt data only. Network latency between PAN and other nodes is not considered in calculating estimates)
Estimated time for each node (in mins):
MNT data is 13 GB, purging this data can reduce upgrade time
tnetise02(SECONDARY PAP,MNT,PDP):179
MNT data is 13 GB, purging this data can reduce upgrade time
tnetise01(PRIMARY PAP,MNT,PDP):167


Final cleanup before exiting...

Application successfully installed
tnetise02/admin#

 

Now, it is time for the actual upgrade. Following the Cisco Upgrade Guide for 2.3 available HERE, the ISE SAN node was upgraded first followed by the ISE PAN node.

 

tnetise02/admin# application upgrade prepare ise-upgradebundle-2.0.x-to-2.3.0.298.SPA.x86_64.tar.gz tnetbackup 

Getting bundle to local machine...
Unbundling Application Package...
Verifying Application Signature...

Application upgrade preparation successful
tnetise02/admin# show clock
Thu May 17 13:43:18 AEST 2018
tnetise02/admin# application upgrade proceed
Initiating Application Upgrade...
% Warning: Do not use Ctrl-C or close this terminal window until upgrade completes.
-Checking VM for minimum hardware requirements
STEP 1: Stopping ISE application...
STEP 2: Verifying files in bundle...
-Internal hash verification passed for bundle
STEP 3: Validating data before upgrade...
STEP 4: De-registering node from current deployment...
STEP 5: Taking backup of the configuration data...
STEP 6: Running ISE configuration database schema upgrade...
- Running db sanity to check and fix if any index corruption
- Auto Upgrading Schema for UPS Model
- Upgrading Schema completed for UPS Model
ISE database schema upgrade completed.
STEP 7: Running ISE configuration data upgrade...
- Data upgrade step 1/100, AuthzUpgradeService(2.0.0.308)... .Done in 73 seconds.
- Data upgrade step 2/100, NSFUpgradeService(2.1.0.102)... Done in 0 seconds.
- Data upgrade step 3/100, UPSUpgradeHandler(2.1.0.105)... .Done in 107 seconds.
- Data upgrade step 4/100, UPSUpgradeHandler(2.1.0.107)... Done in 3 seconds.
- Data upgrade step 5/100, NSFUpgradeService(2.1.0.109)... Done in 0 seconds.
- Data upgrade step 6/100, NSFUpgradeService(2.1.0.126)... Done in 1 seconds.
- Data upgrade step 7/100, NetworkAccessUpgrade(2.1.0.127)... Done in 1 seconds.
- Data upgrade step 8/100, ProfilerUpgradeService(2.1.0.134)... Done in 2 seconds.
- Data upgrade step 9/100, ProfilerUpgradeService(2.1.0.139)... Done in 1 seconds.
- Data upgrade step 10/100, ProfilerUpgradeService(2.1.0.166)... .......Done in 426 seconds.
- Data upgrade step 11/100, NSFUpgradeService(2.1.0.168)... Done in 1 seconds.
- Data upgrade step 12/100, AlarmsUpgradeHandler(2.1.0.169)... Done in 6 seconds.
- Data upgrade step 13/100, RegisterPostureTypes(2.1.0.180)... Done in 33 seconds.
- Data upgrade step 14/100, RegisterPostureTypes(2.1.0.189)... Done in 0 seconds.
- Data upgrade step 15/100, UPSUpgradeHandler(2.1.0.194)... Done in 0 seconds.
- Data upgrade step 16/100, TrustsecWorkflowRegistration(2.1.0.203)... Done in 0 seconds.
- Data upgrade step 17/100, NSFUpgradeService(2.1.0.205)... Done in 0 seconds.
- Data upgrade step 18/100, NetworkAccessUpgrade(2.1.0.207)... Done in 0 seconds.
- Data upgrade step 19/100, NSFUpgradeService(2.1.0.212)... Done in 0 seconds.
- Data upgrade step 20/100, NetworkAccessUpgrade(2.1.0.241)... Done in 1 seconds.
- Data upgrade step 21/100, NetworkAccessUpgrade(2.1.0.242)... Done in 4 seconds.
- Data upgrade step 22/100, UPSUpgradeHandler(2.1.0.244)... Done in 0 seconds.
- Data upgrade step 23/100, ProfilerUpgradeService(2.1.0.248)... Done in 0 seconds.
- Data upgrade step 24/100, NetworkAccessUpgrade(2.1.0.254)... Done in 0 seconds.
- Data upgrade step 25/100, UPSUpgradeHandler(2.1.0.255)... Done in 20 seconds.
- Data upgrade step 26/100, MDMPartnerUpgradeService(2.1.0.257)... Done in 0 seconds.
- Data upgrade step 27/100, NetworkAccessUpgrade(2.1.0.258)... Done in 0 seconds.
- Data upgrade step 28/100, ProfilerUpgradeService(2.1.0.258)... Done in 49 seconds.
- Data upgrade step 29/100, MDMPartnerUpgradeService(2.1.0.258)... Done in 2 seconds.
- Data upgrade step 30/100, UPSUpgradeHandler(2.1.0.279)... Done in 2 seconds.
- Data upgrade step 31/100, NSFUpgradeService(2.1.0.282)... Done in 0 seconds.
- Data upgrade step 32/100, NetworkAccessUpgrade(2.1.0.288)... Done in 0 seconds.
- Data upgrade step 33/100, NetworkAccessUpgrade(2.1.0.295)... Done in 2 seconds.
- Data upgrade step 34/100, CertMgmtUpgradeService(2.1.0.296)... Done in 0 seconds.
- Data upgrade step 35/100, NetworkAccessUpgrade(2.1.0.299)... Done in 0 seconds.
- Data upgrade step 36/100, NetworkAccessUpgrade(2.1.0.322)... Done in 0 seconds.
- Data upgrade step 37/100, NetworkAccessUpgrade(2.1.0.330)... Done in 0 seconds.
- Data upgrade step 38/100, NSFUpgradeService(2.1.0.353)... Done in 1 seconds.
- Data upgrade step 39/100, ProfilerUpgradeService(2.1.0.354)... Done in 0 seconds.
- Data upgrade step 40/100, NSFUpgradeService(2.1.0.427)... Done in 0 seconds.
- Data upgrade step 41/100, NSFUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 42/100, ProfilerUpgradeService(2.1.101.145)... Done in 0 seconds.
- Data upgrade step 43/100, UPSUpgradeHandler(2.1.101.188)... Done in 2 seconds.
- Data upgrade step 44/100, NetworkAccessUpgrade(2.2.0.007)... Done in 2 seconds.
- Data upgrade step 45/100, UPSUpgradeHandler(2.2.0.118)... Done in 9 seconds.
- Data upgrade step 46/100, GuestAccessUpgradeService(2.2.0.124)... .Done in 77 seconds.
- Data upgrade step 47/100, NSFUpgradeService(2.2.0.135)... Done in 0 seconds.
- Data upgrade step 48/100, NSFUpgradeService(2.2.0.136)... Done in 0 seconds.
- Data upgrade step 49/100, NetworkAccessUpgrade(2.2.0.137)... Done in 0 seconds.
- Data upgrade step 50/100, NetworkAccessUpgrade(2.2.0.143)... Done in 17 seconds.
- Data upgrade step 51/100, NSFUpgradeService(2.2.0.145)... Done in 1 seconds.
- Data upgrade step 52/100, NSFUpgradeService(2.2.0.146)... Done in 2 seconds.
- Data upgrade step 53/100, NetworkAccessUpgrade(2.2.0.155)... Done in 0 seconds.
- Data upgrade step 54/100, CdaRegistration(2.2.0.156)... Done in 6 seconds.
- Data upgrade step 55/100, NetworkAccessUpgrade(2.2.0.161)... Done in 0 seconds.
- Data upgrade step 56/100, UPSUpgradeHandler(2.2.0.166)... Done in 0 seconds.
- Data upgrade step 57/100, NetworkAccessUpgrade(2.2.0.169)... Done in 1 seconds.
- Data upgrade step 58/100, UPSUpgradeHandler(2.2.0.169)... Done in 0 seconds.
- Data upgrade step 59/100, NetworkAccessUpgrade(2.2.0.180)... Done in 0 seconds.
- Data upgrade step 60/100, CertMgmtUpgradeService(2.2.0.200)... Done in 0 seconds.
- Data upgrade step 61/100, NetworkAccessUpgrade(2.2.0.208)... Done in 0 seconds.
- Data upgrade step 62/100, RegisterPostureTypes(2.2.0.218)... Done in 1 seconds.
- Data upgrade step 63/100, NetworkAccessUpgrade(2.2.0.218)... Done in 4 seconds.
- Data upgrade step 64/100, NetworkAccessUpgrade(2.2.0.222)... Done in 0 seconds.
- Data upgrade step 65/100, NetworkAccessUpgrade(2.2.0.223)... Done in 0 seconds.
- Data upgrade step 66/100, NetworkAccessUpgrade(2.2.0.224)... Done in 2 seconds.
- Data upgrade step 67/100, SyslogTemplatesRegistration(2.2.0.224)... Done in 0 seconds.
- Data upgrade step 68/100, ReportUpgradeHandler(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 69/100, IRFUpgradeService(2.2.0.242)... Done in 0 seconds.
- Data upgrade step 70/100, LocalHostNADRegistrationService(2.2.0.261)... Done in 0 seconds.
- Data upgrade step 71/100, NetworkAccessUpgrade(2.2.0.300)... Done in 0 seconds.
- Data upgrade step 72/100, CertMgmtUpgradeService(2.2.0.300)... Done in 2 seconds.
- Data upgrade step 73/100, NSFUpgradeService(2.2.0.323)... Done in 0 seconds.
- Data upgrade step 74/100, NetworkAccessUpgrade(2.2.0.330)... Done in 1 seconds.
- Data upgrade step 75/100, NSFUpgradeService(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 76/100, NetworkAccessUpgrade(2.2.0.340)... Done in 0 seconds.
- Data upgrade step 77/100, NetworkAccessUpgrade(2.2.0.342)... Done in 0 seconds.
- Data upgrade step 78/100, AuthzUpgradeService(2.2.0.344)... Done in 0 seconds.
- Data upgrade step 79/100, RegisterPostureTypes(2.2.0.350)... Done in 0 seconds.
- Data upgrade step 80/100, DictionaryUpgradeRegistration(2.2.0.374)... Done in 44 seconds.
- Data upgrade step 81/100, UPSUpgradeHandler(2.2.0.403)... Done in 0 seconds.
- Data upgrade step 82/100, DictionaryUpgradeRegistration(2.2.0.410)... Done in 0 seconds.
- Data upgrade step 83/100, UPSUpgradeHandler(2.3.0.100)... .Done in 64 seconds.
- Data upgrade step 84/100, UPSUpgradeHandler(2.3.0.110)... Done in 2 seconds.
- Data upgrade step 85/100, NetworkAccessUpgrade(2.3.0.145)... Done in 0 seconds.
- Data upgrade step 86/100, NodeGroupUpgradeService(2.3.0.155)... Done in 3 seconds.
- Data upgrade step 87/100, IRFUpgradeService(2.3.0.155)... Done in 0 seconds.
- Data upgrade step 88/100, UPSUpgradeHandler(2.3.0.158)... Done in 0 seconds.
- Data upgrade step 89/100, NetworkAccessUpgrade(2.3.0.178)... Done in 0 seconds.
- Data upgrade step 90/100, NetworkAccessUpgrade(2.3.0.182)... Done in 0 seconds.
- Data upgrade step 91/100, RegisterPostureTypes(2.3.0.189)... Done in 0 seconds.
- Data upgrade step 92/100, CertMgmtUpgradeService(2.3.0.194)... Done in 7 seconds.
- Data upgrade step 93/100, UPSUpgradeHandler(2.3.0.201)... Done in 0 seconds.
- Data upgrade step 94/100, ProfilerUpgradeService(2.3.0.232)... ...Done in 185 seconds.
- Data upgrade step 95/100, GuestAccessUpgradeService(2.3.0.243)... Done in 0 seconds.
- Data upgrade step 96/100, NSFUpgradeService(2.3.0.249)... Done in 0 seconds.
- Data upgrade step 97/100, NSFUpgradeService(2.3.0.273)... Done in 0 seconds.
- Data upgrade step 98/100, NSFUpgradeService(2.3.0.298)... Done in 0 seconds.
- Data upgrade step 99/100, ProfilerUpgradeService(2.3.0.298)... Done in 6 seconds.
- Data upgrade step 100/100, GuestAccessUpgradeService(2.3.0.298)... Done in 21 seconds.
STEP 8: Running ISE configuration data upgrade for node specific data...
 STEP 9: Making this node PRIMARY of the new deployment. When other nodes are upgraded it will be added to this deployment.
STEP 10: Running ISE M&T database upgrade...
Stopping ISE Monitoring & Troubleshooting Log Processor...
ISE database M&T schema upgrade completed.
% Warning: Some warnings encountered during MNT sanity check
% NOTICE: The appliance will reboot twice to upgrade software and ADE-OS. During this time progress of the upgrade is visible on console. It could take up to 30 minutes for this to complete.
Rebooting to do Identity Service Engine upgrade...
=~=~=~=~=~=~=~=~=~=~=~= PuTTY log 2018.05.17 15:30:43 =~=~=~=~=~=~=~=~=~=~=~=


tnetise02/admin# show ver

Cisco Application Deployment Engine OS Release: 3.0
ADE-OS Build Version: 3.0.3.030
ADE-OS System Architecture: x86_64

Copyright (c) 2005-2014 by Cisco Systems, Inc.
All rights reserved.
Hostname: tnetise02


Version information of installed applications
---------------------------------------------

Cisco Identity Services Engine
---------------------------------------------
Version      : 2.3.0.298
Build Date   : Wed Jul 26 03:02:37 2017
Install Date : Thu May 17 15:42:05 2018



tnetise02/admin# show application status ise

ISE PROCESS NAME                       STATE            PROCESS ID
--------------------------------------------------------------------
Database Listener                      running          25639
Database Server                        running          66 PROCESSES
Application Server                     running          30690
Profiler Database                      running          27040
ISE Indexing Engine                    running          32539
AD Connector                           running          21825
M&T Session Database                   running          26942
M&T Log Collector                      running          30823
M&T Log Processor                      running          30736
Certificate Authority Service          disabled
EST Service                            running          14094
SXP Engine Service                     disabled
Docker Daemon                          running          911
TC-NAC Service                         disabled

Wifi Setup Helper Container            disabled
pxGrid Infrastructure Service          disabled
pxGrid Publisher Subscriber Service    disabled
pxGrid Connection Manager              disabled
pxGrid Controller                      disabled
PassiveID WMI Service                  disabled
PassiveID Syslog Service               disabled
PassiveID API Service                  disabled
PassiveID Agent Service                disabled
PassiveID Endpoint Service             disabled
PassiveID SPAN Service                 disabled
DHCP Server (dhcpd)                    disabled
DNS Server (named)                     disabled

tnetise02/admin#



 

The old ISE PAN was upgraded in a similar manner. Which came up as a new ISE SAN node.

Finally, the PAN role was flipped to make Node 1 PAN and Node 2 SAN, as they were before the upgrade.

 

Share this:


Tagged with:
| |

Leave a Comment

Be the First to Comment!

  Subscribe  
Notify of