Upgrade Cisco ISE from 2.7 -> 3.1

How-To: Cisco Identity Service Engine

December 19, 2022

Updated on January 3, 2023

I recently upgraded an 8 node ISE appliance cluster (or “cube” as Cisco calls it) from 2.7 to 3.1. Below is a step-by-step guide to great success. Note that these instructions should be best practice for any version, regardless of being a VM or appliance box.

Summary

The Deployment

New Licensing Model

The Licensing model in 3.x changes from:

• Base

• Plus

• Apex

To:

• Essential

• Advantage

• Premier

The benefit of that is that you will only use either Essential, Advantage or Premier license per session, instead of having to use up to 3 licenses, like in the old model.

Picture from https://www.cisco.com/c/en/us/products/collateral/security/identity-services-engine/ise-licensing-guide-og.html#1MigratefromClassicLicensestoTierLicenses

The drawback is that they moved SGT provisioning to Advantage Level, when it used to only require a base license before. For more information on how to migrate licenses, or switch to smart licensing, follow the link below the picture above.

Note: I will make a separate post about registering ISE to a Cisco Smart Software Manager On-Prem later.

Upgrade Preparations

These will be covered in more detail further down in the post:

• Obtain a backup of the ISE configuration and operational data.

• Obtain a backup of the system logs.

• Disable scheduled backups. Reconfigure the backup schedules after deployment upgrade is complete.

• Export the certificates and private keys.

• Configure a repository. Download the upgrade bundle and place it in the repository.

• Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable. You may need this information to connect to Active Directory or RSA SecurID server after upgrade.

• Make a note of DNS, NTP & AD settings

• Purge the operational data to improve upgrade performance.

• Run the URT = Upgrade Readiness Test

From https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_010.html#id_15402

In addition, you should read the ISE 3.1 release notes to check that your current version is compatible with the ISE 3.1 upgrade.

Also make sure that you have some kind of console access to the ISE nodes, either through KVM terminal or Serial over LAN.

Choose the upgrade sequence

1. By default, the Secondary Administration Node is listed first in the upgrade sequence. After upgrade, this node becomes the Primary Administration Node in the new deployment.

2. The Primary Monitoring Node is the next one in the sequence to be upgraded to the new deployment.

3. Select the Policy Service Nodes and move them to the new deployment. You can alter the sequence in which the Policy Service Nodes are upgraded.
   You can upgrade the Policy Service Nodes in sequence or in parallel. You can select a set of Policy Service Nodes and upgrade them in parallel.

4. Select the Secondary Monitoring Node and move it to the new deployment.

5. Finally, select the Primary Administration Node and move it to the new deployment.

From https://www.cisco.com/c/en/us/td/docs/security/ise/2-4/upgrade_guide/b_ise_upgrade_guide_24/b_ise_upgrade_guide_24_chapter_010.html#id_15492

Detailed Step-by-step Preparations

Check Release notes for ISE 3.1

You might want to look for Supported Hardware, Upgrade Information and Known Limitations and Workarounds

Supported Hardware

Cisco ISE 3.1 can be installed on the following platforms:

• SNS-3595-K9

• SNS-3615-K9

• SNS-3655-K9

• SNS-3695-K9

Upgrade Information

You can directly upgrade to Release 3.1 from the following Cisco ISE releases:

2.6, 2.7 or 3.0

If you are on a version earlier than Cisco ISE, Release 2.6, you must first upgrade to one of the releases listed above, and then upgrade to Release 3.1.

This is from Cisco: “We recommend that you upgrade to the latest patch in the existing version before starting the upgrade.

Cisco ISE 3.1 has parity with 2.6 Patch 9, 2.7 Patch 4​, and 3.0 Patch 2.”

Known Limitations and Workarounds

This section provides information about the various known limitations and the corresponding workarounds.

Authentication Might Fail for SNMP Users After Upgrade due to Wrong Hash Value

If you are upgrading from Cisco ISE 2.7 or earlier release to Cisco ISE 3.1, you must reconfigure the settings for SNMP users after the upgrade. Use the following commands to reconfigure the settings for SNMPv3 users:

no snmp-server user <snmp user> <snmp version> <auth password> <priv password>
snmp-server user <snmp user> <snmp version> <auth password> <priv password>

https://www.cisco.com/c/en/us/td/docs/security/ise/3-1/release_notes/b_ise_31_RN.html

Setup a SFTP Linux Repository

ISE Configuration

From the ISE GUI:

Step 1: Navigate to Administration > System > Maintenance > Repository

Step 2: Create a new SFTP Repository

Repository Name: Example-Repo
Protocol: SFTP
Server Name: example-repo.com
Path: /full/path/to/sftp-dir
Enable PKI authentication: No
User Name: myusername
Password: XXXXXXXXXX

Note: After creating the repository, it will show up in the running configuration on all the ISE nodes:

repository Example-Repo
  url sftp://example-repo.com/path/to/sftp-dir
  user myusername password hash **********

This suggest that it should be possible to configure from CLI aswell. However, if configured from CLI, you have to add the repository manually to all nodes, and it won’t be visible in the GUI, therefore not recommended.

Step 3: Generate a keypair

Click on Generate Key pairs. When that is done you can click Export Public Key. That key will be used on the repository under Authorized_Keys

Note: You can generate keypair and export keys from the CLI but you need to specify a repository to do it. I realize that it looks like a catch-22 problem, if it is your first repository.

Step 4: Add the Linux Repository host key on the ISE nodes.

From the CLI, on all the ISE nodes, add the host_key for the linux Repository DNS and optionally an IP:

crypto host_key add host example-repo.com
crypto host_key add host 2001:DB8:1234:ABCD::50

copy running-config startup-config

Linux Repository Configuration

Step 1: Remove the # infront of the lines in this file:

sudo nano /etc/ssh/sshd_config

AuthorizedKeysFile      .ssh/authorized_keys
PubkeyAuthentication yes
RSAAuthentication yes # if it exists

sudo service sshd restart

Step 2: Edit the Authorized Keys in your home directory. Insert the exported key from ISE and copy it for every node

sudo nano ~/.ssh/authorized_keys

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-pan01 <--- Hostname of the ISE node

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-mnt01

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-psn01

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-psn02

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-psn03
ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-psn04

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-pan02

ssh-rsa
******************** SECRET KEY WITH ALOT OF RANDOM LETTERS ********************$
root@ise-mnt02

Backup & Restore

Backup

Backed up data is only used in case something bad happens during the upgrade. If you are lucky, you won’t be needing this data later.

From the PAN node:

First, save the running configuration

copy running-configuration startup-configuration

Backing up Configuration data

backup ISE_ConfigBackup_20221213 repository Example-Repo ise-config encryption-key plain XXXXXXXX

From the M&T node:

Backing up Operational Data

backup ISE_OperationalBackup_20221213 repository Example-Repo ise-operational encryption-key plain XXXXXXXX

On all the nodes:

Backup System Logs

backup-logs ISE_Logs20221213_ise-pan01 repository Example-Repo encryption-key plain XXXXXXXX

Backing up Certificates

1. From the GUI, go to Administration > System > Certificates. Choose the certificates you want to backup and click Export.

2. Export both Certificate and Private Key and set at passphrase.

Note: The interesting certificates are those issued from an internal or public CA. The self-singed certificates can just be replaced with new ones.

Note: One reason that URT can fail, is if the deployment have expired certificates in it’s store. I recommend cleaning up expired certificates before exporting.

Backing up CA Store

From the PAN node:

application configure ise

Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data
[4]Reset M&T Database
[5]Refresh Database Statistics
[6]Display Profiler Statistics
[7]Export Internal CA Store <---
[8]Import Internal CA Store
...
[0]Exit

7
Export Repository Name: Example-Repo
Enter encryption-key for export:

For importing CA store, choose 8 instead.

Restore

This part should not be necessary if upgrade works fine. But if you are really unlucky, and you need to reimage your whole deployment from scratch:

Importing configuration data

restore ISE_ConfigBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX

Importing configuration data with ADE-OS config (initial configuration like IP and DNS)

restore ISE_ConfigBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX include-adeos

Importing operational data (same command as configuration)

restore ISE_OperationalBackup_20221213 repository Example-Repo encryption-key plain XXXXXXXX

Note: If you have the Primary Administration Node (PAN) auto-failover configuration enabled in your deployment, disable this configuration before you restore a backup. You can enable the PAN auto-failover configuration after the restore is complete.

Disable scheduled Backups

From GUI:

Administration > System > Backup & Restore > Turn schedule status for configuration and operational data to disabled.

Purge Operational Data

On the M&T Nodes:

application configure ise

Selection configuration option
[1]Reset M&T Session Database
[2]Rebuild M&T Unusable Indexes
[3]Purge M&T Operational Data <---
[4]Reset M&T Database
…
[0]Exit

3

Enter number of days to be retained in purging MnT Operational data [between 1 to 90 days]
For instance, Entering 20 will purge MnT Operational data older than 20 days
Enter 'exit' to return to the main menu without purging
Enter days to be retained: 1
You are about to purge M&T data older than 1 from your database.
Are you sure you want to proceed? y/n [n]: y
M&T Operational data older than 1 is getting removed from database

Disable PAN auto-failover

Administration > System >Deployment > PAN Failover > Uncheck PAN auto failover

Make a note of Active Directory join credentials and RSA SecurID node secret, if applicable.

You may need this information to connect to Active Directory or RSA SecurID server after upgrade. Usually it reconnects fine after upgrade.

Make a note of DNS, NTP & AD settings

You can find these settings with a “show running-config” from CLI.

Run the Upgrade Readiness Test

Step 1: Go to the Cisco website, or search for ISE 3.1 download, and download the the upgrade readiness tool. In the same page where you can download ISE 3.1, scroll down to the bottom:

Step 2: Add the URT to the Repository

Step 3: Run the URT command from CLI on the Secondary PAN node

application remove urt <-- Cleans up old URT config.
application install ise-urtbundle-3.1.0.518-1.0.0.SPA.x86_64.tar.gz Example-Repo

Upgrade Procedure

After URT checks out OK, you can proceed with the upgrade. You can upgrade from both GUI and the CLI. However, In my experience, upgrading from CLI is more stable.

Upgrading from GUI:

• Pros: Automatic upgrade in a user-defined sequence

• Cons: Sometimes doesn’t work

Upgrading from CLI:

• Pros: More control, and more stable

• Cons: have to start upgrade manually on each node

Upgrade from CLI

On each node:

In case something wrong happened and you need to try again:

application upgrade cleanup

Upgrade commands:

application upgrade prepare ise-upgradebundle-2.6.x-3.0.x-to-3.1.0.518b.SPA.x86_64.tar.gz Example-Repo
application upgrade proceed

Note: These commands are recommended to be entered from a console connection

In my experience, upgrading a node takes around 1 – 2 hours, depending on what kind of persona it runs.

Upgrading the Secondary PAN and the Primary PAN nodes

Typically you will start upgrading the Secondary PAN node first, that will become the Primary PAN in the new deployment. It will also become the primary M&T node for the new deployment, until the Primary M&T node has been upgraded

Once the Primary M&T node has been upgraded, you can disable the M&T persona on the upgraded PAN node:

Administration > System > Deployment. Click on the PAN node and disable Monitoring

Upgrading the PSN nodes

When the new deployment have at least one PAN and one M&T node, PSN nodes can be added. If you don’t have that much load on the network, you can upgrade half the nodes at the same time, saving some time.

Note: Once the new PSN nodes comes up, they will start authenticating for the new deployment. Make sure that the first batch of PSN nodes are able to authenticate users and machines before proceeding with the next batch.

Upgrading the Secondary M&T node and the Primary PAN node

When you reach the point when it is only the secondary M&T and Primary PAN node left, upgrade cannot proceed on the M&T node, because there won’t be any M&T node left in the old deployment. Therefore you have to enable the M&T persona on the Primary PAN:

Administration > System > Deployment. Click on the PAN node and enable Monitoring

Then, you can proceed with upgrading the Secondary M&T node, and finish with the primary PAN.

Patching via CLI

I actually tried at first to patch via GUI, but it failed with an error message. So my stance is still that CLI is the best choice. Fortunately, patching takes way less time than upgrading and you can patch many nodes in parallel.

patch install ise-patchbundle-3.1.0.518-Patch4-22091704.SPA.x86_64.tar.gz Example-Repo

Patching Order

Patching order is a bit different from upgrade, although I’m not sure if it matters that much. My preferred order is as follows:

1. Primary PAN node (which is the Original Secondary PAN, unless you have changed it back already)

2. Primary M&T node and The first batch of the PSN nodes

3. The second batch of the PSN nodes, The Secondary M&T node and the Secondary PAN node.

Cleanup

When everything is complete, remember to:

• Activate PAN Auto-Failover

• Activate Scheduled Backups

• Activate Primary PAN node.

Manually Promote Secondary PAN to Primary

Step 1: Log in to the secondary PAN GUI.

Step 2: Go to Administration > System > Deployment.

Step 3: In the Edit Node window, click Promote to Primary.

Step 4: Click Save.

From https://www.cisco.com/c/en/us/td/docs/security/ise/3-0/admin_guide/b_ISE_admin_3_0/b_ISE_admin_30_deployment.html#reference_4E8632DBFC474C7787EC3AE18A940677

Where did the Live Logs go?

Update January 3, 2023

I noticed that after the ISE 3.1 upgrade, the RADIUS and TACACS live logs were missing, although the authentications are working. Apparently you have to disable “Use “ISE Messaging Service” for UDP Syslogs deliver to MnT” under Administration > System > Logging > Log Settings.

Thank god for bloggers: https://bohemiangrove.co.uk/cisco-ise-live-logs-broken/

The Next Steps…

Migrate to new licensing model

Now we have to migrate to use the new 3.x licenses, at least within the next 90 days. I recommend this link from Cisco for instructions on how to do that.

Upgrade CIMC

Since this upgrade was performed on hardware appliance boxes, it is recommended to also upgrade firmware and BIOS on the CIMC = Cisco Integrated Management Controller interface. That procedure will be covered in a later post