Handle ReadTimeout during IOS install-mode upgrades and verify the requested image

[liacob]

I ran into this while upgrading a Catalyst 9200-48P to IOS XE 17.15.6 using PyNTC 2.0.0. The switch upgraded successfully, but install_os() raised a Netmiko ReadTimeout, causing our automation to report the upgrade as failed.

The error was:

netmiko.exceptions.ReadTimeout:
Pattern not detected: 'ROBS008E\#' in output.

The traceback pointed to this call inside install_os():

Pattern not detected: 'ROBS008E\#' in output.

Things you might try to fix this: 1. Explicitly set your pattern using the expect_string argument. 2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

. Traceback: Traceback (most recent call last): File "/usr/local/lib/python3.12/site-packages/naa/controllers/os_upgrader.py", line 53, in do_os_upgrade changed = pyntc_device.install_os( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pyntc/devices/ios_device.py", line 721, in install_os self.show(command, read_timeout=read_timeout) File "/usr/local/lib/python3.12/site-packages/pyntc/devices/ios_device.py", line 1069, in show return self._send_command(command, expect_string=expect_string, netmiko_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/pyntc/devices/ios_device.py", line 191, in _send_command response = self.native.send_command(command_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/netmiko/base_connection.py", line 111, in wrapper_decorator return_val = func(self, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/netmiko/utilities.py", line 667, in wrapper_decorator return func(self, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/site-packages/netmiko/base_connection.py", line 1841, in send_command raise ReadTimeout(msg) netmiko.exceptions.ReadTimeout: Pattern not detected: 'ROBS008E\#' in output.

Things you might try to fix this: 1. Explicitly set your pattern using the expect_string argument. 2. Increase the read_timeout to a larger value.

You can also look at the Netmiko session_log or debug log for more information.

The command being executed was:

install add file <filesystem><image> activate commit prompt-level none

This command performs the installation and reboots the switch. Netmiko waits for the normal CLI prompt, but that prompt may never return on the original SSH session once the switch starts rebooting.

PyNTC already handles IOError at this point and continues with reboot detection and image verification. However, Netmiko’s ReadTimeout is not an IOError, so it escapes that handler and stops the upgrade workflow before those checks run. The same gap is still present on develop.

Without the original session log, I cannot confirm exactly when the prompt stopped returning. What was confirmed on the device was that the upgrade had completed despite the exception.

Changes

• Handle ReadTimeout specifically around the install command that reboots the switch.
• Log a warning and continue through the existing reboot detection and image verification.
• Keep the requested image filename for the final verification instead of replacing it with packages.conf.
• Extend the existing tests to cover both IOError and ReadTimeout.
• Add recovery tests for a successful upgrade, a device returning on the wrong version, and a reboot timeout.

The timeout is not treated as proof of success. The device still has to pass reboot detection and image verification. The handler does not retry the installation or catch timeouts from unrelated commands.

Why Change Image Verification?

Previously, install-mode upgrades replaced image_name with packages.conf before calling _image_booted().

Both the old and new IOS XE versions can boot through packages.conf, so finding that filename in show version does not establish that the requested upgrade succeeded. This becomes particularly important when recovering from a timeout.

Keeping the requested filename allows the existing _image_booted() logic to extract the expected version and compare it with show version. The boot configuration still points to packages.conf; this change only affects verification.

Testing

Before the fix, the new ReadTimeout cases failed while the existing IOError cases passed.

The patch has not yet been tested on a physical switch. It also leaves the existing reboot detector unchanged, including its requirement that the device report an uptime below 10 minutes.