### Human: Can you tell about HDD or SSD disk troubleshooting. ### Assistant: This article describes how to troubleshoot a failed disk and identify which component is causing the failure. If you have received an alert for a failed disk, you can troubleshoot a specific node instead of examining an entire cluster.. When a drive is experiencing recoverable errors, warnings or a complete failure, the Stargate service marks the disk as offline. If the disk is detected to be offline 3 times within the hour, it is removed from the cluster automatically and an alert is generated (KB 4158 or KB 6287).

If an alert is generated in Prism, the disk must be replaced. Troubleshooting steps do not need to be performed.

If a failed disk is encountered in a Nutanix Clusters on AWS, once the disk is confirmed to have failed proceed to condemn the respective node. Condemning the affected node will replace it with a new bare metal instance of the same type.. Once the disk is replaced, an NCC health check should be performed to ensure optimal cluster health. However, if an alert is not generated or further analysis is required, the steps below can be used to troubleshoot further.Before you begin troubleshooting, verify the type of HBA controller.Caution: Using the SAS3IRCU command against an LSI 3408 or higher HBA can cause NMI events that could lead to storage unavailability. Confirm the HBA controller before using these commands. To determine what type of HBA is used, look for the controller name located in /etc/nutanix/hardware_config.json on the CVM.Example of the output when SAS3008 is used.  In this case, the command SAS3IRCU is the correct command to use. Note the "led_address": "sas3ircu:0,1:0" line:

"node": {

Example of the output when SAS3400/3800 (or newer) is used. In this case, using SAS3IRCU would be ill-advised. Use the storcli command instead. For information on StorCLI refer to KB-10951. Note "led_address": "storcli:0" line.

"node": {

Identify the problematic disks

	Check the Prism Web console for the failed disk. In the Diagram view, you can see red or grey for the missing disk.Check the alerts in the Prism Web console for the disk alerts, or use the following command to check for disks that generate the failure messages.
		"node": {
		Check if any nodes are missing mounted disks. The two outputs should match numerically.
		Check the disks that are mounted on the CVM (Controller VM).
			"node": {
			Check the disks that are physical in the CVM.
			"node": {
			Check if the status of the disks is all Online and indicated as Normal.
			"node": {

		Validate the expected number of disks in the cluster.
		"node": {

		The output of the command above should be the sum of the outputs of steps 1c.i and 1c.ii.

		There are instances where the number can be higher or lower than expected. So, it is an important metric that can be compared to the disks listed in step 1b.
		Look for extra or missing disks.
		"node": {
		Check that all disks are indicated as mounted rw (read-write) and not ro (read-only).
		"node": {

	Identify the problems with the disks nodes
	Orphaned disk ID
		This is a disk ID that the systems no longer use but was not properly removed. Symptoms include seeing an extra disk ID listed in the output of ncli disk ls.

		To fix the orphaned disk ID:

		"node": {

		Ensure that you validate the disk serial number and that the device is not in the system. Also, ensure that all the disks are populating using lsscsi, mount, df -h, and counting the disks for the full-disk population.
		Failed disk and/or missing disk
		A disk becomes failed and/or is no longer visible to the system for several reasons. There are two approaches to solving a disk problem. The top-down approach is the software representation of the disk down to the hardware/physical layer. The bottom-down approach documents the investigation of a disk from a bottom-up perspective since the 'top' layer can be relative to the problem reported by the software layer (Prism, Hades, Stargate, etc.).

		Check if the disk is visible to the controller as it is the device whose bus the disk resides on.

		lspci - displays the PCI devices seen by the CVM.

			NVME device - Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01).SAS3008 controller - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02) - LSI.SAS2008 controller - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03).SAS2308 controller (Dell) - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05).MegaRaid LSI 3108 (Dell) - RAID bus controller: LSI Logic / Symbios Logic MegaRAID SAS-3 3108 [Invader] (rev 02).LSI SAS3108 (UCS) - Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS3108 PCI-Express Fusion-MPT SAS-3 (rev 02).
			lsiutil - displays the HBA (Host Bus Adapter) cards perspective of the ports and if the ports are in an UP state. If a port is not up, either the device on it does not respond, or the port or connection to the device is bad. The most likely scenario is the device (disk).
			"node": {
			lsscsi - lists the SCSI bus devices seen that include any HDD or SSD (except NVME, which does not pass through the SATA controller).sas3ircu - reports slot position and disk state. It is useful for missing disks or verifying that disks are in the correct slot. (Do NOT run the following command on Lenovo HX hardware as it may lead to HBA lockups and resets)
			"node": {
			storcli - reports slot position and disk state. Commonly found on Lenovo Chassis but will be included on NX G8 platforms.
			Slot # (should match the physical slot)

			State is the state of the disk. New disks will be listed as Available (AVL). After bootup and SCSI inquiry healthy disks will show as Ready (RDY).			In the example below, a disk was detected but was never marked as Ready (RDY). It remains in Available (AVL) status due to a bad disk. In the next section, we see the related mpt3sas dmesg output showing the SCSI inquiries constantly polling this disk.  			(Note that the Serial No has been truncated and replaced by the Firmware Revision LC72 repeated for this platform. This was consistent for the systems in this particular cluster. This does not indicate a problem, simply a unique behaviour.)

			"node": {
			Check the CVM's dmesg for LSI mpt3sas messages. We should typically see one entry for each physical slot. (The below example shows SAS address "0x5000c5007286a3f5" is repeatedly checked due to a bad/failed disk. Note how the other addresses are detected once, and the suspect is repeatedly being polled.)
			"node": {
			smartctl - if Hades indicate that a disk is checked by smartctl 3 times in an hour, it is automatically failed.
			"node": {

			See KB 8094 for troubleshooting with smartctl.
			Check for offline disks using NCC check disk_online_check.
			"node": {

			See KB 1536 for further troubleshooting offline disks.
			Check if NCC has any errors indicated for disks, mount paths being offline, or failed disks in the system.Confirm if disks are seen from LSI Config Utility:
			Enable LSI HBA OPROMGo to “Advanced-->PCIe/PCI/PnP Configuration”, change CPU1 SLOT2 PCI-E 3.0 x8 OPRM to “Enabled” as below showing,Press "F4"Enter LSI Config Utility on bootup:

				Press <Enter> twice to confirm if disks are seen or not:

			There can be a case where the disk is DOWN in lsiutil, usually after a replacement or an upgrade of the disks. When all the above checks are carried out, and the disk is still not visible, compare the old and new disk "disk caddy or tray". Ensure the type is the same. There can be cases where an incorrect disk type is dispatched, and it does not seat properly in the disk bay hence not being detected by the controller.

Identify the node type or the problematic node

	Run ncli host ls and find the matching node ID.

	Specific node slot location, node serial, and node type is important information to document in case of recurring issues. It also helps to track the field issues with the HBA's, node locations, and node types.
	Identify the failure occurrence
	Check the Stargate log.
		The stargate.INFO log for the corresponding period indicates if Stargate saw an issue with a disk and sent it to the Disk Manager (Hades) to be checked or had other errors accessing the disk. Use the disk ID number and serial number to grep for in the Stargate log on the corresponding node the disk is in.
		The Hades log contains information about the disks it sees and the health of the disks. It also checks which disk is metadata or Curator disk and selects one if one did not already exist in the system or was removed/disappeared from the system. Check the Hades log.Check df -h in /home/nutanix/data/logs/sysstats/df.INFO to see when the disk was last seen as mounted.Check /home/nutanix/data/logs/sysstats/iostat.INFO to see when the device was last seen.Check /home/log/messages for errors on the device, specifically using the device name, for example, sda or sdc.Check dmesg for errors on the controller or device. Run dmesg | less for the current messages in the ring, or look at the logged dmesg output in /var/log.
	Identify the reasons for disks failure
	Check when the CVM was last started if the disk's last usage data were not available. Again, reference the Stargate and the Hades logs.Check the Stargate log around the time of disk failure. Stargate sends a disk to Hades to check if it does not respond in a given time and ops timeout against that disk. Different errors and versions represent it differently, so always search by disk ID and disk serial.
	Check the count of disk failure
	If a drive failed more than once in this slot and the disk was replaced, it would indicate a potential chassis issue at that point.
	Check if lsiutil is showing errors.
	If lsiutil shows errors evenly on multiple slots, it indicates a bad controller.

Sample command output for reference:

"node": {

On a Nutanix Clusters on AWS:

"node": {

To run the commands in a Hyper-V environment, use winsh on the CVM:

"node": {

"node": {

NOTE: For ID: 191, G-Sense_Error_Rate for Seagate HDD's can be safely ignored unless there is performance degradation. G-Sense_Error_Rate value only indicates HDD adapting to shock or vibration detection. Seagate recommends not to trust these values as this counter dynamically changes the threshold during runtime.

"node": {. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Pre-upgrade Check: AOS Upgrade (test_sha_mac_ssh_config). ### Assistant: This pre-upgrade check runs during an AOS upgrade. It confirms that no hosts on your cluster are configured to only accept deprecated SSH ciphers, and flags any such hosts.. test_sha_mac_ssh_config is a pre-upgrade check that runs during AOS upgrades. It checks whether your hypervisor is configured to use the proper SSH ciphers. Starting with AOS 5.15.2 and 5.17.x, we start to enforce the use of sha2 ciphers for SSH and without your host configured to accept this cipher the upgrade will stall on the affected host. You need to correct this problem before starting the upgrade.Error messages generated on UI by this check:
Host(s) : [u'x.x.x.x'] configured to only accept deprecated SSH ciphers, Please refer KB 9730
On any CVM, you can run the following command to confirm which SSH ciphers are being accepted by your hypervisor

nutanix@cvm$ hostssh 'grep "^MACs" /etc/ssh/sshd_config'
The expected value currently for hosts with CVMs running AOS 5.15.2+ and 5.17.1+ are MACs hmac-sha2-256,hmac-sha2-512,hmac-sha1. Manually updating the sshd config:

On each host which is reported by the pre-check, do the following

SSH to the hypervisorEdit the /etc/ssh/sshd_config file with vi or other text editorFind the line which starts with "MACs" such as this
	MACs hmac-sha1,hmac-sha1-96
	Change the line to reflect the newer SSH ciphers:
	MACs hmac-sha1,hmac-sha1-96

 . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about How to obtain the SEL output from the IPMI or CLI. ### Assistant: The SEL (System Event Log) output is an indispensable asset in countless scenarios, serving as a vital resource for pinpointing the underlying causes of issues, devising effective solutions, and conducting thorough hardware diagnostics. This article will provide comprehensive guidance on retrieving this information through the user-friendly web GUI and the command line interface.. Frequently, when troubleshooting hardware issues it becomes necessary to obtain the SEL or System Event Log output. Nutanix support has the option to retrieve this information from Insights. However, there are instances where Pulse is not enabled or the command fails to run successfully. This article aims to provide customers with a set of instructions to obtain the SEL output either through the webUI (IPMI) or the command line (CLI).. Navigating to the IPMI from Prism Element:

Prism Element > Hardware > Diagram > select the Node for the IPMI you want to access > IPMI should be hyperlinked

You can also obtain the IPMI address from the CLI with the following command:

nutanix@cvm:~$ panacea_cli show_all_ips

Note: If you need to update or change your password to the IPMI, click here for instructions how.

Option 1: Downloading the SEL output from the WebUI

In G5, G6 and G7 IPMI, you will find the SEL output under Server Health > Health Event Log

To download the output as a .csv file you will click 'Save' highlighted in red.

In G8 IPMIs, the SEL output can be found by expanding System > Health Event Log

To download the output as an excel file you will click 'Export to Excel' highlighted in red

Option 2: Gathering the SEL output from the CLI

AHV

nutanix@cvm:~$ panacea_cli show_all_ips
ESXi

nutanix@cvm:~$ panacea_cli show_all_ips
Hyper-V

nutanix@cvm:~$ panacea_cli show_all_ips. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Pre-check: test_is_nos_upgrade_supported_on_esx. ### Assistant: The pre-check test_is_nos_upgrade_supported_on_esx verifies if the version of ESXi hosts on the cluster is supported on the target AOS version.. test_is_nos_upgrade_supported_on_esx check is a pre-upgrade check that verifies if the version of ESXi hosts on the cluster is supported on the target AOS version.

The pre-check may fail if the ESXi version is lower than 6.7.0 and the target AOS version for the upgrade is 5.18 or above for STS or 5.15.3 or above for LTS, with the failure message below:

Host x.x.x.x is running on ESXi version 6.5.0 which is lower than supported version 6.7.0. Upgrade host, Please refer KB 9646. Make sure the ESXi version is 6.7.0 or above before upgrading AOS to 5.18 or above for STS or 5.15.3 or above for LTS.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: ahv_remote_log_forwarding_check. ### Assistant: NCC 4.1.0 ahv_remote_log_forwarding_check detects issues with AHV host log forwarding to the configured remote server.. The NCC health check ahv_remote_log_forwarding_check detects whether log forwarding to remote server is operating normally.This check runs on only AHV, but also only if the cluster has had additional STIG DoDIN APL enablement performed on supported AOS/AHV versions. By default, and unless specific steps have been taken to enable this functionality on-demand, this check will return a PASS result.When remote log forwarding is needed to forward the AHV host logs to a remote machine, the host itself records the failure or success of sending these records to the remote machine (configured by the user by following the Portal Guide; Acropolis Advanced Administration Guide \ Configuring the Remote Syslog Server Settings). This status is maintained in an rsyslog stats file (/var/log/rsyslog-stats) on each AHV host. The records of success/failure is updated regularly and this health check parses the recent entries in that file to recognise any failures and trigger any alert workflows configured on the cluster such as email, syslog, SNMP trap, Prism Alert notifications.It can be run as part of the complete NCC check by running:
nutanix@cvm$ ncc health_checks run_all
 or individually as:

nutanix@cvm$ ncc health_checks run_all
 You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.This check runs on each CVM to assess its AHV host. It does not run on Prism Central.This check is scheduled to run every 5 minutes and produces a WARN alert after 3 consecutive failures.Sample Output:Check Status: PASS

nutanix@cvm$ ncc health_checks run_all
Check Status: WARN

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID111088DescriptionCheck log forwarding to remote server.Causes of failureForwarding logs to remote server is failing.ResolutionsCheck failure causes for forwarding the logs to remote server. Refer KB10811.ImpactRemote server will not be receiving log output from cluster.Alert IDA111088Alert TitleForwarding logs to remote server is failing.Alert MessageForwarding logs to remote server is failing on host {host_ip}.
 . A warning produced by this check could indicate one or more of the following possible conditions:
Network connectivity issue between the AHV host and the remote rsyslog serverAHV host uplink or local network issueWAN/VPN/inter-site routing issue between Nutanix AHV cluster and remote rsyslog serverFirewall/ACL blocking AHV host communication to the remote rsyslog serverRemote rsyslog server offline/powered off, services failed, storage fullRemote rsyslog server unable to handle incoming volume of messages from one or more AHV hosts, or other rsyslog clients in addition to the AHV hosts.
From the affected AHV host(s), consider using the following commands/utils to help isolate the issue;

Ping the configured remote rsyslog server to expose any basic switching/routing issues.Use NCAT to test a protocol:port specific connection is able to be opened (or blocked by firewall/ACL)

[root@AHV ~]# nc -zvu -w3 <remote_syslog_server_IP_FQDN> <remote_port_usually_514>

Investigate the contents of the /var/log/rsyslog-stats log file for any entries around the timestamp reported by this NCC check/alert.

[root@AHV ~]# nc -zvu -w3 <remote_syslog_server_IP_FQDN> <remote_port_usually_514>
In case the above mentioned steps do not resolve the issue, consider engaging Nutanix Support at https://portal.nutanix.com/To speed up the resolution time and minimize possible impact, collect and attach the following information to the support case:

A complete NCC report:

[root@AHV ~]# nc -zvu -w3 <remote_syslog_server_IP_FQDN> <remote_port_usually_514>

A log bundle generated from the cluster. This can be collected through Prism Web Console's health page. Select Actions > Collect Logs. Logs can also be collected through command line using logbay (KB 6691 - NCC - Logbay Quickstart Guide):

[root@AHV ~]# nc -zvu -w3 <remote_syslog_server_IP_FQDN> <remote_port_usually_514>

A copy of the /var/log/rsyslog-stats log file from any affected AHV host.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about CVM services won't start after the Hardware replacement for VMware NSX Enabled ESXi servers. ### Assistant: Customer has VMware Infrastructure and using the SDN (Software Defined Networking) solution NSX-V for DFW (Distributed Firewall) or Guest Introspection features. When there is hardware replaced like a DIMM or SATADOM, VMware Administrator remove the ESXi host from NSX solution which generally removes the CVM from the exclusion list in the NSX solution. After the hardware part is replaced, we notice the communication between CVM & ESXi via 192.168.x.x network is broken and services can't start with ssh key based messages in the genesis.out log file.. CVM services failed to start with the below symptoms:The /home/nutanix/data/logs/genesis.out will show the following:
nutanix@cvm:~$ tail -f genesis.out
Running the fix_host_ssh fails to fix this issue:
nutanix@cvm:~$ tail -f genesis.out
CVM failed to communicate via the eth1 network interface to the ESXi host.
nutanix@cvm:~$ tail -f genesis.out
The steps below steps do not resolve the issue:
Re-create the vmk1 and check the communicationRe-create vSwitchNutanix but failed to communicateInject the Static route via the arp commandReboot the CVM and enabled/disabled the NIC2 for CVM - tried network services restartReboot the ESXi host. Confirm that CVM is part of the exclusion list:Procedure:

Navigate to Exclusion List settings.
	In NSX 6.4.1 and later, navigate to Networking & Security > Security > Firewall Settings > Exclusion List.In NSX 6.4.0, navigate to Networking & Security > Security > Firewall > Exclusion List.
	Click Add.Move the VMs (CVM with services issue) you want to exclude to Selected Objects.Click OK.
Reference. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Windows Server 2008 R2 VM on non-AHV hypervisor boots into recovery mode when VirtIO is uninstalled. ### Assistant: Uninstalling VirtIO 1.1.6 or higher from Windows Server 2008 R2 VM on non-AHV hypervisor makes Windows boots into recovery mode.
Also, Migrating Windows Server 2008 R2 VM by Nutanix Move 4.2.1 or higher from ESXi/Hyper-v makes the source VM boots into recovery mode.. Uninstalling VirtIO 1.1.6 or higher from Windows Server 2008 R2 VM on non-AHV hypervisor makes Windows boots into recovery mode.Also, Migrating Windows Server 2008 R2 VM by Nutanix Move 4.2.1 or higher from ESXi/Hyper-V makes the source VM boots into recovery mode.. Perform the following steps to recover the VM:
Press F8 key during the VM powering up to show Windows advanced boot options.Select "Disable Driver Signature Enforcement".	Windows boots up normally.Start Registry Editor. Click on the Start button and search for regedit.exe.Delete "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\vioscsi" branch.Reboot Windows and confirm Windows boots up normally.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about On mixed-hypervisor cluster with AHV storage-only nodes, VM management fails with error "Error occurred while getting network list". ### Assistant: VM management in a mixed hypervisor cluster fails to due to a missing entry in /etc/nutanix/hardware_config.json on AHV storage-only nodes. VM create or update operations on mixed ESXi+AHV clusters may fail with the following error:

Error occurred while getting network list

This issue can typically occur after cluster expansion and adding AHV storage-only nodes.

the following error can be found in the /home/nutanix/data/logs/uhura.out log:

Error occurred while getting network list

The minimal_compute_node field is missing in the /etc/nutanix/hardware_config.json file on the storage-only CVMs:

Error occurred while getting network list

To check all nodes, run the following command:

Error occurred while getting network list. Starting from AOS 6.1 it is possible to select HCI or Storage-only node ензу while expanding a cluster. HCI to Storage-only or Storage-only to HCI node conversion is taken care of by the Expand cluster operation itself. Remove affected nodes from the cluster and re-add them while selecting the correct node type.In case removing the node is not possible due to space constraints, consider engaging Nutanix Support. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A130348 - VmInPausedState. ### Assistant: NCC 4.2: This Nutanix article provides the information required for troubleshooting the alert "VM is in Paused state" for your Nutanix cluster.. This Nutanix article provides the information required for troubleshooting the alert "VM is in Paused state" for your Nutanix cluster.Alert OverviewThe "VM is in Paused state" alert is generated when the Acropolis service detects a VM in Paused state. This is an unexpected VM state which should not be seen during daily operations.Sample Alert
Warning: VM found in paused state.
Potential ImpactVM is not operational.Output Messaging
Check ID130348DescriptionVM found in paused state.Cause of FailureFailed migration or some hypervisor issue.ResolutionsPower cycle the VM to get it out of this state.ImpactVM in paused state most likely means downtime and it's in an irrecoverable state.Alert Smart TitleVM {vm_name} with ID {vm_uuid} has been found in paused stateAlert TitleVM is in Paused stateAlert MessageVM {vm_name} with ID {vm_uuid} has been found in paused state. TroubleshootingPower Off the affected VM and then power it On to recover. Contact Nutanix Support at https://portal.nutanix.com as soon as possible to fully troubleshoot the problem. Collect additional information and attach them to the support case.

Collecting Additional Information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB-2871.Collect the NCC output file ncc-output-latest.log. For information on collecting the output file, see KB-2871.Collect Logbay bundle using the following command. For more information on Logbay, see KB-6691.

nutanix@cvm$ logbay collect --aggregate=true

If the logbay command is not available (NCC versions prior to 3.7.1, AOS 5.6, 5.8), collect the NCC log bundle instead using the following command:

nutanix@cvm$ logbay collect --aggregate=true
Attaching Files to the CaseAttach the files at the bottom of the support case on the support portal.If the size of the NCC log bundle being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations. See KB-1294.Requesting AssistanceIf you need further assistance from Nutanix Support, add a comment to the case on the support portal asking Nutanix Support to contact you. If you need urgent assistance, contact the Support Team by calling one of our Global Support Phone Numbers. You can also click the Escalate button in the case and explain the urgency in the comment, and Nutanix Support will be in contact.Closing the CaseIf this KB resolves your issue and you want to close the case, click on the Thumbs Up icon next to the KB Number in the initial case email. This informs Nutanix Support to proceed with closing the case. You can also update the support case saying it is okay to close the case and Nutanix Support will close the case.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Pre-upgrade check: Hypervisor Upgrade (test_host_upgrade_versions_compatible). ### Assistant: This pre-upgrade check runs during hypervisor upgrades. It checks if the base and target ESX versions are upgrade compatible.. test_host_upgrade_versions_compatible is an ESXi pre-upgrade check that runs during hypervisor upgrade. It checks whether the base and target ESXi versions are upgrades compatible. The reason for this check is that ESXi introduced a new certificate for VIB signing into the trust store in 2018 to begin the transition from the old certificate (which expired on 31st December 2019).

Error messages generated on UI by this check:

"Current host version %s and the target host version %s are not compatible for upgrade". VMware recommends upgrading ESXi hosts to a minimum of the builds (listed below), which were released after March 2018 and have the newer signing certificate in the trust store. Doing so prevents future installation and upgrade failures due to the untrusted signing certificate:
6.0 ESXi600-201807001 (Build: 9239799) or later 6.5 U2 (Build: 8294253) or later6.7 GA (Build: 8169922) or later.

The upgrade should be retried using a compatible ESXi release (above). Once upgraded to a compatible build, you may upgrade to the target release version. For all ESXi build numbers and versions, refer to VMware KB 2143832.VMware has addressed this issue in VMware KB 76555.For any queries consider engaging Nutanix Support.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Nutanix Files - File and folder permissions. ### Assistant: Configure Nutanix Files Admin and default permissions for a home and general purpose shares.. It may be difficult to determine user permissions without access to Active Directory (AD). If there are issues with share access or domain joining because of permissions, and you are certain that there are no issues with connectivity, follow the steps below to verify if the logged-in user has AD user privileges.. Execute the following command on Windows SMB Client to retrieve a list of groups of which the logged-in user is a member. Any user that is a member of the BUILTIN\Administrators group (most of the time, Domain Admins) should be able to access the share.

PS C:\Windows\system32> whoami /groups /FO list

Sample output:

PS C:\Windows\system32> whoami /groups /FO list

Default Permissions for Nutanix Files SharesThere are two share types: distributed and standard. In the example output below, nufs-sec-smb1 is a distributed share while nufs-sec-smb2 is a standard share.

PS C:\Windows\system32> whoami /groups /FO list

Look at the default permissions of these shares (from the client machine). In the example output below, the named directory UserJohn is the TLD (top-level directory) at the root of the distributed share.

PS C:\Windows\system32> whoami /groups /FO list

PS C:\Windows\system32> whoami /groups /FO list

PS C:\Windows\system32> whoami /groups /FO list

The same permissions can be checked using icacls or Explorer.

In Nutanix Files 3.0 and later, a user or a group can be added as a File Server Admin/Backup Admin in Files user interface. Alternatively, use the ncli fs add-admin-user command.

Get files/directories authorization information.From Nutanix Files version 3.8 onwards there is a new CLI introduced to query files and directories authorization information.

PS C:\Windows\system32> whoami /groups /FO list
In the following example, a user named "Admin" is unable to access the folder "users" in the share "Eng". Use the following steps in order to validate the permissions or authorization for a user named "Admin" in the folder "users".1. SSH into any FSVM2. List current ACL set on a share and folder. In this example, it is share "Eng" and folder "users". In the output below, AD user "Admin" has set explicit DENY ACL in the "users" folder.

PS C:\Windows\system32> whoami /groups /FO list. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Migration of resource heavy VMs might get stuck on AHV clusters. ### Assistant: On AHV clusters, migration of large VMs might fail. In the Prism web console, you can see the migration progress up to 99%, and then the migration stops indefinitely.. VM live migration can be manually initiated or be a result of putting the host into maintenance mode (to perform firmware or hypervisor upgrade, for example). Live migration of the VM that has a high rate of change to its memory contents may take a significant amount of time or look "stuck" and the process eventually fails. This can happen on a cluster where 1G and 10G uplinks are configured. In both cases, this happens when the change rate of the VM memory exceeds the network bandwidth.

Prism will generate an alert when the migration task fails. Furthermore, sample errors can be found in the acropolis.out log on the Acropolis leader. Follow the steps from KB 2305 to find the Controller VM (CVM) with the Acropolis leader role.

The following error messages are commonly seen in the acropolis.out log on the Acropolis leader on clusters configured with 1G uplinks:

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error

The following error messages are commonly seen in the acropolis.out log on the Acropolis leader on clusters configured with 10G uplinks:

2020-05-08 14:24:03 ERROR enter_maintenance_mode_task.py:887 Failed to evacuate 18657db7-1f76-48e3-a7b9-ea1311423d28: HypervisorError: Src:[operation failed: migration job: Not converging after 50 attempts at 99%.]; Dest:[internal error: qemu unexpectedly closed the monitor: 2020-05-08T20:24:02.501332Z qemu-kvm: load of migration failed: Input/output error. Perform the following steps to investigate the issue:

Check maximum available NIC bandwidth and load balancing mode:
	Run the following commands on CVM:
		nutanix@cvm:~$ manage_ovs show_uplinks
		In the output of commands mentioned above, check which NICs are assigned to br0 bridge and their speed:
		nutanix@cvm:~$ manage_ovs show_uplinks

	Check VM migration progress by running the following command multiple times with a delay of 1-2 minutes on the source AHV host:
	nutanix@cvm:~$ manage_ovs show_uplinks
	To identify the network bandwidth utilization, you can check the sar stats for the AHV source host.
	For example, where VM migration failed on 1st Feb at 08:43 AM, the following command can be run on the sar file. The command is filtering occurrence where physical NIC is transferring more 8Gb per second of data.

	nutanix@cvm:~$ manage_ovs show_uplinks

	Starting from AHV 20201105.2159, in /var/log/libvirt/qemu/<vm_uuid>.log on the source host, you can also see the memory dirty rate of the migrated VM which is helpful when troubleshooting failures:

	nutanix@cvm:~$ manage_ovs show_uplinks

For further troubleshooting, consider checking /home/data/logs/acropolis.out on the Acropolis leader. KB 7949 describes the normal VM Live Migration Process and can be used as a reference.

Workarounds to consider:

Power off memory-intensive VMs and use acli to start it on the target AHV host instead of live migrating the VM. Run the following command on any CVM in the cluster to start VM on a specific AHV host:
	nutanix@cvm:~$ manage_ovs show_uplinks
	Schedule the VM migration or maintenance at a time when the VM is as idle as possible.Use faster uplinks if available.Verify that the network uplinks are not degraded.

Note: Use of LACP load balancing mode may not improve the situation as a single TCP connection can only use the bandwidth of a single NIC.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Lazan service restarts unexpectedly due to non-ASCII characters in VM disk names. ### Assistant: Lazan service restarts frequently due to non-ASCII characters being present in the VM name/VMDK files on the cluster after upgrading to AOS 6.5.x or 6.0.x. Lazan service restarts frequently due to non-ASCII characters in the VM name/VMDK files on the cluster after upgrading to AOS 6.5.x or 6.0.xThe following alert is generated:
One or more cluster services have restarted within 15 minutes in the PCVM/CVM, the total restarts being >= 10. The following are the service(s) that have restarted : ['lazan'].
The Following CRITICAL error signature can be seen in the /home/nutanix/data/logs/lazan.out logs:

One or more cluster services have restarted within 15 minutes in the PCVM/CVM, the total restarts being >= 10. The following are the service(s) that have restarted : ['lazan'].. This issue is resolved in:
AOS 6.5.X family (LTS): AOS 6.5.3
Please upgrade AOS to versions specified above or newer.Engage Nutanix Support at https://portal.nutanix.com/ to recover the cluster.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about LCM Log Collection Utility. ### Assistant: Overview and instructions on using LCM Log Collection Utility..  PurposeLCM (Life Cycle Manager) Log Collection Utility provides an independent script designed to collect all the important log files necessary to debug any LCM issue.

NCC log_collector bundle is significant in size and contains multiple files that are irrelevant to LCM. In contrast, Log Collection Utility collects only the files that are relevant to LCM workflow thereby reducing the log bundle size and speeding up the process of log collection.The utility is available in LCM version 2.2 or later by default. It could also be used on earlier versions of LCM.Functionality

Collects LCM log files.Detects if the node is stuck in Phoenix and collects kernel logs.Collects logs for the services with which LCM interacts (Example: Foundation, Prism, etc.).The utility state is independent of such of the LCM hence it is capable of collecting logs even if the framework is down.Parses LCM configurations from Zookeeper nodes.

CaveatsThe script cannot pull logs from the node booted into Phoenix when the node's IP address is not reachable over the network. In that case, apply the respective CVM (Controller VM) IP address to the Phoenix instance using IPMI following KB 5346 before running the script. Once the network is restored, /tmp/phoenix.log should also be collected from phoenix for troubleshooting purposes.Files and information collected

Node level information
	LCM leader
	Logs from CVM
	LCM leader
	Command output from Phoenix
	LCM leader
	. For LCM 2.3.1 or later (automatic log collection)

LCM 2.3.1 automatically triggers the LCM log collector utility when any LCM operation fails.Log bundle will be created in /home/nutanix/data/log_collector/.For LCM 2.3.1.1 and later, LCM prints Log location along with the failure message as below. Log into the CVM IP to fetch the LCM log bundle and upload it to the Support case for review.

Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).

For LCM 2.2.x or later (manual log collection)

It can be used to trigger a new log collection.Run the following command from a CVM to begin log collection.

Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).

Sometimes you might  see below error message :

Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).

If you get the above error, please look for the file python lcm_log_collector.pyc instead and run the command shown below:

	Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).

Log bundle will be created in /home/nutanix directory on the node from which the script is executed, with the name in the following format:

Operation failed. Reason: LCM operation kLcmUpdateOperation failed on phoenix, ip: [xx.xx.x.xx] due to Multistage upgrade encountered an error: ('The read operation timed out',).

For LCM 2.2.x, log bundle will be created in /home/nutanix/data/logs/lcm_logs/.

For LCM 2.3 and later, the log bundle will be created in /home/nutanix/data/log_collector/.

For Uploading the logsOnce the logs are collected refer to KB-1294 for different methods to upload the LCM logs to the case.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: hwclock_check. ### Assistant: The NCC health check hwclock_check verifies hardware clock functionality and any time differences.. The NCC Health Check hwclock_check checks if the hardware clock is healthy. It checks the date from the real-time clock (hardware clock) on the hypervisor against the date of the Controller VM (CVM).Time differences in the hardware clock and system clock can cause problems. The hardware clock operational check ensures that the hardware clock is functional.

Running the NCC CheckYou can run this check as part of the complete NCC Health Checks:

nutanix@cvm$ ncc health_checks run_all

Or you can run this check separately:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every 10 minutes, by default.

This check will generate an alert after 5 consecutive failures across scheduled intervals.

Sample output
For Status: FAIL

nutanix@cvm$ ncc health_checks run_all
For Status: PASS

nutanix@cvm$ ncc health_checks run_all

Output messaging

DescriptionCheck if the Hardware clock has failedCauses of failureThe RTC clock on the host has failed, or the RTC battery has died.ResolutionsReplace the node. Refer to the Nutanix documentation for instructions.Alert TitleHardware Clock FailureAlert MessageHardware clock in host host_ip has failed.

Note: This hardware-related check executes on the below hardware

Nutanix NXDell XCHPELenovo HXNutanix SXFujitsu XFLenovo SRPoweredgeIntelIBMHyveVoyager TDC. This NCC check or alert for the hardware clock can be raised for the following reasons:
Network failure between the NTP server and the host.The system clock and hardware clock on the host are incorrect.Incorrect time on the NTP server.The RTC is powered by the same battery as the CMOS. This battery may have expired, or the RTC clock on the host has failed.

If this check fails, consider engaging  Nutanix Support. Additionally, gather the output of ncc health_checks run_all and attach it to the support case. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about PC Cluster Runway/Scenarios not working on NCC 3.7.1/3.7.1.x. ### Assistant: Due to ENG-225071, cluster runway or new scenarios may stop working when you upgrade to NCC 3.7.1 or 3.7.1.x prior to upgrading PC. You can work around the issue by reinstalling NCC.. Scenario#1 Due to a known issue, Prism Central or Cluster Runway scenarios may stop working. This happens when a customer upgrades to NCC 3.7.1 or NCC 3.7.1.1 in advance before upgrading Prism Central to PC 5.10.3.x. You may notice the following symptoms:

Cluster Runway is showing N/a instead of data.Cluster Runway shows, "We are currently collecting usage data. Cluster usage data will be displayed here in 24 hours." for an extended period (longer than 24 hours)."RPC error kTransportError raised" is seen when creating a new scenario.
Scenario#2

The same issue is reported on pc.2022.1, where the Neuron server is continuously crashing, because of which the Capacity planning feature does not work.As per recent instances, the issue may get resolved by upgrading PC to latest version i.e 2022.1.0.2 and a fix for this is also tracked in above mentioned tickets.. Scenario#1:

Long-term fixes for this issue are included in AOS 5.10.4, AOS 5.11, and NCC 3.7.2. Upgrade to AOS 5.10.4, 5.11 or later, and to NCC 3.7.2 or later.A workaround for this issue is to reinstall NCC manually on Prism Central and then wait for 24 hours for data to populate. Instructions for this are available on the portal; for example, this link for 5.10 https://portal.nutanix.com/page/documents/details?targetId=Acropolis-Upgrade-Guide-v5_18:ncc-ncc-install-t.html.

Scenario#2:

The issue is resolved in pc.2022.9, pc.2022.6.0.1, pc. 2023.1.0.1. Please upgrade to those releases for the fix.

 . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Support during a critical upcoming upgrade/maintenance. ### Assistant: Support during a critical upcoming upgrade/maintenance. Can Nutanix Support join a customer in a call to guide them and provide coverage during critical upcoming activities? Nutanix SREs are often expected to engage in customer remote sessions for installation, upgrades or configuration activities during their maintenance window.

While Nutanix Support would always like to be available for our customers, we are not equipped to handle requests related to installation, upgrades or configuration. If we remain on a remote session for the duration of such an activity, then we become unavailable to address critical break/fix issues.

Nutanix Support's primary responsibility is addressing and troubleshooting customer break/fix issues, not handling other maintenance activities.. Nutanix prides itself on providing robust tools and documentation to facilitate maintenance activities such as installation, upgrades, etc. We highly recommend leveraging these resources. The Acropolis Upgrade Guide outlines the steps and resources needed for such activities.

Use the Life Cycle Manager (LCM) to upgrade all or some of the software and firmware components. In the event LCM cannot be used, use the Legacy 1-click method from the Web Console.

Ensure that all the prerequisites of maintenance activity are met. It is possible that during the maintenance, you exhaust all the documentation (both KB articles and guides) and still require assistance from a Nutanix SRE. These instances may be, but are not limited to, the following situations:

An installation/upgrade is stalledAn installation/upgrade completes but one or more nodes do not come upIntermittent service crashesInter/intracluster connectivity issuesNetwork/storage unavailability or instability

In the event assistance is still needed, the best way to get support is to raise a case via the Support Portal.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A111066 - Failed to send alert Emails. ### Assistant: This Nutanix article provides the information required for troubleshooting the alert: "Alert E-mail Failure" for your Nutanix cluster.. Alert Overview

The Alert E-mail Failure alert can be generated due to:

Errors in email-alerts scriptError while executing email-alerts scriptErrors while creating an email file for the alert

Check ID111066DescriptionFailed to send alert emails.Causes of failureAlert is raised if a Controller VM is unable to send an alert email.ResolutionsEngage Nutanix Support.ImpactAlerts notification emails will not be sent.Alert TitleAlert E-mail FailureAlert MessagesUnable to send alert E-mail from Controller VM {service_vm_external_ip} due to “E-mail client error”. Capture alert manager service logs, email-alerts.log, and send-email.log from all the CVMs in the cluster. Open case with Nutanix Support and attach the log files to the case.Log files location:
~/data/logs/alert_manager.out~/data/logs/email-alerts.log~/data/logs/send-email.log
To collect these logs using the Logbay tool and upload them to the support case:

logbay collect -t alerts,email --dst=ftp://nutanix -c <case number>
For more information about the log collection tool Logbay, please refer to KB-6691.Note: If the alert persists in Prism UI. Please try disabling and re-enabling in the Health page in Prism.In Prism select Health from menu. To find the alert select the Checks tab on right hand side of page. Then search for "Alert E-mail Failure". Click 'Turn Check Off,' then 'Turn Check On' to clear its status.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Frozen I/O State in Microsoft SQL Server During application-consistent snapshots. ### Assistant: MSSQL databases can remain frozen indefinitely during application-consistent snapshots..

During the creation of application-consistent snapshots, there is a possibility of encountering a problem that can cause Microsoft SQL Server (MSSQL) databases to become frozen. This issue specifically occurs during VSS-based backups, where the MSSQL databases can remain frozen indefinitely.

In MSSQL Logs, the error logs will show as below (The logs can be obtained by following this Microsoft article)

2023-05-18 23:48:52.79 spid77     I/O is frozen on database [DATABASE_NAME]. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.

Obtain the Microsoft SQL version the UVM is running to confirm if this KB applies.Method 1:Connect to the instance of SQL Server, and then run the following query:

2023-05-18 23:48:52.79 spid77     I/O is frozen on database [DATABASE_NAME]. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.

The result of running the query above would be the following:

2023-05-18 23:48:52.79 spid77     I/O is frozen on database [DATABASE_NAME]. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.

Method 2:Another way of checking the version is by connecting to the instance of SQL Server, and then run the following query in SQL Server Management Studio (SSMS):

2023-05-18 23:48:52.79 spid77     I/O is frozen on database [DATABASE_NAME]. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.

The result of running the query above would be the following:

2023-05-18 23:48:52.79 spid77     I/O is frozen on database [DATABASE_NAME]. No user action is required. However, if I/O is not resumed promptly, you could cancel the backup.. This is a known issue (reported in MS KB-4523102) detected by Microsoft on SQL databases that contains memory-optimized filegroup and VSS based backups. Microsoft has released the following cumulative updates for SQL Server:

Cumulative Update 2 for SQL Server 2019 (Build version: 15.0.4013.40)Cumulative Update 19 for SQL Server 2017 (Build version: 14.0.3281.6)Cumulative Update 10 for SQL Server 2016 SP2 (Build version: 13.0.5492.2). ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NDB might get an alert about Nutanix cluster connectivity during a cluster upgrade. ### Assistant: When a software or firmware upgrade is performed on the Nutanix cluster, NDB can generate an alert about the cluster connectivity.. During a software or firmware upgrade on the Nutanix cluster, NDB can generate an alert about the cluster connectivity status:
Health check failed for cluster (name:<Cluster_Name>), Please make sure the cluster is in valid state
The era_server.log on the NDB server will have the following entry:
Health check failed for cluster (name:<Cluster_Name>), Please make sure the cluster is in valid state
 . If the alert is generated during an upgrade activity, it is expected and can be safely ignored as long as the cluster connectivity status goes back to normal when the upgrade is completed.To check the cluster connectivity status: NDB -> Administration -> Nutanix clusters.The reason why it happens is because during most of the upgrades, the CVMs reboot in a rolling manner and the virtual IP address gets re-hosted from the Prism leader CVM to a different CVM. That can take a few seconds, so the virtual IP can be briefly unavailable, which causes the API checks coming from the NDB to fail and it is enough to generate the alert in NDB.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: stale_recovery_points_check. ### Assistant: The NCC health check stale_recovery_points_check checks if there are any recovery point entries in the Insights database whose underlying snapshots have been deleted.. The NCC health check stale_recovery_points_check checks if there are any recovery point entries in the Insights database whose underlying snapshots have been deleted.It can be run as part of the complete NCC check by running
nutanix@cvm:~$ ncc health_checks run_all
or individually as:

nutanix@cvm:~$ ncc health_checks run_all
You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every 12 hours, by default.This check will generate an alert after 2 failure.

Sample outputFor status: PASS

nutanix@cvm:~$ ncc health_checks run_all
For Status: ERROR

nutanix@cvm:~$ ncc health_checks run_all

Output messaging

Check ID110405DescriptionChecks if there are any recovery point entries in the Insights database whose underlying snapshots have been deleted.Causes of failureThe snapshot was deleted without removing the associated stale entries in the Insights database.ResolutionsCheck the running status of the Polaris service.ImpactUpcoming replications might be impacted by the stale remnant entries in the Insights database.
 . If this NCC check fails, engage Nutanix Support.Additionally, gather the following command output and attach it to the support case:
nutanix@cvm:~$ ncc health_checks run_all
 . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Lose local attach vDisks after adding a NIC. ### Assistant: After adding the second NIC to Windows Server, non-bootable local attach vDisks disappear from within guest OS. After adding the second NIC to Windows Server, non-bootable local attach vDisks disappear from within guest OS. Only drive C is present.Migrating or Rebooting the VM does not solve the issue. . Open disk management in windows and check the disks should be marked as offline.To bring the disks online, right click on each vdisk and click "Online". Now you should be able to see and use the vdisks within guest OS.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about OVA UI may take longer to load due to unreachable Prism Element cluster registered to Prism Central. ### Assistant: When connected clusters are unreachable the OVA UI may take longer to load. OVA UI may take longer to load if one or more registered Prism Element clusters are unreachable.
The /home/nutanix/data/logs/aplos.out log file on Prism Central will contain the following error:

2023-01-06 02:18:40,257Z ERROR resource.py:231  Traceback (most recent call last):
You may also notice the following RPC timeouts in the logs:
2023-01-06 02:18:40,257Z ERROR resource.py:231  Traceback (most recent call last):

 . Prism Central tries connecting to every connected Prism Element cluster to check compatibility.To find the unreachable Prism Element UUID, you can refer to the below logs:
2023-01-06 01:45:22,964Z WARNING fanout_protobuf_rpc.py:230 Error # 5 sending RPC [500] to http://127.0.0.1:9444/v3/fanout_proxy?remote_cluster_uuid=0005eea4-b61d-36d5-408d-7cc255078170&url_path=remote_rpc_requ
If the Prism Element cluster is deleted without properly unregistering, refer to KB 4944 to clean the Prism Element cluster details.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Enabling Jumbo MTU on AHV for UVMs. ### Assistant: This article describes how to enable jumbo MTU on AHV for UVMs.. Nutanix Controller VMs (CVMs) use the standard Ethernet MTU (maximum transmission unit) of 1,500 bytes for all the network interfaces by default. The standard 1,500-byte MTU delivers excellent performance and stability. Nutanix does not support configuring the MTU on a CVM's network interfaces to higher values.You can enable jumbo frames (MTU of 9,000 bytes) on the physical network interfaces of AHV, ESXi, or Hyper-V hosts and user VMs if the applications on your user VMs require them. If you use jumbo frames on hypervisor hosts, enable them end to end in the desired network and consider both the physical and virtual network infrastructure impacted by the change.. PreparationMake sure that the following changes are implemented before making any changes:

The physical L2 switches must be configured to support a 9000-byte Ethernet payload and any additional network overhead such as VLAN tags and VXLAN encapsulation. This means the switching infrastructure is often configured to support a maximum MTU of 9216 bytes, larger than the endpoint MTU.Any routed L3 interfaces between networks must also be configured to support the larger MTU.
Consult the documentation of your network vendor for this configuration. Only after the entire physical network is ready to handle jumbo frames between all endpoints can we configure our Nutanix AHV hosts.

Check the current network configurationTo check the current host network configuration, connect to the CVM you will be making changes on and run the following command:

nutanix@cvm$ manage_ovs show_uplinks

Sample output:

nutanix@cvm$ manage_ovs show_uplinks

One of the main items we want to check at this step is the bond_mode. If balance-slb or balance-tcp (LACP) modes are used, restore this configuration after making the MTU changes.

Next, connect to the AHV host and verify the current MTU value of the physical interfaces. These typically show up as ethX, but may vary depending on the hardware in use.

nutanix@cvm$ manage_ovs show_uplinks

Change MTU (AOS 5.19 and newer)

Refer to the Prism Web Console Guide for your appropriate AOS version for instructions on how to update your Cluster Virtual Switch's MTU.

Login Prism > Settings > Network Configuration > Virtual Switch > Edit Virtual Switch

	Set the MTU and Save the Update (MTU on Nutanix Host must be set to a value in 1500 ~ 9000 inclusive range)

Change MTU (AOS 5.18 and older)Perform the following steps on every host in the cluster that needs these changes.
Connect to the host using the IPMI console and log in to the AHV host as root.Log on to the Controller VM via SSH to the management address.
	nutanix@cvm$ manage_ovs show_uplinks
	Put the node and CVM in maintenance mode. Refer to Putting a node into maintenance mode chapter from AHV Administration Guide for more details.Run the following command to enable jumbo frames on the interfaces.
	nutanix@cvm$ manage_ovs show_uplinks
	where <bond mode> equals one of the following modes: active-backup, balance-slb or balance-tcp. The current bond_mode can be found in the output of the manage_ovs show_uplinks command. If --bond_mode parameter is not specified, then the bond mode will be reset to active-passive, which may affect network connectivity.	Note: In the above command the syntax "--mtu 9000" or "--mtu=9000" are both valid.If LACP is used then a few more additional parameters should be added:
	nutanix@cvm$ manage_ovs show_uplinks

	Example:

	nutanix@cvm$ manage_ovs show_uplinks

	If a bridge other than br0 is used for UVMs, the configuration should be similar to the following example:

	nutanix@cvm$ manage_ovs show_uplinks

	The above example applies to configuration with and without backplane network segmentation, as it only changes MTU of physical NICs without touching interface br0, or interface br1-backplane if configured.		Note: --require_link=false is only required if any physical NIC is down while configuring, which you can check with manage_ovs show_interfaces.
	WARNINGS:

	On clusters running any AOS release between 5.10 and 5.10.3.2 DO NOT run "update_uplinks" with a single interface specified as this may impact cluster availability. If you have a node with a single interface in bridge, upgrade AOS to 5.10.4 or newer, or engage Nutanix Support if you want to change the bond configuration.Updating uplinks using manage_ovs will delete and recreate the bond with the default configuration.The use of the allssh manage_ovs update_uplinks command may lead to a cluster outage. Only use it if a cluster is not in production and has no user VMs running.
	If balance-slb or balance-tcp (LACP) modes were used, then refer to the following KB articles to restore the configuration. Otherwise, if LACP is not enabled, go directly to step 7:
	balance-slb: KB 5135balance-tcp (LACP): KB 3263
	Gracefully shut down the Controller VM on this host:
	nutanix@cvm$ manage_ovs show_uplinks
	Reboot the AHV host.
	nutanix@cvm$ manage_ovs show_uplinks
	Once the configuration is completed, make sure both host and CVM are accessible via network.If all tests are successfully completed, remove CVM and node from maintenance mode. Refer to Exiting a node from the maintenance mode chapter from AHV Administration Guide for more details.Create a VM network.
	nutanix@cvm$ manage_ovs show_uplinks
	Enable jumbo MTU inside UVM guest OS.Test the reachability of the jumbo frame between UVMs.	Linux:
	nutanix@cvm$ manage_ovs show_uplinks
	Windows:

	nutanix@cvm$ manage_ovs show_uplinks

	Note: "-M" and "-f" are to set df-bit (don't fragment), and 8972 is ICMP data size, which equals 9000B (MTU size) - 20B (IP header) - 8B (ICMP header). There are another 14 bytes/18 bytes for Ethernet header (6B source MAC + 6B destination MAC + 2B ethertype + 4B VLAN tag (VLAN tag is optional)), which are not part of MTU size. Therefore, the maximum frame size allowed after the above configuration is 9014B/9018B (with VLAN tag).
	. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Open Source Support. ### Assistant: This article describes how Nutanix supports open source repositories.. This article describes how Nutanix supports open source repositories.. OverviewNutanix is supportive of open-source technologies and leverages a community-supported model. Nutanix welcomes pull requests and responds to issues on a best-effort basis. Nutanix maintains public GitHub repositories for initiatives that help customers integrate the Nutanix platform with other third-party products. Support for these initiatives is handled directly via the GitHub repository. Issues and enhancement requests can be submitted in the Issues tab of each repository. Search for and review existing open issues before submitting a new issue.

RepositoriesThe Github repositories locations:

https://github.com/nutanixhttps://github.com/nutanix-cloud-native

Repositories include:

Ansible. Nutanix ansible collection nutanix.ncp is the official Nutanix ansible collection to automate Nutanix Cloud Platform (NCP). Location: https://github.com/nutanix/nutanix.ansibleCluster-API provider for Nutanix AHV. The Cluster API brings declarative, Kubernetes-style APIs to cluster creation, configuration and management. Cluster API Provider for Nutanix Cloud Infrastructure is a concrete implementation of Cluster API for Nutanix Cloud Infrastructure. Location: https://github.com/nutanix-cloud-native/cluster-api-provider-nutanixKubeflow. The Kubeflow project is dedicated to making deployments of machine learning (ML) workflows on Kubernetes simple, portable and scalable. Location: https://www.kubeflow.org/docs/distributions/nutanix/install-kubeflowTerraform. An infrastructure-as-code tool for building, changing and versioning infrastructure safely and efficiently. Location: https://github.com/nutanix/terraform-provider-nutanixPacker Plugin for Nutanix AHV. The Nutanix Packer builder is able to create Nutanix AHV virtual machine images and export them. Location: https://github.com/nutanix-cloud-native/packer-plugin-nutanix

The complete list of repositories:

https://github.com/orgs/nutanix/repositorieshttps://github.com/orgs/nutanix-cloud-native/repositories

Reporting an IssueTo report an issue:

Navigate to the specific GitHub repository and click on Issues. For example:

	Click New issue.

	Select Bug report or Feature request, then click the corresponding Get started button to the right.

	. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: check_cvm_ssh_security. ### Assistant: The NCC health check check_cvm_ssh_security checks whether the cluster uses Nutanix SSH security settings, such as lockdown and SSH security level.. The NCC health check check_cvm_ssh_security checks whether the cluster uses Nutanix SSH security settings, such as lockdown and SSH security level. This check was introduced in NCC 4.6.4.Running the NCC checkYou can run the check as part of the full NCC health check, as shown below.
nutanix@cvm$ ncc health_checks run_all
You can also run this check independently, as shown below.

nutanix@cvm$ ncc health_checks run_all
You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check runs on Controller VMs only. It does not run on Prism Central.This check is scheduled to run every 24 hours, by default.This check will raise an NCC alert when a cluster does not use our SSH security settings, such as lockdown and SSH security level.
Sample Pass output:For Status: PASS

nutanix@cvm$ ncc health_checks run_all
If the check result is a PASS, Password-based remote login is not enabled. Everything is good.For Status: INFO

nutanix@cvm$ ncc health_checks run_all
If the check result is a INFO, Password-based login is enabled. It is recommended to not use password-based ssh login access to the cluster.Output messaging
Check ID6219DescriptionCheck the strength of SSH security of the CVM.Causes of failureThe SSH security of the CVM is weak.ResolutionsChange the SSH security setting of the CVM.ImpactThe CVM's security might get compromised.ScheduleThis check is scheduled to run every 24 hours by default.Alert IDA6219Alert TitleThe cluster is using password based ssh access.Alert Smart TitleThe cluster is using password based ssh access for the cvm {cvm_ip} Alert MessageThe cluster is using password based ssh access for the cvm {cvm_ip} . Nutanix recommends using key-based ssh access to the CVM instead of password-based ssh access for better security. Refer to the Controlling Cluster Access section in the AOS Security Guide to set up key-based SSH access.Please perform the following actions to clear the INFO message in NCC:
Click the gear icon in the main menu and then select Cluster Lockdown in the Settings page.Disable remote login access by unchecking the Enable Remote Login with Password box. Refer Controlling Cluster Access.Click the New Public Key button and add Public Key for SSH access. Refer Controlling Cluster Access. Follow KB 1895 on how to set up password-less SSH Key pair.
Nutanix recommends avoiding ssh access to the CVM appliances as much as possible. Key-based SSH access is preferred instead of password-based access in the rare cases where one has to login to the CVM.Adding rsa/ecdsa public key will allow the admin and nutanix users to access CVMs and hosts using the ssh key pair.Note: Password-based SSH access will be deprecated in 2024.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Under what conditions do Curator scans run?. ### Assistant: Lists the scenarios in which Curator Scheduled and Triggered scans are run.. There are two main types of Curator scans.
Scheduled Scans
	Curator Full ScanCurator Partial Scan
	Triggered Scans. Scheduled Scans
Scheduled scans run periodically on the cluster for the general maintenance of the system. There are two types of scheduled scans.

	Partial Scans

	Partial scans are responsible for performing the following tasks:

	Information Life Cycle Management (ILM)Snapshot chain severingDelete data that was marked to_remove but not dedupedCorrect block awarenessDelete or replicate over-replicated or under-replicated data

	By default, partial scans run 1 hour (3600 seconds) after the last partial scan completes.

	Full Scans

	In addition to all the Partial scan functions, Full scans also perform on-disk deduplication.	As only full scans perform on-disk deduplication, you might have to perform multiple scans to ensure that deduped data is deleted completely. Full scans run on their own timer and run 6 hours (21600 seconds) after the last Full scan completes. When this timer appears and if another scan is running, the timer waits for the scan to complete before running another Full scan.

Triggered Scans
Triggered scans respond to a situation in the cluster where Curator is urgently required.The most common triggered scans are as follows:

ILM - If the hot tier gets full, an ILM scan is triggered to drain some data to the cold tier.Disk Failure - If a disk fails, a scan is triggered to replicate the data having less replicas.Node Failure - This scan is similar to a disk failure scan, but runs to replicate data from all disks having had replica on the failed node.User - Manually triggered by the SRE team in certain situations. This can be a Full or Partial scan.
 These scans are all partial scans (unless the user manually triggers a full scan). They reset the clock on the next periodic partial scan (or full scan if it is a user triggered scan).. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A1120 - CuratorJobRunningTooLong. ### Assistant: Investigating CuratorJobRunningTooLong issues on a Nutanix cluster.. This Nutanix article provides the information required for troubleshooting the alert CuratorJobRunningTooLong for your Nutanix cluster.
Alert Overview
The Curator Job Running Too Long alert A1120 is generated if the active Curator jobs have been running for too long. An alert will be raised if a Curator scan runs for more than 21600 seconds (6 hours) and then after every 12600 seconds (3.5 hours).
Sample Alert

For status: INFO

Block Serial Number: 19SMXXXXXXXX

Output messaging

Check ID130015DescriptionThe Curator job has been running for too long.Causes of failureVariousResolutionsContact Nutanix support if this issue persists.ImpactBackground cluster maintenance tasks might get affected in the future. The root cause should be addressed as soon as possible.Alert IDA1120Alert TitleCurator Job Running Too LongAlert MessageCurator job name with id execution_id has been running for a long time i.e. elapsed_time_secs seconds.. Troubleshooting

Curator scan times depend on the cluster size (number of nodes), global metadata size, and metadata scan speed in the Map phase of the Map/Reduce job, among other things.For more information on local vs global metadata, see Scalable metadata chapter in the Nutanix bible.

In order to check metadata size, run the following command:

nutanix@CVM:~$ nodetool -h 0 ring

Example output:

nutanix@CVM:~$ nodetool -h 0 ring

One possible reason for high metadata usage can be using the Deduplication feature for workloads not suitable for it. For more information on Deduplication Best Practices, see the corresponding chapter in Prism Web Console Guide.Another reason for long-running Curator Jobs, or failing Curator scans, might be the RPO settings on clusters having dense nodes.Check both the frequency and retention policy for the snapshots, if data protection features are in use:

1 hour vs 6 hours vs 24 hours RPO,for 1 hour RPO, if supported, the 12/1/1/1 retention should not be exceeded.
In case if cluster contains dense nodes, run the NCC dense_node_configuration_checks and refer to KB 7196 for more details.

Resolving the issue

If you need assistance or in case the above-mentioned steps do not resolve the issue, consider engaging Nutanix Support.Collect additional information and attach them to the support case.

Collecting Additional Information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Collect the NCC output file ncc-output-latest.log. For information on collecting the output file, see KB 2871.Collect Logbay bundle using the following command. For more information on Logbay, see KB 6691.
	nutanix@CVM:~$ nodetool -h 0 ring

Attaching Files to the Case

When viewing the support case on the support portal, use the Reply option and upload the files from there.If the size of the NCC log bundle being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations. See KB 1294.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Upload of images larger that 16 Gb on AOS 5.20.2 or newer failing with "Image upload stream was interrupted" error. ### Assistant: Upload of images larger that 16 Gb on AOS 5.20.2 or newer failing with "Image upload stream was interrupted" error. Image uploads using Prism Element may fail with an "Image upload stream was interrupted" error if all of the following conditions apply:
Cluster is running AOS 5.20.2 or newer.The image file size is larger than 16 Gb.
The following stack trace may be found in /home/nutanix/data/logs/prism_gateway.log:

ERROR 2021-12-09 15:53:53,207Z http-nio-127.0.0.1-9081-exec-7 [] commands.acropolis.UploadImage.doExecute:405 java.net.SocketTimeoutException
 . This issue is resolved in:
AOS 5.20.X family (LTS): AOS 5.20.4AOS 6.5.X family (LTS): AOS 6.5.1
Please upgrade AOS to versions specified above or newer.WorkaroundUse From URL method.
Prepare a web server and share the image file via HTTP protocol.Select "From URL" as "Image Source" of "Image Configuration" and specify the URL to a disk file.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Configuration Steps for Okta Integration with Prism Central. ### Assistant: Okta is a supported Identity Provider that can be integrated with Prism Central for user authentication. This KB documents the steps required to configure Okta integration from both the PC and Okta Administration side.. Nutanix supports several different Identity Providers for integration with Prism Central (PC) including:

Active Directory Federated Services (ADFS)Azure ADFSPingOneShibbolethKeycloakOkta

This document details specifically what needs to be configured on both the Prism Central and Okta Administration side in order to successfully integrate both for user authentication.. From the Okta Administration side:

Create a new SAML 2.0 application

	Configure the following SAML settings:
	Single Sign On URL: https://<PC_IP>:9440/api/iam/authn/callback	Audience URI (SP Entity ID): https://<PC_IP>:9440/api/iam/authn		NOTE:	Whether to use the FQDN or PC IP address for above configuration, can be decided by checking the metadata config file from following Prism Page			Check the following entityID and Location fields in metadata xml from PC, highlighted in following screenshot.	Audience URI  should match with entityID URI and Single Sign On URL should match with Location URL in okta settings.					Attribute Statements:

	Note: If you intend to assign Groups to the PC Application, then configure the Group Attribute Statement as follows:

	Assign Users and/or Groups to the PC Application in Okta

	View the Identity Provider metadata from the PC SAML application and save it to your local machine as an .xml file.

From the Prism Central side:

The Microservices Infrastructure (CMSP) needs to be enabled from the Prism Central Settings -> Manage Prism Central -> Prism Central on Microservices Infrastructure

	More detailed information on Microservices enablement can be found on the Nutanix Support Portal at the following link: https://portal.nutanix.com/page/documents/details?targetId=Prism-Central-Guide:mul-cmsp-enable-pc-t.html.
	The Okta Identity Provider can be added from Prism Central Settings > Authentication > Identity Providers
	Note: If you intend to perform Group Role Mapping in PC, then configure the Group Attribute Name as follows:

	Assign individual user role mappings from Prism Central Settings > Role Mapping or group role mappings from Administration > Roles
	Note: The format of the username must match Okta's expected "Application username format"

The user can now authenticate to Prism Central with Okta credentials.

Using Okta Native SAML group:We can also use okta SAML group and assign the required users to this group to enable okta authentication for them.For this we need to do following additional configuration both on Okta and Nutanix side.Configuration from Okta side:1. In the Okta Admin dashboard, go to Directory → GroupsClick on "Add Group" to create a new group and name it.2. Assign people to that groupAfter assigning required people click on "Done"3. Then go the "PC application" on okta side and assign that application to this group.4. Under the Sign On Section of the app, following values need to be given for group attributes.And save the changes for the application settings.Configuration from PC side:1. Modify IDP configuration in PC to specify the 'Group' attribute2. Navigate on PC to Administration → RolesYou can create a custom role or just use one of the built-in ones.3. Then go to "manage assignment" option of that role and create select Okta user_group(SAML) from drop down option.NOTE: Make sure you give exact name for the group as given on okta side, the SAML user/group names does not get auto-populated as PC does not have Okta credentials to query and search user/group from Okta.4. Then you should see the okta group listed under "user_group" in this role information.After completing all the above configuration any user assigned to the group on Okta side should be able to login to PC using Okta.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Register for my.nutanix.com and Support Portal access. ### Assistant: Registering for my.nutanix.com Portal account and how to gain access to the Support Portal. The following article describes how to register for the my.nutanix.com Portal and access the Support Portal.. Please perform the following procedure to register for the my.nutanix.com Portal. Once successful you can then perform the additional steps to access the Support Portal.
In the address bar of a web browser, type: https://my.nutanix.com and press EnterClick Sign up nowType the values of First Name, Last Name, Email, Password in the fields and click Submit	Note: Follow the specified password complexity requirements when you are creating the password	A confirmation page will be displayed and you will receive an email from mynutanix@nutanix.com after you have successfully completed the registration process. The following is an example of the email:

Hi First Name,

Click the link provided in the email to confirm the registration process. A message briefly appears in the browser confirming your email address and the my.nutanix.com Portal opens in the browser.Type the Email and Password that you used to register and click the arrow in the Password field. The my.nutanix.com Portal Welcome Back page appears.
For access to the Support Portal, continue with the following steps:

Select Support Portal and click the arrow in the Support Portal dialog box. Note: You must first activate the account before you can browse the Support Portal and create support casesIn the Activation Required dialog box that appears, enter the Business Email and a valid Nutanix Serial Number or Service Tag and click Activate. The Activation Complete! screen appears after you have successfully completed the activationClick the box next to I have read and agreed to the above terms and conditions to accept the terms. The Support Portal page will appear
If you are still experiencing access issues after following the steps above please contact portal-accounts@nutanix.com.Note: For access to the Partner Portal, there are 2 ways.

if your account does not have any assets or SW registration ID

Go to https://www.nutanix.com/partner-signup Fill out the Request Portal Access form and submit

If your account  has assets or SW registration ID

Click Activate on the Partner Portal tileFill out the Request Portal Access form and submit
Refer to the Partner team for any assistance with Partner Portal access. For help please send an email to partnerhelp@nutanix.com. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: ilo_bootdrive_raid_check. ### Assistant: NCC 3.10.0. The NCC health check ilo_bootdrive_raid_check checks the iLO boot drive RAID status.. The NCC health check ilo_bootdrive_raid_check checks the iLO boot drive RAID status.

This check does the following:

Check the logical drive status.If the status is not okay, check available number of physical drives.If only one physical drive is available, generate a WARN alert with the working drive's information. The non-working drive can then be easily found and replaced.If both physical drives are available, check their health. If health is not okay, generate a WARN alert with the failed drive's information. The failed drive can then be replaced.If the drives are healthy, this check returns PASS.

This check runs only on the following:

iLO-based nodesAOS 5.11.1.5, 5.11.2, 5.16 or higherWith RAID boot drive

Running the NCC checkIt can be run as part of the complete NCC check by running:

nutanix@cvm$ ncc health_checks run_all

Or individually as:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every hour by default.

This check will generate an alert after 1 failure.

Sample output

For status: PASS

nutanix@cvm$ ncc health_checks run_all

For status: WARN

nutanix@cvm$ ncc health_checks run_all

nutanix@cvm$ ncc health_checks run_all
For status: ERR

nutanix@cvm$ ncc health_checks run_all

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID106066DescriptionChecks the status of HPE Integrated Lights-Out(iLO) boot drive.Causes of failureHypervisor RAID Boot Device is not reachable or is faultyResolutionsCheck the status of the RAID device. Refer KB-9064 to resolve this issue. If the problem persists, please engage Nutanix Support.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert Smart TitleHypervisor RAID Boot Device failed on ip_address serial number SerialNumberAlert TitleHypervisor boot drive RAID status check.Alert MessageWorking Boot Drive Info: LogicalDriveNumber: ldrive, SerialNumber: serialnumber in Bay: drive_bay, Model: model on Host IP x.x.x.x			Failed Boot Drive Info: LogicalDriveNumber: ldrive, SerialNumber: serialnumber in Bay: drive_bay, Model: model on Host IP x.x.x.x
Note: This hardware-related check executes on the below hardware: HPE. . Refer to the Hypervisor Boot Drive Replacement procedure (given below) and follow the instructions mentioned there.

DX4200 Gen10 24LFF - Hypervisor Boot Drive Replacement (Failed Boot Device) for DX4200 Gen10 24LFF with Mirrored RAID Hypervisor Boot DrivesDL380 Gen10 - Hypervisor Boot Drive Replacement (Failed Boot Device) for DL380 Gen10 with Mirrored RAID Hypervisor Boot DrivesDX NS204i - Hypervisor Boot Device Replacement (Failed Boot Device) for HPE DX NS204i Boot Device

Replacement drives can be obtained from HPE.HPE NS204i-p NVMe OS Boot Device detects Warning intermittently because the fan speed is low. On this occasion, increasing fan speed to cool down the temperature of NVMe resolves this WARN.HPE ProLiant DL560 Gen10, HPE ProLiant DL580 Gen10, HPE ProLiant DL360 Gen10, HPE ProLiant DL380 Gen10, and HPE ProLiant DX380 Gen10 Plus - Intermittent Server Shutdown Due to Thermal Issues with HPE NS204i-p NVMe OS Boot DeviceThe status ERR for hpe_bootdrive_raid_check is a false positive and resolved after upgrading NCC to 4.2.0 and beyond.When the status ERR for ilo_bootdrive_raid_check "ILO boot drive RAID info Not avaliable" happens, please reset the iLO on the node and try the check again. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about "Cluster Memory Usage" widget may report inconsistent values on AHV cluster. ### Assistant: On the AHV cluster running AOS 5.15.1 or newer, "Cluster Memory Usage" widget may report inconsistent values.. On the AHV cluster running AOS version 5.15.1 or newer, "Cluster Memory Usage" widget may report inconsistent values.

Without an actual change in memory usage on a cluster, you may see the "Cluster Memory Usage" widget showing inconsistent values:

. This issue is resolved in:

AOS 5.15.X family (LTS): AOS 5.15.4AOS 5.19.X family (STS): AOS 5.19.1

Upgrade AOS to the version specified above or newer.

If upgrading to the previously mentioned versions doesn't resolve your issue, consider opening a case with Nutanix  support . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A160180 - PC is not AZ paired. ### Assistant: Troubleshooting and resolving alert "Remote file server of data protection policy is registered to a Prism Central which is not in an Availability Zone pair".. This Nutanix article provides the information required for troubleshooting the alert Host-limit-check for your Nutanix Files cluster.
Alert overview

The PC-is-not-AZ-paired alert is generated when the number of hosts exceeds the maximum of 32.

Sample alert

Block Serial Number: 23SMXXXXXXXX
Output messaging

Check ID160180DescriptionNewly Registered File Server is a Participant in an Existing Data Protection Policy with a Remote File Server Registered to a Prism Central, which isn't in an Availability Zone Pair with the Local Prism CentralCause of failureThe newly registered file server is part of an existing data protection policy with a remote file server registered to Prism Central, which isn't in an Availability Zone Pair with the Local Prism CentralResolutionsAdd an Availability Zone pair between local Prism Central and Prism Central of the remote file server. Alternatively, you can unregister the newly registered Prism Element and register it to another Prism Central in an Availability Zone pair with the Prism Central of the Remote File Server. If you have any issues, please refer to KB article 14865.ImpactThe remote file server is not visible on the local Prism Central. This means that the data protection policy cannot be managed appropriately. Also, DR workflow cannot be accomplished.Alert IDA160180 Alert TitleRemote file server of data protection policy is registered to a Prism Central which is not in an Availability Zone pairAlert MessageRemote file server {remote_fs_name} for data protection policy {policy_uuid} with file server {fs_name} is registered to Prism Central {remote_pc_name} ({remote_pc_uuid}), which is not in an Availability Zone pair with the local Prism Central
 . Resolving the issueThis alert ensures that Availability Zones are maintained through File Server Migrations. If a File Server, or a File Server Remote DR site, moves to a new Prism Central instance, and the new Prism Central is not included in the Availability Zones, this alert will trigger. Add an Availability Zone pair between local Prism Central and Prism Central of the remote file server. Alternatively, you can unregister the newly registered Prism Element and register it to another Prism Central in an Availability Zone pair with the Prism Central of the Remote File Server.  For information about Availability Zones and their management, refer to the Nutanix Disaster Recovery Guide.

If you need assistance or if the above-mentioned steps do not resolve the issue, consider engaging Nutanix Support. Collect additional information and attach them to the support case.

Collecting additional information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Run a complete NCC health_check on the cluster. See KB 2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*NVM logs stored within the NVM at ~/data/logs/minerva*To collect the file server logs, run the following command from the CVM, ideally run this on the Minerva leader as there were issues seen otherwise, to get the Minerva leader IP, on the AOS cluster, run
		nutanix@cvm:~$ afs info.get_leader
				Once you are on the Minerva leader CVM, run:
		nutanix@cvm:~$ afs info.get_leader

		For example:

		nutanix@cvm:~$ afs info.get_leader

Attaching files to the caseTo attach files to the case, follow KB 1294. If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A160171 - FileServerTieredFileRestoreFailed. ### Assistant: Investigating FileServerTieredFileRestoreFailed issues on a Nutanix cluster.. This Nutanix article provides the information required for troubleshooting the alert FileServerTieredFileRestoreFailed for your Nutanix cluster.

After deleting a file server or a share, there could be files tiered referring to objects in the object-store. These objects could become stale if there are no references to them.

Nutanix recommends expiring these objects after the retention period configured for the object store profile.

Alert Overview

The FileServerTieredFileRestoreFailed alert is generated on share delete and FS delete operation.

Sample Alert

Block Serial Number: 16SMXXXXXXXX

Output messaging

Check ID160171DescriptionTiered File Restore Failed.Causes of failureTiered file restore failed unexpectedly.ResolutionsRefer to KB article 14507. Contact Nutanix support if issue still persists or assistance needed.ImpactTiered file is in inconsistent state.Alert IDA160171Alert TitleTiered File Restore FailedAlert Message{message]. Troubleshooting

The message from the alert will provide one of two reasons for failing to restore the file.

Scenario 1: "No object store configuration was found for profile." This means the profile was somehow deleted or not accessible.Scenario 2: When object was not found because it was deleted manually from object store or it was removed from object store as part of garbage cleaning.

Resolving the issue

Scenario 1: Contact Nutanix Support for assistance. See "Collecting Additional Information" below.Scenario 2: Attempt to undo the share-restore and try again with a newer snapshot.

Collecting additional information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Run a complete NCC health_check on the cluster. See KB 2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*NVM logs stored within the NVM at ~/data/logs/minerva*To collect the file server logs, run the following command from the CVM. Ideally, run this on the Minerva leader as there were issues seen otherwise. To get the Minerva leader IP on the AOS cluster, run:
		nutanix@CVM$ afs info.get_leader
		Once you are on the Minerva leader CVM, run:

		nutanix@CVM$ afs info.get_leader

		For example:

		nutanix@CVM$ afs info.get_leader

Attaching files to the case
To attach files to the case, follow KB 1294. If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A160170 - FileServerCAShareConfigCheck. ### Assistant: Investigating FileServerCAShareConfigCheck issues on a Nutanix cluster.. This Nutanix article provides the information required for troubleshooting the alert FileServerCAShareConfigCheck for your Nutanix Files cluster.

Alert overview

The FileServerCAShareConfigCheck alert is generated if Continuous Availability (CA) is enabled for SMB standard share or nested share.

Sample alert

Block Serial Number: 16SMXXXXXXXX

Output messaging

Check ID160170DescriptionCheck if CA is configured on SMB standard or nested share on a File Server.Cause of failureContinous Availability is enabled for SMB standard share or nested shareResolutionsDisable Continous Availability feature on SMB standard or nested sharesImpactEnabling Continous Availability on an SMB standard / nested share could result in performance issues.Alert IDA160170Alert TitleFile Server CA Share Config CheckAlert MessageMisconfig of Continous Availability detected on a File Server SMB standard share / nested share.. Use of Continuous Availability (CA) should be limited to distributed shares. If it is enabled on Standard/General shares, performance might be impacted.

TroubleshootingCheck to see if any shares have CA enabled and if they are Standard (General) shares.

nutanix@FSVM$ afs share.list|grep 'Continuous\|Share type\|Share path\|Share name'

Resolving the issue

If you have any Standard shares with CA enabled, disable CA using:

nutanix@FSVM$ afs share.list|grep 'Continuous\|Share type\|Share path\|Share name'

If there are any concerns, or additional assistance is needed, contact Nutanix Support.

Collecting additional information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Run a complete NCC health_check on the cluster. See KB 2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*NVM logs stored within the NVM at ~/data/logs/minerva*To collect the file server logs, run the following command from the CVM. Ideally, run this on the Minerva leader as there were issues seen otherwise. To get the Minerva leader IP on the AOS cluster, run:
		nutanix@FSVM$ afs share.list|grep 'Continuous\|Share type\|Share path\|Share name'
		Once you are on the Minerva leader CVM, run:

		nutanix@FSVM$ afs share.list|grep 'Continuous\|Share type\|Share path\|Share name'

		For example:

		nutanix@FSVM$ afs share.list|grep 'Continuous\|Share type\|Share path\|Share name'

Attaching files to the case

To attach files to the case, follow KB 1294.If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.

Requesting assistanceIf you need assistance from Nutanix Support, add a comment in the case on the support portal asking Nutanix Support to contact you. If you need urgent assistance, contact the support team by calling one of our Global Support Phone Numbers. You can also press the Escalate button in the case and explain the urgency in the comment, and then Nutanix Support will be in contact.

Closing the caseIf this KB resolves your issue and you want to close the case, click the Thumbs Up icon next to the KB Number in the initial case email. This informs Nutanix Support to proceed with closing the case. You can also update the support case saying it is okay to close the case.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: check_cvm_panics. ### Assistant: The NCC health check check_cvm_panics checks the CVM serial console log for panics.. The NCC health check check_cvm_panics checks the CVM (Controller VM) serial console log for panics and warns the user if the CVM has rebooted due to a panic recently. If a CVM panics, the panic output from the kernel is captured in the serial console log in the host in the NutanixCvmConsole event log.

Running the NCC check

This check will run as part of the full NCC health check:

nutanix@cvm$ ncc health_checks run_all

It can also be run as a standalone check:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is not scheduled to run on an interval.

This check does not generate an alert.

Sample output

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID106469DescriptionCheck if any CVMs have panicked.Causes of failureAt least one CVM panicked recently.ResolutionsFollow the instructions in KB 4379.ImpactStorage may have been unavailable while the CVM was restarting.. Determine current cluster state.

nutanix@CVM:~$ cs | grep -v UP

SSH to cluster and run confirm uptime.

nutanix@CVM:~$ cs | grep -v UP

SSH to the CVM which generated the alert and run the command last reboot to find the precise time of the panic.This alert references a brief event, rather than an ongoing state.  After this alert is triggered, the cluster usually returns to a healthy state.If the check fails, and the cluster is NOT in a healthy state (i.e. CVM down, services down, etc), contact Nutanix Support for additional diagnosis.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.