### Human: Can you tell about Alert - A160176 - ConfigChangeNotifyFailed. ### Assistant: Investigating ConfigChangeNotifyFailed issues on a Nutanix cluster. This Nutanix article provides the information required for troubleshooting the alert ConfigChangeNotifyFailed for your Nutanix Files cluster."
Alert Overview

The ConfigChangeNotifyFailed alert is generated when there is a discrepancy between the File Servers and Prism Central configuration.

Sample Alert

Block Serial Number: 23SMXXXXXXXX
Output messaging

Check ID160176DescriptionFailed To Notify Config Change To File ServerCause of failureFile Server is down or not reachable while config change happens on PC/PEResolutionsCheck if File server is up and all the services are running. If File server is up, check network connectivity between File Server and PC/PE controller VM. If the failure persists, then refer to KB article 14745 to push the config change using afs cli.ImpactNAAlert IDA160176Alert TitleFailed To Notify Config Change To File ServerAlert MessageFailed to generate PCConfigChangeNotifyFailure alert due to 'reason'.. Troubleshooting
Check if the Files Server is up.

nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

Check if all service are up and running on the FSVMs.

nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

Check the connectivity between Prism Central and Prism Element, including the various required ports.  This information can be found on the Nutanix Portal.Run NCC on Prism Element to confirm that no "Prism Central is unreachable" alerts are raised.

Resolving the Issue

If Prism Central is reachable you can push the Files configuration to Prism Central to resolve this issue.For syncing config changes from Prism Element to Prism Central

nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"
For syncing changes from Prism Element to Files

nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

Collecting Additional Information

Attaching Files to the Case

If this alert automatically generates a support case, add the following sections:

"If you need further assistance from Nutanix Support, add a comment to the case on the support portal asking Nutanix Support to contact you. If you need urgent assistance, contact the Support Team by calling one of our Global Support Phone Numbers. You can also click the Escalate button in the case and explain the urgency in the comment, and Nutanix Support will be in contact."

"If this KB resolves your issue and you want to close the case, click on the Thumbs Up icon next to the KB Number in the initial case email. This informs Nutanix Support to proceed with closing the case. You can also update the support case saying it is okay to close the case and Nutanix Support will close the case."

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Run a complete NCC health_check on the cluster. See KB 2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*
		NVM logs stored within the NVM at ~/data/logs/minerva*

		To collect the file server logs, run the following command from the CVM, ideally run this on the Minerva leader as there were issues seen otherwise, to get the Minerva leader IP, on the AOS cluster, run

		nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

		Once you are on the Minerva leader CVM, run:
			nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

			For example:

			nutanix@NTNX-A-CVM:~$ ncli fs ls | grep "File server status"

	To attach files to the case, follow KB 1294.If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.

Requesting Assistance

If you need assistance from Nutanix Support, add a comment in the case on the support portal asking Nutanix Support to contact you. If you need urgent assistance, contact the support team by calling one of our Global Support Phone Numbers. You can also press the Escalate button in the case and explain the urgency in the comment, and then Nutanix Support will be in contact.

Closing the Case

If this KB resolves your issue and you want to close the case, click on the Thumbs Up icon next to the KB Number in the initial case email. This informs Nutanix Support to proceed with closing the case. You can also update the support case saying it is okay to close the case and Nutanix Support will close the case.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about cpupower frequency-info loads acpi-cpufreq driver instead of intel_pstate. ### Assistant: This article describes an issue where the command cpupower frequency-info loads acpi-cpufreq driver instead of intel_pstate.. This issue affects the following components combination:

Hardware: Intel Sapphire Rapids-based (SPR) nodes, including NX-G9AHV: 5.10.170-2.el7.nutanix.20220304.420.x86_64AOS: 6.5.3.1

SymptomsIn BIOS, the recommended setting for Hardware P-States is Disable by default on G9 SPR node.

In this case, cpupower frequency-info would use the acpi-cpufreq driver instead of the expected intel_pstate.

[root@host]# cpupower frequency-info

The turbostat command will continue to work when explicitly run.

This issue is caused by the intel_pstate driver being unable to retrieve CPU frequency values on SPR nodes. It requires support for SPR CPU set on AHV.. cpupower frequency-info using the acpi-cpufreq driver is a known behavior on SPR nodes.

Nutanix Engineering is aware of this behavior and is working on a fix to use the intel_pstate driver in a future release.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Objects - Manual steps to configure emails for Objects Alerts. ### Assistant: Below are the steps to configure email notification for Objects Alerts. Currently, Prism Central does not show Objects Alerts and one can use the following workaround until Objects Alerts are integrated with Prism Central.. Customers using Nutanix Objects no longer receive notifications for Objects Alerts on Prism Central. They have to launch the Objects page from Prism Central and navigate to the Alerts page to view and take appropriate action based on alert severity. Below are steps to enable email notifications for Objects Alerts until they are integrated with Prism Central in future releases.Note: review the steps thoroughly before applying the workaround. If you have questions or concerns, contact Nutanix Support for assistance.. To set up email notifications, update the Alertmanager configuration presented to the Alertmanager Pod(s) as a Kubernetes secret on MSP cluster. Refer to KB-8170 for steps to access Objects cluster.Overall steps to configure email notifications:
Update the Alertmanager configuration with an appropriate template.Base64 encode the updated Alertmanager configuration.Update the YAML template containing the Alertmanager secret using the Base64 encoded value of the new configuration.Apply the secret on the MSP cluster.

Update the Alertmanager configuration with an appropriate template	Refer to the below YAML file and change the highlighted parameters as desired. The highlighted parameters will vary based on your environment. Save it as alertmanager.yaml.

	global:
	Base64 encode the alertmanager.yaml
	global:
	Generate secret from the YAML. Reference the below YAML file and change the highlighted parameter with the base64 encoded string. Save it as alertmanager-secret.yaml.
	global:
	Update the secret.	Copy over the alertmanager-secret.yaml to the MSP master and apply the configuration using the kubectl command:
	global:
	Verify if template is applied successfully to alertmanager container:
	global:

	global:
		After applying the secret successfully, you should start receiving emails for Objects Alerts based on template configuration.			         Fig 1: Email notification for Objects Alert	 Check if the emails can be sent: 	Review alertmanager logs to see if there are any issues sending emails:
	global:
		If there are issues similar to the above, test sending email to the SMTP server manually using telnet or netcat as described in KB 1638.		You can compare sending emails from PC or PE cluster if they are registered with the same SMTP server.
Template examplesBelow are a few examples of templates that can be used to configure email notifications based on your requirements. Refer Prometheus documentation for more details.

Receive an email for custom Alerts:
	global:
	Receive customized email with formatting:
	global:
	Receive all email for all except few Alerts:
	global:
	Receive email for all Alerts:
	global:
	. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Files upgrades and FSM requirements. ### Assistant: Files requires upgrades of FSM to minimum versions.. Nutanix Files requires a component in AOS known as Files Server Module (FSM). The version of FSM required varies depending on the desired version of Files. Normally, this requirement is enforced during a Nutanix Files upgrade via Life Cycle Manager (LCM) or 1-click. This requirement can be circumvented if Nutanix Files 4.0.0.2 or higher is downloaded from the Nutanix Portal, and then uploaded manually to the Controller VMs (CVMs) for upgrade.. Nutanix Files 4.0.0.2 requires a minimum FSM version 2.1.1. Upgrade FSM to version 2.1.1 and above prior to upgrading.

Nutanix encourages the use of LCM for Files upgrades, which will enforce the proper FSM requirement.

To determine the version of FSM currently running, check LCM in Prism Element.

Should FSM show a version of 2.1.0 or earlier, then LCM is required to update FSM.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A160185 - File Server Found Stale Reverse Data Protection Policy During Failover. ### Assistant: Alert for handling Stale Replication policies. This Nutanix article provides the information required for troubleshooting the alert Stale File Server DB entries found for your Nutanix Files cluster.Alert Overview

If entries in the File Server database are found to be stale, it raises an alert to have the entries verified and cleaned.

Sample Alert

Block Serial Number: 23SMXXXXXXXX

Output messaging

Check ID160185DescriptionChecks if stale DB File server entry is present or notCauses of failureFile Server DB entries are found to be stale.ResolutionsFor additional detail, please refer to KB-15033.ImpactFile Server share access could be potentially impactedAlert IDA160185Alert TitleStale File Server DB entries foundAlert Smart TitleFile Server {file_server_name} has stale DB entriesAlert Message{alert_msg}. Troubleshooting    Should this alert trigger, a stale File Server Database entry has been detected.  There is no additional troubleshooting to be done.  Please proceed with the resolution.

Resolving the Issue

Perform failback:  From the target PC (Prism Central to which the target File Server of the protection policy is registered), click on Files ->Data Protection -> Protected File Servers -> FailbackCheck whether or not the stale reverse replication policy (whose UUID is given in the alert) is still listed in the Prism Central UI (Files -> Data Protection -> Policies)If it is still present, delete itPerform failover again

Should the repeated failover have any issues, or if you have any additional concerns, please contact Nutanix Support

Collecting Additional Information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Run a complete NCC health_check on the cluster. See KB 2871.Collect Files related logs. For more information on Logbay, see KB-3094.
	CVM logs stored in ~/data/logs/minerva_cvm*
		NVM logs stored within the NVM at ~/data/logs/minerva*

		To collect the file server logs, run the following command from the CVM, ideally run this on the Minerva leader as there were issues seen otherwise, to get the Minerva leader IP, on the AOS cluster, run

		nutanix@cvm:~$ afs info.get_leader

		Once you are on the Minerva leader CVM, run:
			nutanix@cvm:~$ afs info.get_leader

			For example:

			nutanix@cvm:~$ afs info.get_leader

Attaching Files to the Case

To attach files to the case, follow KB 1294.If the size of the files being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: notifications_dropped_check. ### Assistant: The NCC Health Check notifications_dropped_check identifies if any alert notifications from any system components have been dropped by Cluster Health and/or Alert Manager due to various possible downstream service or configuration issues. The check aims to proactively expose when alerts can not be delivered successfully and thus a genuine issue on the cluster that requires attention may otherwise go unnoticed.. The NCC Health Check notifications_dropped_check identifies if any alert notifications from any system components have been dropped by Cluster Health and/or Alert Manager due to various possible downstream service or configuration issues within the last 24 hours. The check aims to proactively expose when alerts can not be delivered successfully and thus, a genuine problem on the cluster that requires attention may otherwise go unnoticed.When the queue of alert notifications waiting to be delivered becomes >= 1000, it may suggest that services such as cluster_health and alert_manager cannot process and deliver the alerts, preventing emails, SNMP traps, and Rsyslog messages from being sent. Notifications that arrive when the queue depth is already at 1000 are dropped, and an email notification is sent to the cluster's configured alert email address(es) to inform that alerts and notifications were dropped.

Two reasons why notifications might be dropping :

Alert Manager service is down.The RPC queue is full.

Running the NCC CheckRun this check as part of the complete NCC Health Checks.

nutanix@CVM:~$ ncc health_checks run_all

Or run this check separately:

nutanix@CVM:~$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check does not generate an alert.

Sample outputFor Status: PASS

nutanix@CVM:~$ ncc health_checks run_all

For Status: FAIL

nutanix@CVM:~$ ncc health_checks run_all

For Status: INFO

nutanix@CVM:~$ ncc health_checks run_all

Or:

nutanix@CVM:~$ ncc health_checks run_all

Example email:

nutanix@CVM:~$ ncc health_checks run_all

Output messaging

Check ID111052DescriptionSome alert notifications were droppedCauses of failureOne or more cluster services are not healthy.ResolutionsContact Nutanix Support.ScheduleThis check is scheduled to run every day by default.
Note: This NCC check has multiple enhancements in NCC v4.1.0 - Please upgrade your cluster to NCC v4.1.0 or above and re-run the check.. If this alert is seen and/or emails are received, ensure the NCC is up to date and running the latest version (at least NCC-3.9.2.1 or higher). Due to an upstream change in how alert retries are handled and logged, this health check is over-sensitive in NCC 3.9.0/3.9.1 and may be triggered unnecessarily. After the upgrade, re-run the check and confirm the PASS result. If you continue to see a non-PASS result, you may have a genuine issue, or the now-stale trigger in the JSON state file needs to be cleaned up (per step #5 below.) Use the following steps to confirm and resolve.Check the Alert page in the Prism web console to verify what alert events are accounted for.
	If you see there are no events/notifications, but the check still fails, check :

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json

	If we look at the .json file, we can see that timestamps for dropped notifications increment even though there are no alerts in Prism. This is because we keep retrying to send stuck notifications:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Investigate why notifications are being dropped.
	One of the reasons could be that you might be using "admin" account to access some APIs. You should create a separate user with admin privileges instead. Refer to 5.11 Security Guide - Creating A User AccountConfirm all CVMs and Cluster Services are up and stable:
		nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
		Validate that the configured SMTP server is reachable and can relay the alert notification emails from the cluster's configured source email address to the destination email address(es).Validate the configured SMTP server security settings.
	Check older health_server.logs also to make sure the error has not rolled off:
	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	The notifications_dropped_check health check might still complain about the same issue based on data on a CVM which was previously but not currently the Cluster Health service leader, as it is looking for any indication of issues by way of a non-zero value in the JSON state file, despite current Cluster Health leadership. Drops might increment only on the leader, so we have to clear the stats on the old leader where the problem may have previously occurred.	SCP the file from the CVM where the values are 0 and replace it. To replace this for all CVMs, you can use this script:
	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Example:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	Run the NCC health check again and ensure the check results in a PASS status.

If the above solution does not help you or none of the CVMs notifications_dropped_state.json file has the value of 0 to scp, collect the following information, and consider engaging Nutanix Support.

	An output of the following NCC health check:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json

	List of alerts:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json

	An output of the following command from the PE cluster or the PC VM depending on where the alert was received:

	nutanix@CVM:~$ allssh ls -l /home/nutanix/data/serviceability/notifications_dropped_state.json
	. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: host_nic_error_check. ### Assistant: The NCC health check host_nic_error_check checks whether the NICs on the host are experiencing errors.. The NCC health check host_nic_error_check checks whether the NICs on the host are experiencing errors. It checks if each NIC has fewer than 1% errors during a span of execution of 1 hour (3600 seconds).

Running the NCC Check

You can run this check as part of the complete NCC health checks:

nutanix@cvm$ ncc health_checks run_all

Or you can run this check individually:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

Sample Output

For status: PASS

nutanix@cvm$ ncc health_checks run_all

For status: FAIL

nutanix@cvm$ ncc health_checks run_all

nutanix@cvm$ ncc health_checks run_all

For status: WARN (changed in NCC-4.6.2)

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID6011DescriptionChecks whether the NICs on the hypervisor are experiencing errors.Causes of failureNIC may be failing.ResolutionsCheck NIC health; if required, replace the NIC as soon as possible. Refer to the Nutanix documentation for instructions.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC packet receive an error rate highAlert Message
			nic_errors packet receive errors on NIC nic_name out of nic_packets in the last {period} seconds
			ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.
Note: Check_id 6011 does not generate an alert when the check fails prior to NCC-4.6.2
Check ID6013DescriptionChecks whether the NICs on the hypervisor are experiencing rx_crc errors.Causes of failureNIC or its peripherals may be failing.ResolutionsReplace the NIC, cable, SFP+ adapter, or switch port. Refer to the Nutanix documentation for instructions.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC RX_CRC Error Rate HighAlert MessageNIC nic_name in host host_ip has encountered many error_type.ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.

Check ID6014DescriptionChecks whether the NICs on the hypervisor are experiencing rx_missed errors.Causes of failureRX buffer size or transient condition.ResolutionsRX buffer size might be too small to absorb current inbound traffic. This can be a transient condition. If the alert reappears frequently, contact Nutanix support.ImpactCluster performance may be significantly degraded. In the case of multiple nodes with the same condition, the cluster may become unable to service I/O requests.Alert TitleNIC RX_Missed Error Rate HighAlert MessageNIC nic_name in host host_ip has encountered many error_type.ScheduleThis check is scheduled to run every hour by default.Number of failures to alertThis check will generate an alert after 1 failure.

This hardware-related check executes on the below hardware:

Nutanix NXDell XCHPELenovo HXNutanix SXFujitsu XFLenovo SRPowerEdgeIntelIBMHyveVoyager TDC. Ensure that you are running the latest version of NCC when investigating NIC errors, whether you have found them through NCC or Health alerts in Prism. For information on upgrading NCC, refer to KB-2871.

Note: ixgben version 1.7.1 is the minimum version required. If required, the drivers can be upgraded to ixgben version 1.7.17, the latest qualified driver.

What is a CRC error?The sending host computes a cyclic redundancy check (CRC) of the entire Ethernet frame and puts this value in the Ethernet frame's FCS (frame check sequence) section after the user payload. The intermediate switch and the destination host check this computed value against the value they compute to determine if the frame has been corrupted in transit.

Possible causes of NIC errors:

Link flap (interface continually goes up and down)Cable disconnect/connectFaulty external switch portMisconfiguration of the external switch portFaulty NIC portFaulty cableFaulty SFP+ module

Faulty cables and/or SFP+ modules are the most common cause of these errors. If the problem occurs regularly on a particular interface, you should perform testing in a controlled fashion during a change window to isolate the faulty component by swapping cables, modules, and switch ports as needed to isolate the issue.

How do I determine if the Nutanix node is experiencing NIC errors?

Log in to the hypervisor and use the ethtool command to retrieve the RX counts of the NIC interface reported in the alert. Continue checking the counters to determine if they are increasing.

ESXi

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	Or:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Depending on the driver version, the output of "ethtool -S" may differ. Some old versions do not have details like the above. In this case, you can try the command below (ESXi only):

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	We want to avoid any CRC errors, which could greatly impact usability. Collect all errors on all node interfaces and from the upstream switch.

	For each node, run the following command:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	As per the upstream switch, it depends on the vendor. For most, the command "show interface <interface>" should work from a privileged console.

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Look for the following output:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Note: For Mellanox, use:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	AHV
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Depending on the driver version, the output of "ethtool -S" may differ. Some old versions do not have details like the above. In this case, you can try this command:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	For enhanced usage of ethtool, you can incorporate additional counters to effectively detect network issues

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"
	By running this command, you can retrieve information about various counters related to network errors, faults, checksums, and frame integrity. This can assist you in identifying potential network issues that require further investigation or troubleshooting.Hyper-V
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

The two big error counters are rx_crc_errors and rx_over_errors (and, in conjunction, rx_missed_errors/rx_fifo_errors).

rx_crc_errors are caused either by faults in layer 1 (in the past, we have seen failed twinax cables and incorrect types of fibre being used), or issues with jumbo frames on the network. In an environment with 10 Gig switches that use cut-through forwarding (Cisco Nexus, Arista, and Cisco devices using IOS default to Store and Forward switching), any packets that come into the switch will get forwarded out the destination interface once the switch has read the destination MAC address. If that packet has an MTU over what is configured on the interface, it will cut off the packet at the designated MTU, causing the server to receive a malformed packet, which will throw a CRC error.

rx_over_errors are caused when the hardware receive buffer on the physical NIC is full, and some of the received packets have to be dropped at the physical NIC layer. In most cases, the values reported by this counter will equal rx_missed_errors and rx_fifo_errors. The packet drops can happen during high bursts of traffic.Note: rx_crc_errors_phy is the counter used on AHV for some Mellanox cards to count CRC errors. From NCC-4.3.0 onwards, the counter is added to the check.  For NCC versions prior to 4.3.0, this counter should be checked manually using the steps mentioned in the solution section of this KB article.

Should I be concerned that my NIC port shows any NIC errors?As indicated above, NIC errors can be triggered by several scenarios and can usually be ignored. However, continuously increasing NIC errors typically point to a failing physical layer component. The Nutanix alert is triggered when the error rate is considered excessive. This alert should be investigated whenever it is raised, and the source should be corrected.

How does the packet error handling between cut-through and traditional store-and-forward switches differ?Traditional store-and-forward switches store each Ethernet frame and perform various checks such as valid frame length and checksum check. If the frame is valid, it is switched to the destination port. In the succeeding example, if the non-Nutanix node generates a corrupted Ethernet frame, switch port 3 drops it. This behavior differs with cut-through (also known as low latency) switches, which do not store the entire frame before forwarding it. The switch usually only needs to look at the first 12 bytes of a packet to decide. The downside of cut-through processing is that a corrupted frame is forwarded to the destination. Cut-through forwarding relies on the destination (host NIC) to perform the error check and discarding, if necessary.

Example setup for scenarios 1 and 2 below:

Nutanix node A ------- NIC port 0 -------- Switchport 1Nutanix node B ------- NIC port 0 -------- Switchport 2Non-Nutanix node --- NIC port 0 -------- Switchport 3

The following two scenarios are the most common for NIC errors.

Scenario 1. NIC errors are only increasing on node A. Nutanix Node B is not experiencing increasing NIC errors.

Perform basic physical layer checks between Nutanix node A and switch port 1.
	Make sure the cables are properly connected.Check cables for physical damage, pinched fibre, and bend radius issues.If the cable is Twinax, make sure that it is supported. Nutanix ships lengths of 1, 3, and 5 meters Twinax cables. 7-meter cables might work, but the signal can be marginal and varies between switch vendors. Some switch vendors may be able to drive a signal 7 meters while others may be marginal. Use a 5-meter cable if in doubt.If the cable is optical, ensure the proper optics are used. Currently, we only support multimode short-reach optics (850 nm), and 1310 nm optics are not supported.
	Determine if the switch port causes the NIC errors.	If the NIC error is incrementing on Nutanix node A, check switch port 1 for tx output or similar transmission errors. Transmit output errors are usually a good indication that the switch port is the source of the NIC errors.Further isolation of the NIC errors.	If the external switch port does not show any NIC TX errors, then the physical layer media (switch port transceiver, cable) could be causing the NIC RX errors. Proceed if the NIC RX errors continue to increase.
	If using Twinax 10GE cabling, replace the cable. Proceed to step 3d if the NIC errors continue to increase.If using optical links, start by replacing the multimode cable. Proceed to the next step if the RX CRC errors continue to increase.If using optical links, replace the optical transceiver associated with the switch port. Proceed to the next step if the RX CRC errors continue to increase.Reboot the host.Replace the host NIC.

Scenario 2. NIC errors are increasing on all Nutanix nodes at the same time.

This scenario usually indicates the use of a cut-through Ethernet switch being used. These switches are unable to perform frame discarding at the switch input port. The first step in troubleshooting this scenario is to confirm that the cut-through switch is being used.	Check the external cut-through switch for incrementing error counters. Typically, a single switch port logs RX input errors, and the remaining ports report TX output errors. In this case, the port reporting RX input errors is attached to the node generating the illegal (CRC, length) frames.show tech from the switch or show interface for all the enabled interfaces on the switch. Look for any interfaces that are flagging RX errors.Run ethtool every minute for 5 minutes. Take rx_crc rate of increase of errors on the Nutanix interfaces.Note the Nutanix nodes that are not flagging these errors. This is to find out where the good nodes are connected to. A network diagram would be helpful.

General Troubleshooting

Remediation Steps for CRC Errors:

There are a few different ways of looking at things.If you have a layer 1 issue, you will see rx_crc_errors, not on every but some nodes.
	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	This points to this node sending out malformed packets, and we should see a related CRC error on its upstream switch port. This is because the packets coming out of this host will never be sent back to the port from which they were sent.
	If you see many rx_crc_errors on all nodes, verify that there is no "noisy neighbor" on the network.This could be that a host is broadcasting or multicasting out traffic at a larger MTU size. This process will require more network troubleshooting that may fall outside of Nutanix Support.

The general rule is that the rx_over_errors should be 1 in every 5,000 rx_packets at the most. Use the following command on an ESXi host to determine the total number of packets and the number of rx_over_packets:

[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

Remediation Steps for Missed Errors:

Check the current NCC version if you do not see any performance impact. Make sure that it is upgraded to the latest compatible version. Previous versions of NCC were aggressive in reporting the NIC error rates.If there is a need for flow control, enabling the Ethernet flow control feature will allow the physical NIC to signal the switch to slow down the packet transmission rate.More information about this feature can be found in Wiki - Ethernet flow control.

Flow control should only be enabled if it falls within the SOP for your network environment and if a considerable amount of congestion is encountered on the upstream switch. The impact on the network is as follows:

Flow control operates at a lower layer than TCP or IP and thus is independent of them. In other words, flow control can be used regardless of the higher-level protocols. An important side-effect is that neither TCP nor IP knows what Ethernet's flow control is doing. They operate assuming that there is no flow control other than what they may or may not provide themselves.Flow control functions between two directly connected network devices and flow control frames are never forwarded between links. Thus, two computers connected via a switch will never send pause frames to each other but could send pause frames to the switch itself (and vice versa: the switch can send pause frames to the two computers).Pause frames have a limited duration; they will automatically "expire" after a certain time. The expiration time is set by the device that transmits the pause frame.A paused link is not a discriminator of protocols; it will prevent data from passing across the link other than more pause frames.Another solution is to consider enabling load balancing at the network layer so both interfaces are used and also have additional buffer space. Check KB-5135 for details on how to configure AHV.

Receive Length Error

What is rx_length_error?Each error captured by a driver is explained in Intel 82599 datasheet.

Number of packets with receive length errors. A length error occurs if an incoming packet length field in the MAC header doesn't match the packet length. To enable the receive length error count, the HLREG.RXLNGTHERREN bit needs to be set to 1b. This register counts all packets received, regardless of L2 filtering and receiving enablement.

For rx_length_error, in most cases, the switch port sends BPDU packets without changing the length value properly.Use the two options below to overcome the error:

Portfast
	IOS set STP portfast:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	NXOS set port type edge:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Enable BPDU guard and BPDU filter (in Cisco/Arista switch).

	Highly recommended global BPDU Filter/Guard

	IOS:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	NXOS:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Enable BPDU Filter/Guard on a port:

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	HP Switches (Verify the HP Switch Model before running the below commands):

	[root@esxi]# ethtool -S <vmnic> | egrep "rx_errors|rx_crc_errors|rx_missed_errors"

	Also, consider disabling Jumbo Frames on the Switch ports the Nutanix Nodes are connected to (HP switches can come with Jumbo Frames enabled by default). According to the AHV Networking Best Practice Guide, enabling jumbo frames on AHV is not recommended.

Note: If you see this alert for a vUSB device, upgrade NCC to the latest version to fix the false issue.
If the steps mentioned above do not resolve the issue, consider engaging Nutanix Support for further investigation. . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.