### Human: Can you tell about NCC Health Check: cvm_dvportgroup_binding_check. ### Assistant: NCC 3.6.2. The NCC health check cvm_dvportgroup_binding_check checks if the CVM (Controller VM) NIC is connected to a dvPort group with ephemeral binding and recommends static binding if it is so.. The NCC health check cvm_dvportgroup_binding_check checks if the CVM (Controller VM) NIC is connected to a dvPort group with ephemeral binding and recommends static binding if it is so.

For virtual machines backed by dvPort group with ephemeral/dynamic binding in a ESXi environment, there is a possibility that the virtual machines may lose network connectivity after a host reboot. This is a known VMware issue described in https://kb.vmware.com/kb/1021938.

This check has been introduced for NCC version 3.6.2 and above.

Running the NCC check

This NCC check runs as part of a full health check:

nutanix@cvm$ ncc health_checks run_all

It can also be run as an individual health check:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check will run on ESXi hypervisor platforms.

This check is not scheduled to run at an interval.

This check will not generate an alert.

Sample output

For status PASS:

nutanix@cvm$ ncc health_checks run_all

Output messaging

Check ID3025DescriptionCheck for CVM dVportgroup bindingCauses of failureOne (or more) Nutanix cluster Controller VM NIC is connected to a dVportgroup with ephemeral binding.ResolutionsChange the Dvportgroup binding to staticImpactThe cluster may experience downtime.. If the ephemeral port binding is used, change it to Static binding. For instructions on setting a static binding, refer to VMware documentation - https://pubs.vmware.com/vsphere-51/index.jsp?topic=%2Fcom.vmware.vsphere.networking.doc%2FGUID-809743E1-F366-4454-9BA5-9C3FD8C56D32.html.

This is due to a pynfs issue. Nutanix used to recommend ephemeral binding when configuring dVswitch. This is no longer the case.

If you are using static binding or a standard switch, gather required logs (VMware support bundle) and contact Nutanix Support for further investigation.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about One Click Hypervisor (ESXi) Upgrade Caveats and Troubleshooting. ### Assistant: One-Click Hypervisor (ESXi) Upgrade Caveats and Troubleshooting. One Click Hypervisor Upgrade for ESXi has the following caveats that need to be kept in mind should the same fail.

Issue #1

URLDecoder: Illegal hex characters in escape (%) pattern - For input string:

This comes up if:

The Metadata and Binary files are either incorrect; orThe Binary file is chosen in place of the JSON file and vice versa

So once you click on Software Update - Hypervisor:

HYPERVISOR METADATA FILE - this is the JSON fileHYPERVISOR BINARY FILE - this is the ESXi upgrade image

Upgrade will further fail with a size mismatch if:

The JSON file and the ESXi image build numbers do not matchCorrect JSON build number and ESXi build number are chosen but the ESXi ISO image is used. For this to work, you will need the ZIP (Depot)

So essentially, using the below combination will fail either with a size mismatch or "URLDecoder: Illegal hex characters in escape (%) pattern - For input string:"

VMware-VMvisor-Installer-6.0.0-2494585.x86_64.iso6.0.0-2494585.json

Issue #2

URLDecoder: Illegal hex characters in escape (%) pattern - For input string:

URLDecoder: Illegal hex characters in escape (%) pattern - For input string:

Attempted to modify the Hypervisor Upgrade JSON file to add an ISO file.

After realizing the mistake, uploaded the correct zip file, but received the "Bundle integrity verification" error.

Issue #3

URLDecoder: Illegal hex characters in escape (%) pattern - For input string:

Issue #4

Available compatible versions show "not qualified"

Issue #5

URLDecoder: Illegal hex characters in escape (%) pattern - For input string:

. Issue #1

To resolve this issue, the correct files needed are:

VMware-ESXi-6.0.0-2494585-depot.zip and6.0.0-2494585.json

Note: For this to work, the Virtual Center Server needs to be upgraded to vSphere 6.0 Build 2494585 or later prior to the ESXi upgrade. Else, the upgrade with fail with:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

If you are using the Virtual Center Virtual Appliance (Linux based VCVA), the account that needs to be used is root because administrator@vsphere.local might fail with a username password error.

Logs will show the below:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Should the root account be used and the below messages come up, then the root account does not have access to the Virtual Center Inventory. In such a case, the root account will need to be provided access to the Virtual Center Inventory by using the administrator@vsphere.local and adding the root user to the SSO permissions so it has access to the inventory.

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Or:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

If the above message is seen, log on with the root user to the Virtual Center Server and confirm you are unable to see the Inventory Objects (ESXi hosts, Virtual Machines, etc.)

To add permission for the root user, log on to the Virtual Center Server using the administrator@vsphere.local

Log in to the Web client “https://client-hostname:9443/vsphere-client” with administrator@vsphere.local using the password of whatever you configured SSO password as. *Note you can only configure SSO using the Web Client*Navigate to vCenter Servers > Manage > Permissions, click on the + to add a userNow add the root (localhost) to vCenter along with the permissions required and click on OK.Try and login with the user you just configured and you should now have access to the Web client or the Windows vSphere UI Inventory

Should the below be seen:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

The hosts need to be a part of the vSphere Cluster in vCenter.

Admission control can cause pre-upgrade to fail at 33%.

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Should this be the case, go to Cluster Settings on Virtual Center and check the "Disable: Power on VMs that violate availability constraints".

In vCenter 6.5, this is how you disable Admission Control:

In the vSphere Web Client, browse to the vSphere HA cluster.Click the Configure tab.Select vSphere Availability and click Edit.Click Admission Control to display the configuration options.Select a number for the Host failures cluster tolerates. This is the maximum number of host failures that the cluster can recover from or guarantees failover for.Select an option for Define host failover capacity by:
	DisabledSelect this option to disable admission control and allow virtual machine power on that violate availability constraints.

	* Note the original configuration before disabling admission control so that you can put it back post-upgrade.
	Click OK.

For the One-Click to work, the following needs to be configured as these are a part of the Pre-Upgrade Checks and needs to be enabled prior to the upgrade attempt.

root user needs to have access to the Inventory ObjectsvMotion needs to be configured and enabledDRS needs to be enabled

Issue #2

Remove all downloaded files from CVMs:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Issue #3

Most of the time, this issue is because the host SSH is disabled.

Run "ncc health_checks run_all" and if you see below FAIL messages, enable SSH on the hosts and run the pre-upgrade check from Prism one click.

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Issue #4

After uploading the files (JSON and ZIP), you see a "(not qualified)" next to the ESXi version in Prism and there is a Retry button instead of Upgrade.

You see the following in the genesis.out:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Do the following:

Check MD5 value of the Hypervisor zip bundle and compare it to the vendor provided value (available from the VMware download or Dell download site in case of customized ISO)Check the MD5 value of the JSON file. Ensure that the latest available version is downloaded from portal.nutanix.com. In this case, the JSON had a different MD5 value, and downloading a fresh JSON should work.

Issue #5

Most of the time, this issue is because of the wrong credentials or wrong vCenter IP addresses.

Make sure credentials are correct and to double-check the correct vCenter IP is provided, run "hostssh grep serverIp /etc/vmware/vpxa/vpxa.cfg" and you will see the IP address of vCenter that host is connected.

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Another way to check the vCenter IP address is run "ncli ms list-management-server-info".

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491

Issue #6

The upgrade process may get stuck on a one or multiple ESXi hosts when upgrading from ESXi 6.7 to ESXi 7.0. Customer may report that the upgrade is stuck for a long time at a particular percentage.

Hypervisor logs under /scratch/host-upgrade.log on the ESXi host will show Timeout occurred during entering maintenance modevCenter task "Entering Maintenance mode" for that host will be stuck at ~70% with the message "Waiting for active NFC sessions to terminate". Manually putting the host into maintenance mode will also result in the same issue.Restarting the vpxa service on the affected ESXi host allows the host to enter maintenance mode.

Use the command:

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491
In order to resume the upgrade, restart genesis across the cluster using the below and command and monitor,

2015-09-02 00:27:54 INFO cluster_upgrade.py:351 Current host version: 5.1.0-1065491. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Alert - A130156 - DiskRemovalStuck. ### Assistant: Investigating DiskRemovalStuck issues on a Nutanix cluster.. This Nutanix article provides the information required for troubleshooting the alert A130156 - DiskRemovalStuck for your Nutanix cluster.For an overview about alerts, including who is contacted and where parts are sent when a hardware alert case is raised, see KB 1959.

Alert Overview

The DiskRemovalStuck alert is generated by Curator when no progress in the remaining disk removal tasks is detected after several scans.This is an event triggered alert that is sent once the issue is detected and is not scheduled to run on an interval.

Sample Alert

For status: Critical

alert_type: DiskRemovalStuck

Output messaging

Check ID130156DescriptionDrive Removal StuckCauses of failureDrive Removal Stuck.ResolutionsContact Nutanix support.ImpactDrive cannot be removed from the cluster.Alert IDA130156Alert TitleDrive Removal StuckAlert Smart TitleDrive disk_id Removal Stuck.Alert MessageRemoval of drive disk_id is stuck.. Troubleshooting

The DiskRemovalStuck alert is generated by Curator when no progress is detected upon several scans.Disk removal is a multi-stage process which, depending on whether a disk in question is the metadata drive or not and which features are in use, will need to migrate Oplog, NearSync store, Cassandra metadata, Extent Store data, etc, in specific order. For bigger disks, the system needs to evacuate and redistribute terabytes of data in the background, without affecting VMs operation and without compromising resiliency. This naturally takes time.

As per HW replacement documentation: "Estimated time to complete: Variable, can take up to several hours".

Resolving the Issue

There are multiple possible causes for drive or node removal getting stuck. Consider engaging Nutanix Support at https://portal.nutanix.com/ to investigate and resolve this issue.

Collecting Additional Information

Before collecting additional information, upgrade NCC. For information on upgrading NCC, see KB 2871.Collect the NCC output file ncc-output-latest.log. For information on collecting the output file, see KB 2871.Collect Logbay bundle using the following command. For more information on Logbay, see KB 6691.
	nutanix@cvm$ logbay collect --aggregate=true

Attaching Files to the Case

Attach the files at the bottom of the support case on the support portal.If the size of the NCC log bundle being uploaded is greater than 5 GB, Nutanix recommends using the Nutanix FTP server due to supported size limitations. See KB 1294.

Requesting Assistance

If you need further assistance from Nutanix Support, add a comment to the case on the support portal asking Nutanix Support to contact you. If you need urgent assistance, contact the Support Team by calling one of our Global Support Phone Numbers. You can also click the Escalate button in the case and explain the urgency in the comment, and Nutanix Support will be in contact.

Closing the Case

If this KB resolves your issue and you want to close the case, click on the Thumbs Up icon next to the KB Number in the initial case email. This informs Nutanix Support to proceed with closing the case. You can also update the support case saying it is okay to close the case and Nutanix Support will close the case.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about VM migration on AHV cluster may fail with "Host CPU does not provide required features by guest: monitor" error. ### Assistant: After adding new Lenovo hosts or hosts with Intel Xeon Gold 61xx or Xeon Silver 41XX CPUs to an existing cluster, VM migration to these hosts may fail with the error message "Host CPU does not provide required features by guest: monitor.". After adding new Lenovo hosts or hosts with Intel Xeon Gold 61xx or Xeon Silver 41XX CPUs to an existing cluster, VM migration to these hosts may fail with the error message "Host CPU does not provide required features by guest: monitor." as seen on Tasks page in Prism UI.

~/data/logs/acropolis.out shows the error below:

2018-08-23 15:04:55 INFO base_task.py:611 Task VmMigrate fabf5884-9a85-40be-b6e1-ff652913d232 failed with message: Host CPU does not provide required features by guest:  monitor. This could be due to the MONITOR/MWAIT feature either missing or disabled from the host BIOS. Lenovo servers may have this feature disabled by default.Boot the host into BIOS Setup and enable the MONITOR/MWAIT feature. Below are sample screenshots of BIOS CPU/processor configuration screens where the MONITOR/MWAIT feature can be located and enabled.

Save changes and exit BIOS Setup. VMs should now be able to migrate to this host.NOTE: If the MONITOR/MWAIT option is greyed out and you cannot change it you will need to go into the BIOS option for Operating Mode and set it to "CUSTOM" which will enable MONITOR/MWAIT.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Generic AHV troubleshooting scenarios. ### Assistant: This article provides an overview of generic AHV troubleshooting scenarios.. This article provides a generic description of common issues that can be seen on clusters running AHV and propose troubleshooting methods.

Common scenarios:

Unreachable hostHost in a crash loopUnexpected host rebootUnexpected guest VM rebootInability to perform VM management actionsStuck VM migrationFailed VM live migration due to Jumbo frames misconfiguration.. Logs are the most important source of information when performing troubleshooting. You can find below the list of logs on the CVM (Controller VM) and on the AHV host, which should be checked during troubleshooting.

CVM logsThe most important log file on a CVM to check while troubleshooting AHV related issues is /home/data/logs/acropolis.out as the Acropolis service coordinates most of the activities. Always check acropolis.out on the CVM hosting the Acropolis leader. KB 2305 describes steps on how to find the Acropolis leader.

To find which CVM is the Acropolis leader from logs or to show leader changes if any have occurred recently, run the following command:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Example:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

If the log file on the Acropolis leader does not contain relevant details, consider checking acropolis.out on other CVMs.

Note: acropolis.out is updated with new events together with any action against user VM. Lack of updates means the service may not be healthy on the CVM.

Host logs

Important logs are:

Log fileDescription/var/log/libvirt/libvirtd.logEvents related to libvirtd service/var/log/libvirt/qemu/NTNX-<HOST>-CVM.logThe QEMU log for the CVM on this host./var/log/libvirt/qemu/<vm uuid>.logEvents related to VM lifetime (start, stop, migrate). To find a specific VM's UUID, use acli vm.list from a CVM and take note of that VM's UUID. You can find out which host that VM is currently on by checking the output of acli vm.get <VM_NAME> and taking note of the host_name field./var/log/libvirt/qemu/hook.logContains some basic information on calls that Acropolis makes into QEMU/var/log/acropolis_ovs.logHost-level OVS-related changes and queries from the Acropolis service are logged here on each host. You should see logs when new taps are created, changed, or destroyed for VMs running on this host.ovs-ctl.log, ovsdb-server.log, ovs-vswitchd.log in /var/log/openvswitchEvents generated by ovs-vswitchd service/var/log/iscsi_redirectoriSCSI Redirector process is what manages iSCSI connectivity between the user VM's vdisks on AHV hosts and the Stargate processes on CVMs. iSCSI redirector is what AHV uses to redirect vdisks to be hosted by other CVMs when the local CVM is inaccessible via iSCSI (e.g. CVM is down or Stargate is having issues)./var/log/NTNX.serial.out.0The CVM's console log. If a CVM is powered on, but not reachable via SSH or is in a boot loop, you can check this file to view the CVM's console output.

Note: Always check the timezone configured on CVM and AHV hosts while correlating events. The default timezone on the AHV host is UTC and the default timezone on CVM is PST. The timezone on the AHV host should not be modified. Refer to the Nonconfigurable AHV components section of the AHV Administration Guide. If the timezone on the AHV host was changed, it will be automatically reverted back to UTC on the next AHV upgrade.

Scenarios

1. Unreachable host

A host can become non-accessible due to networking issues or due to the inability to boot the hypervisor.

1A. Host unreachable due to networking issues

Perform the following tests to identify the scope of the issue:

Does the host respond to pings from another host in the same subnet (other AHV hosts for example)?Does the host respond to pings from another host in a different subnet?

If an AHV host responds when queried from hosts in the same subnet but does not respond to hosts in other subnets, make sure that routing is correctly configured on network devices.

If connectivity to the host is intermittent, make sure that the host's IP address is not used by other devices on the network. Perform steps described in the Solution section of KB 3025 to host with the conflicting IP address.

If a host is not reachable, check if the IPMI web interface is available. If not, physical access to the server is required to check power status and network cabling.

If you can log in to the AHV host via IPMI console, perform basic network troubleshooting:

Check the configuration of the br0 bridge and make sure it is correct:
	nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
	Check br0 configuration:
	From AHV host, connect to CVM:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		From CVM, run the following commands:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		In the output of commands mentioned above, make sure that correct NICs are assigned to br0 and other bonds. For example:
		nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
		Update bridge configuration by following steps from KB 8835.Check which NIC is active if active/pass mode is used by following the steps described in How to select active NIC in active/passive bond section of KB 2090.

1B. Host unreachable due to inability to boot the hypervisor

If the connection to the IPMI interface is working fine, then open the server console and check the status of the host. Following scenarios are possible:

The host is powered off. Power it on remotely via IPMI.The host is powered on but stuck during BIOS initialization. Check the hardware event log for hardware-related issues.The host is powered on but is in a crash loop (scenario #2 below).

2. Host in a crash loop

If the host is constantly rebooting, try making the screenshot of the call stack shown on the screen and search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support at https://portal.nutanix.com/.

3. Unexpected host reboot

If the host was unexpectedly rebooted, perform the following checks to scope the issue:

Were there any upgrades performed on the cluster at the same time?Were any events happening in the environment, like power or overheating issues, that could affect the hardware? Check the IPMI event log to verify it.

Collect basic information about host uptime, last reboot date:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Check IPMI event log for hardware errors. In addition to this, check contents of /var/log/mcelog file if it exists. Make sure that the timestamp recorded there matches the date and time of the unexpected reboot.

Sample:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

Run ahv_crash_file_check NCC check to verify if any core dump files are present:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

More details about this check can be found in KB 4866.

If ahv_crash_file_check finds any issues, /var/crash folder on the affected host will contain one or more subfolders with the date and time of crash specified in the name.

Each of these subfolders may contain one of the below:

vmcore and vmcore-dmesg.txt if core dump was successfully created.vmcore-dmesg-incomplete.txt if core dump creation was aborted due to some external factor (like lack of free space for example).

Review vmcore-dmesg.txt and vmcore-dmesg-incomplete.txt files.

Check /var/log/messages and messages.<number>.gz files for any critical events preceding host reboot.Check if the AHV host abruptly rebooted. It generates a series of "recovering journal" and checks events under /var/log/journald_boot.log similar to:

nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
Search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support.

4. Unexpected guest VM reboot

Perform the following steps to identify the scope of the issue:

Write down the names of affected VMs and the time or reboot.Make sure that the VM reboot was not caused by the host reboot by following the steps in the Unexpected host reboot section above.Make sure that all hosts are accessible over the network. If not, then perform the steps described in the Unreachable host section above.Verify that guest OS is supported as per Compatibility Matrix.Run cluster_services_status NCC check to make sure that services on CVM are in a stable state:
	nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
	Check logs inside guest VM as unexpected restart may be a result of issues with the guest OS itself. Log names can be different on different OSes. On Windows VMs, check System and Application event logs. On Linux VMs, check /var/log/messages and messages.<number>.gz files for any critical events preceding host reboot. Consider engaging an OS vendor if you see evidence of issues inside guest OS.Check QEMU logs on the AHV host and compare the signature to find how reboot was triggered:
	Get the VM UUID:
		Prism UI: VM details in VM pageNCLI:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

		Go to the host where the VM was running and check the log file in /var/log/libvirt/qemu/<vm-uuid>.log
		It is possible to find where the VM was running by looking on which host log files are present and checking the last lines to find out the most recent log:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
			The output will be displayed for every host where the VM ran, but only one of them will have the most recent date:
			nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

		Check for the following signature in /var/log/libvirt/qemu/<vm-uuid>.log

		Power OperationLog Snippet"shutdown -P now" in Guest
					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

					"shutdown -r now" in Guest or

					NMI (Kernel panic) - check "NTNX.serial.out.0" log

					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

					"virsh shutdown <VM>" or

					VM power operation in Prism

					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh reboot <VM>" or
					VM power operation in Prism

					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh reset <VM>" or
					VM power operation in Prism

					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'
					"virsh destroy <VM>" or
					VM power operation in Prism

					nutanix@cvm$ allssh 'grep -i Acquired ~/data/logs/acropolis.out*'

If all factors mentioned above are excluded, consider checking /home/data/logs/acropolis.out on Acropolis leader and /var/log/libvirt/qemu/<vm uuid>.log on AHV host where VM was running before restart.

5. Inability to perform VM management actions

In most cases, the inability to perform VM management actions from Prism or acli on one or more AHV hosts is caused by issues with the Acropolis service itself. Start with checking /home/data/logs/acropolis.out on the host where affected VM is located and on Acropolis leader. Look for the "Traceback (most recent call last)" text in the log to identify the start of the stack trace, which indicates that the Acropolis service is not healthy.

Search for known issues on the Nutanix portal. If no matching KB articles can be found, consider engaging Nutanix Support.

6. Stuck VM migration

In most scenarios, VM migration is not stuck, but processing very slowly. Refer to KB 3810 for additional details.

7. Failed VM live migration due to Jumbo frames misconfiguration.

If the Jumbo frames are used, but the physical network switches do not have the correct configuration to allow packets with higher MTU size, the live migration of VMs will fail. Refer to KB 10180 for more information.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Deleted VM disk attached to Windows VM is accessible from guest OS. ### Assistant: A deleted VM disk attached to a Windows VM may be accessible from within the guest OS.. When a VM disk, attached to a Windows VM running on AHV, is deleted from Prism UI may still be online and accessible within the guest OS via File Explorer even after deletion.Performing a refresh operation from the Windows Disk Management utility changes the disk status to "Not Initialized", but the disk is still mounted and accessible from the guest OS.. To completely remove the disk from the Windows guest OS, we need to perform a "Rescan Disks" operation from the Windows Disk Management utility.To perform a rescan operation, follow the steps below.
Open Disk Management utility in the guest OS.Open the "Action" menu in the Disk Management utility.Select "Rescan Disks".

The deleted disk should now be completely removed from the guest OS.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about VMware vCLS insfrature VMs and Robo licenses. ### Assistant: ROBO 1.0 Licensing Model and VMware vCLS infrastructure. ROBO 1.0 licenses (Per-VM licensing) are sold according to a VM count.  ROBO licenses may be purchased based on how many VMs are planned to run within a particular cluster.*Note: This count does not include the Nutanix CVMs and the powered-off VMs. As of VMware vSphere version 7.0 Update 1, an additional VMware Management VM called "VMware vCLS infrastructure" was added. It is responsible for the new vSphere Cluster Services feature. This KB explains why this VMware Management VM is included in the VM count for ROBO licenses even though it is not a user VM.. The "VMware vCLS infrastructure" VM will be counted under the ROBO licensing because it is a VMware management VM; it is not a Nutanix management VM.
Please refer to the screenshot and documentation below for more information. vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1 and newer versions (80472). ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Unable to connect to Prism Central, getting error "Failed to send request to IAMv2". ### Assistant: Unable to connect to Prism Central, getting error "Failed to send request to IAMv2". Versions Affected:

Prism Central (PC) version pc.2023.1.0.1

Symptoms:

Prism Element (PE) cluster is unable to connect to Prism Central.PC-UI login is failing with the following message:
	upstream connect error or disconnect/reset before headers
	All cluster services are up and running, but unable to log in to PC.PC /home/nutanix/data/logs/aplos_engine.out shows the following errors:
	upstream connect error or disconnect/reset before headers
	/home/nutanix/data/logs/prism_gateway.log shows the following:
	upstream connect error or disconnect/reset before headers
	iam pod is in crash loop.
	upstream connect error or disconnect/reset before headers
	. Workaround:

Log in to the Prism Central node where the issue is seen and reboot it using the following command:

nutanix@PCVM$ sudo reboot. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Nutanix DR | AOS 6.5.2.x stuck arithmos replication stats when there are no active replications. ### Assistant: AOS 6.5.2.x stuck arithmos replication stats when there are no active replications. In Protection Summary,Replication Tasks dashboard ongoing tasks shows incorrect number.. In PC, Protection summary global view, under Replication Tasks dashboard, ongoing replication count increments.
To identify the issue, login to PC, go to Protection summary global view, under Replication Tasks dashboard, ongoing replication count keep increments but when you check the current active replications on the corresponding PE clusters, you will see sometimes no replications or very few replications in progress.

From the PE clusters, we can verify the ongoing replications using ncli

nutanix@cvm:~$ ncli pd ls-repl-status protection-domain-type=entity-centric

Login to PC, go to Protection summary global view, under Replication Tasks dashboard, check ongoing replication count
 . Please reach out to Nutanix Support to reset the stats for stuck replications.
 . ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Nutanix Files: Unable to join domain due to 'constraint violation'. ### Assistant: Unable to join Nutanix Files to AD domain due to duplicate SPNs existing in the AD forest.. If there is an AD SPN with the duplicate name as the Nutanix Files server or FSVMs in the Active Directory domain or forest, the below error will be presented.
Join AD failed with error - Failed to set Service Principal Names (SPN) entries on the AFS machine account due to 'constraint violation'.
To validate, run the below command on a Domain Controller to attempt to manually create a SPN for Nutanix Files as a test.
Join AD failed with error - Failed to set Service Principal Names (SPN) entries on the AFS machine account due to 'constraint violation'.
Example

Join AD failed with error - Failed to set Service Principal Names (SPN) entries on the AFS machine account due to 'constraint violation'.
You will get the below error if there is a duplicate object in the domain or the forest.

Join AD failed with error - Failed to set Service Principal Names (SPN) entries on the AFS machine account due to 'constraint violation'.
 . The solution is to rename the Nutanix Files server to a unique name and then rejoin Nutanix Files to the AD domain.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Debugging ESX Machine Check Exception (MCE) PSOD. ### Assistant: This article describes how to debug ESX Machine Check Exception (MCE) purple diagnostic screen (PSOD) with an example.. ESXi servers may halt with a purple diagnostic screen (PSOD) due to MCE (machine check exception) errors. MCE errors are due to hardware issues.

The MCE purple diagnostic screen displays multiple things. Out of these, the MCi_STATUS register is useful in debugging the problem.

Register Details:

The MCi_STATUS register consists of information about the machine check error.The register is displayed next to the “S:” as shown in the example below:
	MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5

	Bits and their significance:

	6362616059585753-5638-5232-3716-310-15VALID flag – If set, then information is validOVERFLOW flag – If set, then may indicate multiple MCEs occurred close to each otherUNC flag- If set, then CPU could not correct the errorEN flagMISCV flag – If set, then MISC register contains more informationADDRV – If set, then the ADDR register contains where the error occurredPCC flag- If set, then it means processor may have been corrupted.Architectural if bit 11 is set, else “other information”Architectural if bit 10 is set, else “other information”Other informationModel specific error code for CPUMachine check error code

To debug the error, the low 16 bits of the MCi_STATUS register are important. These bits indicate a simple or compound error.

Option 1: Using Automatic Tool

NOTE: You can now debug the error with the following tool that automates the steps below:
	https://vmware-psod-reader.github.io/vmware-psod-reader/

Option 2: Using Manual Steps

Simple errors are easy to debug
	MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
	Compound errors
	MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5

	F – Form flag
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5

		Filtering means that some or all of the subsequent corrections to this entry in this structure are not posted.
		TT - Applies to 2 and 4 above.
		Indicates type of transaction:

		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		LL - Applies to 1, 2, 4 and 5 above.
		Indicates the level in the memory hierarchy where the error occurred.

		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		RRRR - Indicates type of action associated with the error. Actions are:
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		PP (Participation) - Describes the role of the local processor in the error.
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		T (Time-out) - 1 = Request timed out.II (Memory or I/O)
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		Memory Controller errors - Defined by MMM and CCCC subfields above.MMM – Memory error
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5
		CCCC – Channel with the error
		MC: PCPU18 B:13 S:0xfe20004000011166 M:0x7246040086 A:0x38c989b100 5

	. Here is an example of how to analyse an MCE purple diagnostic screenshot:

Note down the MCi_STATUS register value is 0xfe20004000011166.Convert it to binary:
	1111 1110 0010 0000 0000 0000 0100 0000 0000 0000 0000 0001 0001 0001 0110 0110
	Take the lower 16 bits of the MCi_STATUS register:
	1111 1110 0010 0000 0000 0000 0100 0000 0000 0000 0000 0001 0001 0001 0110 0110

	Compare it with the compound errors. In this case, this appears to be a cache hierarchy error (type 4).
	Apply the values:
	1111 1110 0010 0000 0000 0000 0100 0000 0000 0000 0000 0001 0001 0001 0110 0110

Conclusion:

It looks like the purple diagnostic screen occurred while there was a prefetch operation on some data in the L2 cache of the processor.Thus, this could be a problem with the L2 cache on the processor.Therefore, we should replace the CPU first and check if the issue is resolved. In case the issue comes up again, we could replace the motherboard.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about LCM not listing all available Redfish updates on nodes and might show "System does not meet Redfish requirements". ### Assistant: LCM does not list all of the available Redfish on the nodes due to the internal interface not being reachable, and shows "System does not meet Redfish requirements" for one or more nodes.. A new issue has been identified in LCM 2.4.x, which does not list all of the available Redfish updates on the nodes due to the internal interface not being reachable. This affects only G6/G7/G8 hardware (not supported on G4/G5 hardware).Only certain host versions are available for the Redfish BIOS update in LCM 2.4.x. Ensure the host meets all requirements and is eligible for a Redfish BIOS update. If the system does not meet Redfish requirements, LCM will continue using the legacy(using Phoenix) method to perform the upgrades.There are two scenarios associated with this same issue.Scenario 1LCM inventory page will show "System does not meet Redfish requirements" for one or more nodes:

Log in to LCM leader CVM, to find the LCM leader run the below command from the CVM:

nutanix@CVM:~$ lcm_leaderLcm leader is at <CVM_IP>

Run the below command from the node that shows the error "System does not meet Redfish requirements":

nutanix@CVM:~$ grep -i 'Max retries exceeded with url:' -B1 ~/data/logs/lcm_ops.out

[2021-02-02 10:36:11.285249] GET - https://169.254.3.254/redfish/v1/odata[2021-02-02 10:36:14.369178] Error while calling https://169.254.3.254/redfish/v1/odata. Details: HTTPSConnectionPool(host='169.254.3.254', port=443): Max retries exceeded with url: /redfish/v1/odata (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f9155d11990>: Failed to establish a new connection: [Errno 113] No route to host',)).

The above logs indicate errors while trying to call internal IP 169.254.3.254.Scenario 2If the BMC/BIOS version is already on the latest version, the LCM inventory page will show "No available updates" for the BIOS Redfish option. Instead, it will show legacy BIOS update options available in LCM for one or more nodes in the cluster:On the node that is not displaying Redfish updates, run the following command :

nutanix@CVM:~$ grep -i 'Failed to ping the internal IP' -A6 ~/data/logs/lcm_ops.out

[2021-05-03 01:31:12.311761] Generic ping at IP: 169.254.3.254[2021-05-03 01:31:33.355680] cmd: ping -c 1 -W 1 169.254.3.254, out: PING 169.254.3.254 (169.254.3.254) 56(84) bytes of data.

--- 169.254.3.254 ping statistics ---1 packets transmitted, 0 received, 100% packet loss, time 0ms, ret: 1, err:[2021-05-03 01:31:33.355847] Failed to ping 169.254.3.254[2021-05-03 01:31:33.355871] No. of retries left: 0[2021-05-03 01:31:33.355891] Failed to ping the internal IP on AHV, resetting the interface

The above log snippet indicates errors regarding failure to ping the BMC internal IP -169.254.3.254.
. Save the IPMI configuration file for backup before implementing the workaround. Customers need to save the current configuration because in step-2 IPMI will reset to factory default setting. IPMI will be restored back to the current configuration once the customer performs step 3.

Note: on ESXi all the ipmitool commands should start with a /. For example:

root@ESXi:~$/ipmitool raw 0x30 0x4a

Download the IPMI configuration of the affected node via GUI or from the command line:
	GUI from the IPMI:
		Log in to IPMI Navigate to "Maintenance > IPMI Configuration"

			Click on Save which will download the .bin file.
		Command-line from the host:
		root@ESXi:~$/ipmitool raw 0x30 0x4a

	Reset the IPMI to factory default from the command line or GUI:
	Reset via GUI from the IPMI:
		Navigate to “Maintenance -> Factory Default” as below, select option "Remove current settings but preserve User configurations" and then click on "Restore":

			Click on "Yes":

			Wait for at least 2 minutes. IPMI will redirect back to the login page.
		Confirm IPMI is accessible.
	Reset via command line from the host:
	root@ESXi:~$/ipmitool raw 0x30 0x4a
	Reload IPMI configuration from the host if IPMI is not accessible. The IPMI may reboot after restoring the configuration. Wait for 120 seconds in order to let the IPMI complete the initialization.
	root@ESXi:~$/ipmitool raw 0x30 0x4a
	Re-run LCM inventory.

If the above steps don't fix the issue, contact Nutanix Support with the downloaded IPMI configuration file from step 1.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Nutanix Database Service | Debugging Windows DB Server VM Provisioning Failures. ### Assistant: This article describes debugging Windows DB server VM provisioning failures.. Note: Nutanix Database Server (NDB) was formerly known as Era.

Scenario 1: Sysprep fails when antivirus software is installedSysprep of a VM could fail when antivirus software is installed in the system.

Operation logs /home/era/era_base/logs/drivers/sqlserver_database/create_dbserver/<operation_id>.log or /home/era/era_base/logs/drivers/sqlserver_database/provision/<operation_id>.log:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

From sysprep logs /home/era/era_base/logs/drivers/sqlserver_database/create_dbserver/<operation_id>_syspre_logs/Panther/setupact.log or /home/era/era_base/logs/drivers/sqlserver_database/provision/<operation_id>_syspre_logs/Panther/setupact.log:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

Scenario 2: Joining a workgroup/domain failsIf the software profile hostname is longer than 15 characters, joining a workgroup gets stuck.

This is identified by checking sysprep_task.log from the zip file generated by the sysprep failure. If the last line is workgroup join, the issue is identified. See sample output below:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

Another situation where this issue can be seen is if there is a domain join failure. In this case, there is a file called sysprep_task_domain.log. The domain join failure would be present in this log file.

Below are some of the reasons for a domain join failure:

The custom domain OU path is incorrect.The credentials provided do not have the necessary permission to join the domain.The credentials are incorrect.Another VM with the same name exists in the domain already.

Scenario 3: Network connectivityWhen there is no network connectivity between the Era server/agent and the DB server VM, the WinRM connection from the Era server/agent would fail with the following message:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

Scenario 4: Disabling WinRM as part of GPO policyWhen the DB server is added to the domain, and you have an active GPO policy that might disable any remote WinRM connection to the newly provisioned DB server, the provisioning would fail with WinRM connectivity failures.

Scenario 5: Missing virtIO installationWhen the software profile does not include VirtIO (a collection of drivers), the DB provisioning process cannot add NICs to the DB VMs. This will cause the provisioning process to fail. Here is the error message in the /home/era/era_base/logs/drivers/sqlserver_database/create_dbserver/<operation_id>.log:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

Sample screenshot:

Scenario 6: Locked service accountWhen the Era worker service account is locked in the domain, DB provisioning fails with the following error message in /home/era/era_base/logs/drivers/sqlserver_database/create_dbserver/<operation_id>.log:

[2022-08-01 10:10:33,189] [140408303626048] [INFO    ]  [0000-NOPID],Got an successful status code

The domain user in the above operation log is the SQL Service Startup Account used when provisioning the DB Server:

Go to the Active Directory -> Users and Computers, and search for this account. You may see that this account is locked.

. Resolution 1: Sysprep fails when antivirus software is installedAntivirus software should be disabled on the software profiles prior to provisioning.

Disable or delete the antivirus software service (see the antivirus software's documentation on how to do this).Take a new software profile (see Creating a Software Profile in the NDB User Guide).Perform provision DB server operation (see SQL Server Database Provisioning in the NDB User Guide).

Resolution 2: Joining a workgroup/domain failsChange the hostname of the source VM to less than 15 characters, or engage Nutanix Support to skip joining a workgroup as part of the provision.

If the sysprep_task.log does not have more than 4-5 lines of output, this is a different issue with the same error message. In this case, engage Nutanix Support for assistance.

To resolve a domain join failure:

Correct the OU path.Give different credentialsDelete the entry of the same name or provision with a new name.

Resolution 3: Network connectivityEnsure that the Era server/agent can connect to the subnet where the VM is being deployed.

Resolution 4: Disabling WinRM as part of GPO policyEnable WinRM remote connection to the host via GPO.

Resolution 5: Missing virtIO installationInstall VirtIO in the gold image DB VM, then create a new software profile from it. For the VirtIO installation, refer to Nutanix VirtIO for Windows.

Resolution 6: Locked service accountUnlock the service account in the domain by ticking the checkbox.

. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Move: How to move a VM from one cluster to another. ### Assistant: How to move a VM from one cluster to another. There are several ways to move a VM from one cluster to another.. To move a VM from a non-Nutanix cluster to a Nutanix cluster, use Nutanix Move available on the Nutanix Portal.
	Nutanix Move is a cross-hypervisor mobility solution to move VMs with minimal downtime.
	To move a VM from one Nutanix cluster to another, use Asynchronous Disaster Recovery Between On-Premise Data Centers.
	Summary of steps:

	Configure a Remote Site on cluster A.Create a Protection Domain on cluster A.Add the VM to be moved to this Protection Domain.Power off the VM.Create a schedule for (N) hours temporarily and initiate a full replication.After replication is complete, restore the VM from snapshot on cluster B and power on the VM.Unprotect the VM from Protection Domain, delete the schedule, snapshots and Protection Domain on cluster A.Un-configure the Remote Site on cluster A.

	The same result can be achieved using Asynchronous Entity Centric Replication using Leap. Check the following documentation for more details.
	To move a VM from an AHV to an ESXi cluster, export it in .ova format from the source and import it in the target cluster.
	Summary of steps:

	For AOS 5.18 or newer:

			Select the Export as OVA action from the Actions dropdown list on Prism Central UI. For more details on how to perform the Disk export, refer to Exporting VM as an OVA in the OVA Management section of the Prism Central Infrastructure Guide.

		For AOS versions prior to 5.18:

		Follow the steps described in the Solution part of KB 3059.
		Deploy the exported VM into the ESXi cluster using the steps described in the following VMware documentation.
	From AOS 5.19, the VM can be Cross Cluster Live Migrated using Leap and AHV SyncRep. Check the following documentation for more details.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: cluster_services_down_check. ### Assistant: The NCC health check cluster_services_down_check verifies if the cluster services are running on all Controller VMs in the cluster. If any cluster service is not running on any of the Controller VMs in the cluster, this check fails.. The NCC health check cluster_services_down_check verifies if the cluster services are running on all Controller VMs in the cluster. If any cluster service is not running on any of the Controller VMs in the cluster, this check fails.

Running the NCC CheckYou can run this check as part of the complete NCC Health Checks:

nutanix@cvm$ ncc health_checks run_all

Or you can run this check separately:

nutanix@cvm$ ncc health_checks run_all

You can also run the checks from the Prism web console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every minute, by default.

Sample outputFor status: PASS

nutanix@cvm$ ncc health_checks run_all

For status: FAIL

nutanix@cvm$ ncc health_checks run_all

For PC (Prism Central) ClusterThis check verifies whether all services are running on PC VM.

Note: If the Cluster Health service status is DOWN for more than 15 minutes, an alert email is sent by the AOS cluster to configured addresses and Nutanix Support (if selected). In this case, no alert is generated in the Prism web console. The email is sent once per 24 hours. You can run the NCC check cluster_services_down_check to see the service status.

Output messaging

Check ID3032DescriptionCheck that no services are down in the cluster.Causes of failureOne or more services in the cluster are down.ResolutionsIf this check occurs once or infrequently, no action is necessary.ImpactCluster performance may be significantly degraded. In the case of multiple services with the same condition, the cluster may become unable to serve I/O requests.Alert IDA3032Alert TitleCluster services are downAlert Smart TitleCluster services down on Controller VM ip_addressAlert MessageCluster Service: service_name is down on the Controller VM ip_address.. Run the following command to confirm if any service is currently down:

nutanix@cvm$ cluster status | grep -v UP

If any services are listed as DOWN, start those services by using the following command:

nutanix@cvm$ cluster status | grep -v UP

If there are issues starting some services, check the following:

	Check if the node is in maintenance mode by running the ncli host ls command on the CVM. Verify if the parameter Under Maintenance Mode is set to False for the node where the services are down. If the parameter Under Maintenance Mode is set to True, remove the node from maintenance mode by running the following command:

	nutanix@cvm$ cluster status | grep -v UP

	You can determine the host ID by using ncli host ls.

	See the troubleshooting topics related to failed cluster services in the Advanced Administration Guide available from the Nutanix Portal's Software Documentation page. (Use the filters to search for the guide for your AOS version). These topics have information about common and AOS-specific logs, such as Stargate, Cassandra, and other modules.

	Check for any latest FATALs for the service that is down. The following command prints all the FATALs for a CVM. Run this command on all CVMs.

	nutanix@cvm$ cluster status | grep -v UP

If a service does not start, or if you see FATAL logs for the service that is down, collect the logs and contact Nutanix Support for assistance.
Collect logbay bundle for the timestamp when the alert was noticed. For more information on logbay, see KB 6691.

For PC VM

Check if any services are down using the below command:

	nutanix@cvm$ cluster status | grep -v UP
	If any service is listed as DOWN, start that service using the below command:
	nutanix@cvm$ cluster status | grep -v UP
	Verify if all services are running using the command provided in step 1.

If the above commands do not help and you still see a service reported as DOWN, open a case to engage Nutanix Support at https://portal.nutanix.com/.
Collect logbay bundle for the timestamp when the alert was noticed. For more information on logbay, see KB 6691.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Move : Online upgrades from newly deployed Move 4.1.0 appliances are not working. ### Assistant: This article describes an issue where online upgrades from newly deployed Move 4.1.0 VMs are not working.. Online upgrades from newly deployed Move 4.1.0 VMs are not working.

Troubleshooting:

SSH to the Move VM. (See Accessing Move VM with SSH.)

Run the /opt/xtract-vm/scripts/check_upgrade_connectivity.sh script. It shows the error "curl: no URL specified!"

For example:

admin@move on ~ $ sh /opt/xtract-vm/scripts/check_upgrade_connectivity.sh. This issue is fixed in Move 4.1.2. Use Move 4.1.2 or later.

Workaround:

Perform the following steps:

SSH to the Move VM.Type the rs command and enter the password.Open the check_upgrade_connectivity.sh file in an editor and delete the contents of the file.
	admin@move on ~ $ vi /opt/xtract-vm/scripts/check_upgrade_connectivity.sh
	Copy and paste the following content:
	admin@move on ~ $ vi /opt/xtract-vm/scripts/check_upgrade_connectivity.sh
	Retry Upgrade Software from the Move UI.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Performance benchmarking with CrystalDiskMark on Nutanix. ### Assistant: Specifics about using the CrystalDiskMark on Nutanix. CrystalDiskMark is an open-source disk drive benchmark tool for Microsoft Windows. Based on Microsoft MIT-licensed diskspd tool, this graphical benchmark is commonly used to test storage performance. It generates a workload with a varying number of queues and threads. The detailed information about using the parent tool diskspd can be found in this Nutanix KB article here. If there is any help needed feel free to open a support case to help you with CrystalDiskMark. . CrystalDiskMark default settingsCrystalDiskMark benchmark is very simple to use and provides a one-click experience for storage benchmarking. However, the default settings are not a summary for testing enterprise-level storage but rather built for PC/laptop local disk testing. Please use the latest version of CrystalDiskMark for testing.Let's take a closer look at the settings of the CrystalDiskMark:Figure 1: Default CrystalDiskMark settings

Working SetBy default, CrystalDiskMark is set to a 1GiB test file size. It is recommended to test with a realistic working set size (the "hot" data which is frequently read and written) of the production application or at least get closer to the size that simulates the real life behaviour. Nutanix as a distributed architecture uses DRAM as read cache as well as Oplog as a write cache for random data and low parallel sequential write patterns. Hence the 1 GiB data will get into Oplog first and later into DRAM while parts of the test data will potentially get read from Oplog. This is unlikely to happen in real life. While customers often compare CrystalDiskMark results on Nutanix with existing SAN based storage systems and local storage solutions, the 1 GiB dataset will come 100% from the SAN cache or Raid Controller cache for both reads and writes and will show how this extremely small dataset with a short run time would behave differently.The below example illustrates the 100% reads from DRAM on a Nutanix based system:Figure 2: Reads with DRAM cachingFor this reason, it is important to understand what the working set would be before doing the benchmarking. Typically, the application owners can provide such information. For spot testing, we recommend using at least a 16 GiB file to bypass the controller cache for both Nutanix and SAN based storage to see realistic synthetic storage performance benchmarking results.

Queues and ThreadsCrystalDiskMark allows setting queues and threads. All four of the default tests are single-threaded, which does not show a realistic I/O profile for the vast majority of workloads as most modern applications are multi-threaded. With Database products like Microsoft SQL, the default of parallel execution is set to a given amount of CPUs configured in the VM. More information about MAXDOP (max degree of parallelism) can be found here. It is highly recommended to understand the application as best as possible and identify whether it uses single-threaded or multi-threaded processing before doing the benchmarking. The test should be as close to the production workload as possible to best understand what the performance for a workload will be.

The amount of queues is the amount of I/O that each thread is generating at once (OIO - Outstanding IO). Both of the default random I/O tests are set to 4K block size. While 4K is a block size that can be seen at times, it is usually not a common block size in most workloads today. When possible the block size in the test should match the application's block size. The common block size for both Oracle and MS SQL is 8 KB as well as 64 KB. While these applications have variable block sizes dependent on the work happening, 8 KB and 64 KB are very commonly used block sizes in the field.CrystalDiskMark tests settings can be adjusted in the "Settings" menu. CrystalDiskMark allows a user to specify the following parameters:

	Type (sequential or random)

	Block Size

	Queues

	Threads

Suggested example for testing Nutanix and other enterprise storageAlways create a new vDisk for the VM before running the tests. The main reason for that is the old disk might have some data from the previous tests or other workloads and it could have been down migrated to the HDD tier (in case of a hybrid cluster). The new disk will guarantee the correct data path of the I/O.Create a 50 GiB or 100 GiB disk. The disk then should be initialized and formatted with the default settings. The disk will get formatted and will get a letter assigned. Make sure that the new disk is selected in the CrystalDiskMark for the tests.NOTE: Nutanix scales performance per node and with the number of vDisks. The testing with CrystalDiskMark is based on a single vDisk on a single VM running on a single Node. While testing is important to understand a specific use case for a single vDisk, it should be clear that CrystalDiskMark does not replace a distributed performance test like Nutanix X-Ray to understand the full capability of a Nutanix cluster.The following settings are recommended as they provide a good mix of block sizes as well as parallel IO vs. single threaded IO:Figure 3: Recommended CrystalDiskMark settingsThe below table explains the different configuration details and differences to the default settings:

			Default tests:

			Recommended tests:

			Explanation:

			Sequential reads and writes of:

				Block size: 1 MB

				Queue depth: 8

				Threads: 1

			Sequential reads and writes of:

				Block size: 1 MB

				Queue depth: 8

				Threads: 8

			The test is very similar to the default sequential 1MB test, the only adjustment is the amount of threads has been changed from 1 to 8.			Such a test would be similar to what happens during a large file copy or during a backup job.

			Sequential reads and writes of:

				Block size: 1 MB

				Queue depth: 1

				Threads: 1

			Sequential reads and writes of:

				Block size: 64 KB

				Queue depth: 16

				Threads: 8

			As the default single-threaded 1MB test won't give much more insight with a queue depth of 1			Nutanix recommends replacing this test with a 64KB block size sequential test as it is more common for applications to use that block size.			The queue depth of 16 and 8 threads will ensure that the test generates enough load on the storage subsystem.

			Random reads and writes of:

				Block size: 4 KB

				Queue depth: 32

				Threads: 1

			Random reads and writes of:

				Block size: 8 KB

				Queue depth: 32

				Threads: 8

			The default 4K block size tests do not reflect the real-life usage of the application. However, 8K block size is very common and is used by most databases. 			The queue depth of 32 and 8 threads will ensure that the test generates enough load on the storage subsystem.

			Random reads and writes of:

				Block size: 4 KB

				Queue depth: 1

				Threads: 1

			Random reads and writes of:

				Block size: 8 KB

				Queue depth: 1

				Threads: 1

			The default 4K block size tests do not reflect the real-life usage of the application. However, 8K block size is very common and is used by most databases.			This test is left with a single thread and a queue depth of 1 just to show the difference between the multi-threaded test above and the single-threaded result.

Table1: Explanation of default vs. recommended CrystalDiskMark settingsA good Measure Time recommendation is 30 seconds. The default setting of 5 seconds won't generate enough load for that short time frame.The good Interval Time recommendation is 60 seconds. That is to make sure that there is a quiet time between write tests. Nutanix starts flushing the write cache after 30 seconds of inactivity, so a default interval of 5 seconds is not a good setting. The test wants to ensure that the write cache is empty before running the next test to achieve fair results. While this is Nutanix specific, an application would rarely read the same data after it has been written. These settings reflect a realistic ramp up time which can be similarly used on other Enterprise Storage Systems.A good recommendation for a testing file size: 16 GiBThat way the file is big enough to bypass read/write caches and test Storage performance rather than a cache.Example of test results with recommended settingsThe results will greatly vary depending on the underlying hardware configuration and software versions. The below results were achieved on the following setup:

VM details: Windows Server 2019 Datacenter Build 17763vCPUs: 8Memory: 8GbDisk size: 100Gb

Hardware details:3-node cluster NX-3061 (G3).2xSSDs 800Gb and 4 HDDs 1Tb per node

Nutanix AOS version: 5.19.1.5Hypervisor: AHV 20201105.12Storage container settings:Inline compression - onDeduplication - offErasure Coding (EC-x) - off

.Figure 4: CrystalDiskMark results. For throughput equivalents of the results, simply multiply the IOPS by the block size (i.e. 8k, 64k, 1M). For example, 1629 IOPS x 1M = 1629MB/sec.

Data randomness.

Data randomness is quite important because the storage engine of Nutanix will be compressing the file internally if compression is enabled on the storage container level. CrystalDiskMark has two options for laying the file:

	random (default)

	All 0x00 (0 Fill)

This can be configured in Settings - Test Data. It is generally recommended to use the random pattern to get realistic results because Nutanix will optimize the NULL byte files and testing on NULL byte files will not reflect the performance of real applications.Figure 5: CrystalDiskMark data pattern

The difference can be demonstrated by running a test in each of the modes and then compressing the file into a .zip archive using the built-in Windows compression tool. Even though the compression algorithm is different on the Nutanix storage level, it will still show the difference. So, if the test file is 1Gb in size, we see a 1Gb file in Windows Explorer:Figure 6: The test file sizeIf the file is created with the default random pattern, compressing the file into .zip will result in the same file size:Figure 7: The test file compressed into .zip size with a random data patternHowever, if the file is created with the All 0x00 pattern and then compressed into .zip, the archive will be taking ~1Mb of storage space instead of 1Gb, because the 0s will be mostly eliminated by the compression algorithm:Figure 8: The test file compressed into .zip size with All 0x00 data patternA similar compression happens on the storage side and without true data randomness, the test file will utilise a lot less of the storage space and will likely fit more into the cache, which will result in an unfair test.Unlike the diskspd tool, CrystalDiskMark creates a file with a random data pattern by default. There are no tweaks needed to achieve randomness.

File system software cachingUnlike the default settings of diskspd, CrystalDiksMark is disabling the software caching in its tests. This will show the same amount of IOPS/Throughput on the Nutanix system side as we see in the CrystalDiskMark test.Test example in CrystalDiskMark:Figure 9: CrystalDiskMark vs. Nutanix Storage PerformanceThe results from CrystalDiskMark vs. the Nutanix storage are very close. The reason for this difference is below example shows a single iteration at the moment of the screenshot capture and the CrystalDiskMark shows the average of the whole test.Reads:Figure 10: Nutanix Storage Performance read Writes:Figure 11: Nutanix Storage Performance write NOTE: CrystalDiskMark is an easy to use benchmarking tool which is generally a simplified version of diskspd with GUI. It lacks some control and settings compared to diskspd, so there is less tuning possible. If the settings are correctly set, the results will be realistic.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about NCC Health Check: pcvm_disk_usage_check. ### Assistant: The NCC health check pcvm_disk_usage_check verifies that the amount of disk space used in the Prism Central VM is within limits.. The NCC health check pcvm_disk_usage_check verifies that the amount of disk or system partition usage in the Prism Central (PC) VM is within limits.

This check has the following parts:

	Checking the individual data disk usage (added in NCC 3.5.1):

	If usage is more than 75% for several hours, a WARNING is returned to identify the disk.If usage is more than 90% for several hours, a FAIL is returned to identify the disk.

	Checking the overall data disk usage (added in NCC 3.10.1):

		If overall usage is more than 90% for several hours, a WARNING is returned.

	Checking the Prism Central VM system root partition usage (added in NCC 3.9.4). Returns only FAIL message if the partition usage exceeds 95%.

	Checking the Prism Central VM home partition usage (added in NCC 3.9.4):

	If the usage is more than 75%, a WARNING is returned.If the usage is more than 90%, a FAIL is returned.

	Checking the Prism Central VM CMSP partition usage (added in NCC 3.10.1):

	If usage is more than 75%, a WARNING is returned.If the usage is more than 90%, a FAIL is returned.
	Checking the Prism Central VM Upgrade disk partition usage (added in NCC 4.6.0):
	If the usage is more than 70%, a FAIL is returned.This check runs every 5 mins.If there are more than 5 failures (30 mins), a critical alert is raised.

Please note: If you are running LCM-2.6 or LCM-2.6.0.1, LCM log collection fills up /home directory please refer KB 14671 for workaround.

Running the NCC checkRun the NCC check as part of the complete NCC health checks.

nutanix@pcvm$ ncc health_checks run_all

Or run the pcvm_disk_usage_check check separately.

nutanix@pcvm$ ncc health_checks run_all

You can also run the checks from the Prism Web Console Health page: select Actions > Run Checks. Select All checks and click Run.

This check is scheduled to run every 5 minutes, by default.

This check will generate an alert after 5 consecutive failures across scheduled intervals.

Sample OutputsFor status: PASS

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM data disk, e.g. /dev/sdc1)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM overall MultiVDisk)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM data disk, e.g. /dev/sdc1)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on root partition, i.e. /)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM home partition, i.e. /home)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM home partition, i.e. /home)

nutanix@pcvm$ ncc health_checks run_all

For Status: WARN (on Prism Central VM CMSP partition, i.e. /dev/sde)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM CMSP partition, i.e. /dev/sde)

nutanix@pcvm$ ncc health_checks run_all

For Status: FAIL (on Prism Central VM upgrade disk partition, i.e. /home/nutanix/upgrade)

nutanix@pcvm$ ncc health_checks run_all

Check Disk Usage in PC VMFollowing is an example of how to check disk usage on a PC VM.

nutanix@pcvm$ ncc health_checks run_all

Partition for upgrade disk would show up similar to:

nutanix@pcvm$ ncc health_checks run_all

Output messaging

Check ID101059DescriptionCheck that disk space usage on the Prism Central VM is within limits.Causes of failureHigh disk usage in the Prism Central VM.ResolutionsRefer to KB 5228 for further details.ImpactPrism Central VM may run out of storage space to store data.Alert IDA101059Alert TitlePrism Central VM disk usage highAlert MessagePrism Central VM IP disk usage exceeds warning limit x % for disks: disks.			Prism Central VM IP overall MultiVDisk usage exceeds warning limit of  2321329924 KB			Prism Central VM IP disk usage exceeds critical limit x % for disks: disks.

Check ID200316DescriptionChecks if Prism Central system root partition usage is within the threshold to ensure uninterrupted operations.Causes of failureIncreased Prism Central VM system root partition usage due to excessive logging or incomplete maintenance operation.ResolutionsReduce Prism Central VM system root partition usage by removing any known temporary or unneeded files. Refer to KB 5228 for further details.ImpactIf the Prism Central VM system root partition is highly utilised, certain maintenance operations, such as upgrades, may be impacted. If the Prism Central VM system root partition is 100% utilized, services may stop and impact Prism Central cluster management functions.Alert IDA200316Alert TitlePrism Central VM System Root Partition Space Usage HighAlert MessageDisk space usage for root partition mount_path on entity-ip_address has exceeded threshold%.

Check ID200317DescriptionChecks if Prism Central home partition usage is within the threshold to ensure uninterrupted operations.Causes of failureIncreased Prism Central VM home partition usage due to excessive logging or incomplete maintenance operation.ResolutionsReduce Prism Central VM home partition usage by removing any known temporary or unneeded files. Refer to KB 5228 for further details.ImpactIf the Prism Central VM home partition is highly utilised, then certain maintenance operations, such as upgrades, may be impacted. If Prism Central VM home partition is 100% utilised, then services may stop and impact cluster storage availability.Alert IDA200317Alert TitlePrism Central VM home partition disk usage highAlert MessagePrism Central VM IP home partition disk usage exceeds warning limit x %.			Prism Central VM IP home partition disk usage exceeds critical limit x %.

Check ID200328DescriptionCheck that platform disk space usage on the Prism Central VM is within limits.Causes of failureHigh disk usage in the Prism Central VM.ResolutionsRefer to KB 5228 for further details.ImpactPrism Central VM may run out of storage space to store data.Alert IDA200328Alert TitlePrism Central VM platform disk space usage highAlert Smart TitlePrism Central VM svm_ip platform disk space usage highAlert MessagePlatform disk space usage in Prism Central VM svm_ip exceeds percentage_exceed% for disk(s): disk_paths.

Check ID200334DescriptionChecks if Prism Central upgrade disk usage is within the threshold to ensure uninterrupted upgrade operations.Causes of failureIncreased Prism Central VM upgrade disk usage due to the presence of multiple Prism Central Installer filesResolutionsReduce Prism Central VM upgrade partition usage by removing the Prism Central Installer files which are not needed. Refer to KB 5228 for further details.ImpactIf the Prism Central VM upgrade disk is highly utilised, then the Prism Central Upgrade would fail due to lack of space in the upgrade disk.Alert IDA200334Alert TitlePrism Central VM upgrade disk usageAlert MessagePrism Central VM <IP> upgrade disk usage exceeds critical limit x%

Scenarios that trigger pcvm_disk_usage check Warn/Fail on /home partition
Scenario 1The /home directory on long-running PC instances might reach close to its maximum limit as older Prism code is not cleaned up:

nutanix@pcvm$ ncc health_checks run_all

Nutanix Engineering is aware of the issue and is working on a fix in a future release to further clean up older Prism codes. For the workaround, refer to the corresponding solution section.

Removing older sysstats logsIf you have checked all the usual places but still need to clean up space, you can get the customer’s permission to remove older sysstats logs. Double-check that there are no open support cases with pending RCAs before proceeding, as this data may not yet have been collected.

nutanix@pcvm$ ncc health_checks run_all

Scenario 2Hyperkube logs (kublet logs) are not being cleaned up after enabling CMSP / microservices on Prism Central. If Cluster Maintenance Utilities (CMU) has been updated to version 2.0.3 using LCM, the included Scavenger version is missing the capability to clean up certain logs related to CMSP microservices. As a result of this issue, users may find that Prism Central services are not starting or that they cannot log in to PC UI. This issue will first fill up the /home/nutanix/data/sys-storage/NFS_.../ directory, after which the kubelet logs will start filling up the root partition in the /tmp folder.

nutanix@pcvm$ ncc health_checks run_all

We may find hyperkube.ntnx* logs getting generated and not rotated correctly when CMSP is enabled. Check if the directory /home/nutanix/data/sys-storage/NFS.../kubelet/ is excessively filled with these logs.

nutanix@pcvm$ ncc health_checks run_all

When working as intended, we should see less than 10G used in this kubelet folder, and the file count for kubelet* would be less than ~15
We may additionally see similar kubelet logs filling up space in /tmp, after space in the /home/nutanix/data/sys-storage/NFS... directory has been exhausted.

nutanix@pcvm$ ncc health_checks run_all

Scenario 3Starting PC.2022.6 - a dedicated 30 GB disk is created and mounted for PC upgrades. This Vdisk will be used for downloading and extracting upgrade binaries from consecutive upgrades. If the Prism Central VM upgrade disk is highly utilised, then the Prism Central Upgrade would fail due to lack of space in the upgrade disk.

nutanix@pcvm$ ncc health_checks run_all

Note: In case several services are enabled on Prism Central, such as msp, karbon, calm, flow, and objects, we would see /home usage to be high as each of the services will generate several logs and configuration files. Nutanix engineers are constantly working towards improving the /home usage. If none of the above scenarios matches and this affects Prism Central upgrade, engage Nutanix Support team to help manually cleanup /home by trimming the logs. Scenario 4Due to the log file rotation issue, Adonis logs directory usage is high. This log directory usage should not be over than 1G. For Scale-Out Prism Central deployments, check the file usage on each of the Prism Central VMs:

nutanix@pcvm$ ncc health_checks run_all

Scenario 5In certain cases, catalina.out may consume a large amount of space on the Prism Central VM.

SSH to the Prism Central and check if /home/nutanix/data/logs/catalina.out is consuming a huge amount of space:

nutanix@pcvm$ ncc health_checks run_all. If the check reports a WARN or FAIL status, disk usage is above the threshold and needs investigation. Generally, space utilization can be queried using df -h. The output below shows the mount points as follows:

/dev/sdb1 is root partition/dev/sdb3 is home partition/dev/sdc1 is data disk partition

nutanix@pcvm$ df -h

Data disk usage (/dev/sdXX) or overall multivdisk usage:

Verify the number of VMs supported for the particular Prism Central size is within the limit (consult the Prism Central Guide for your version from the Support Portal for the limits). Contact Nutanix Support. While opening a support case, attach the output of the following commands to the case.

nutanix@pcvm$ df -h

Prism Central VM home partition (/home):

Inspect the NCC output to determine which Prism Central VM has high usage, then perform the following:

Log in to the Prism Central VM.Use the cd command to change the location to the /home partition.List the contents of the directory by size using the command below:
	nutanix@pcvm$ df -h

	Examine the output for any large unused files that can be deleted.
	Run the du command below to list the usage of each file and sub-directory:
	nutanix@pcvm$ df -h

	Examine the output of large sub-directories. You can run the du command for each sub-directory in question to further identify large unused files that can be deleted.
	Below are some common sub-directories of /home where large unused files are likely to exist:
	/home/nutanix/software_downloads/ - delete any old versions other than the versions you are currently upgrading./home/nutanix/software_uncompressed/ - delete any old versions other than the versions you are currently upgrading./home/nutanix/data/cores - delete old stack traces that are no longer needed./home/nutanix/data/log_collector/ - delete old NCC Logs with NCC-logs-2018-07-20-11111111111111-1032057545.tar format./home/nutanix/foundation/isos/ - old ISOs./home/nutanix/foundation/tmp/ - temporary files that can be deleted.

If the above steps do not resolve the issue or if the issue matches one of the scenarios presented earlier in this article, follow the solution steps outlined below.
Prism Central VM root system partition (/):Consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

Prism Central VM CMSP partition (/dev/sde):Consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

Scenario 1As a workaround, remove the directories that do not have registered clusters with the corresponding version.

Find the AOS versions of all the registered PEs. You can do so from Prism Central > Hardware > Clusters > AOS version column.List the PE apache console directories at /home/apache/www/console/

nutanix@pcvm$ df -h

If any PE apache console directories that do not correspond to registered PE versions are present, they should be safe to clean up.
If you require further assistance with the cleanup, consider engaging Nutanix Support. Gather the output of the commands below and attach it to the support case:

nutanix@pcvm$ df -h

Removing older sysstats logsIf you have checked in all the usual places but still need to clean up space, you can get the customer’s permission to remove older sysstats logs. Please double-check that there are no open support cases with pending RCAs before proceeding, as this data may not yet have been collected.

nutanix@pcvm$ df -h

You can use the following command to remove gzipped sysstats logs older than a certain date. In the example below, you will remove sysstats logs from all PCVMs that are older than 3 days (or 4320 minutes).

nutanix@pcvm$ df -h

Scenario 2If your Prism Central instance matches this scenario, refer to KB-12707 Scenario #2 and open a case with Nutanix Support for assistance with in recovering from the issue.Scenario 3The increase in Prism Central VM, upgrade disk usage, is due to multiple Prism Central Installer files. Reduce Prism Central VM upgrade partition usage by removing the Prism Central Installer files that are not needed.This partition /home/nutanix/upgrade is designed to hold upgrade-related files. So, any other file within this directory can be deleted.The partition usage as listed in df -h:

nutanix@pcvm$ df -h
To check the contents of the disk:

nutanix@pcvm$ df -h

To remove the unwanted files, use the below command:

nutanix@pcvm$ df -h
Scenario 4Nutanix is aware of the issue. The fix for this issue will be made available in a future PC release. For a workaround, engage Nutanix Support.

Scenario 5

If you see catalina.out log file is consuming a lot of space, use the following command to restart the prism service on the PCVM.

nutanix@pcvm$ df -h. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about MOVE: Nutanix Move Basic Troubleshooting Guide. ### Assistant: This article describes basic troubleshooting for Nutanix Move (version 3.x & 4.x).. With the release of Nutanix Move 3.0, the Move services are dockerised and all the Move agents now run as Docker Containers. This is a major milestone that allows add/update features without much service disruption as well as provides flexibility to run Move anywhere.

If you are running an older version of Move, upgrade to Move 4.x and later.

To upgrade, find the latest version and procedure below:

Latest Move bundle: here

Upgrade Procedure: here

Note: If you face issues in booting the VM after migration to Nutanix AHV, please collect logs on AHV cluster using Logbay.. Overview

Move File locationsWhich user do we use to SSH into the Move VMLog files locations for troubleshootingBasic CommandsHow to generate Move support bundle from cliHow to configure static IP on Move VMFirewall Ports requirements for MoveTesting Connectivity between Move and ESXi/vCenter/Hyper-VCommon user permissions required on the Windows OS if using the domain administrator or local administrator accountsMove is stuck at 0% while "Seeding Data"Move VM migration failure due to missing snapshotWhat happens if Move upgrade is stuckAre there any logs created on the VM by MoveIf there is a backup schedule, will it affect Move

Move File locationsMove will use the below host directories. All of them are created under /opt/xtract-vm/.

Files LocationDescription/opt/xtract-vm/certsPath to the certificate key-pairs used by Move/opt/xtract-vm/keyPath to the SSH keys generated and used by Move/opt/xtract-vm/kvstoreKVStore directories used by srcagent and diskwriter/opt/xtract-vm/logsPath to Move service logs

Which user do we use to SSH into the Move VMWhen logging in to the Move VM using SSH, use:

Username : admin
Note: The admin user on Move does not really have all the major rights, so best way is to change the user to root. To do that, on the Move VM, run:

Username : admin

Enter the password for admin. You will be able to change the mode to root user and have control over the Move VM.

Log files locations for troubleshootingBelow are the log files which will be important for troubleshooting.

Logs File LocationDescription of the Log File/opt/xtract-vm/logs/diskreader.logUses NFC mechanism to take a copy of the disk and ship it to the AHV side using "pipe"./opt/xtract-vm/logs/diskwriter.logThis is the receiver of the copy disk through pipe and it writes it to the container mounted on the Move VM./opt/xtract-vm/logs/mgmtserver.logOrchestrator service, exposes REST APIs to source, target side. If this service is working fine, then the UI will load perfectly.opt/xtract-vm/logs/srcagent.logDealing with source side, it prepares the migration by enabling CBT (Changed Block Tracking), shutting down the VM and shipping the last snapshot before the VM finally boots up on AHV side./opt/xtract-vm/logs/tgtagent.logDealing more on the AHV side. Collects cluster information and mounts the needed container to Move during migration and power on the VM on the AHV side.

Basic CommandsTo check the status of the services, run the command svcchk as below. Note that the last column will give you the name of the service (for example, bin_srcagent_1, bin_tgtagent_1, bin_diskreader_1).

Username : admin

Note: If restarting any service/Docker with an active Migration Plan, the Migration Plan will turn into failed state and the target vdisk can get corrupted. In this case, a manual cleanup on AHV and a new Migration Plan are needed.To restart any single service, run the command "docker restart <service name>" as shown below. You can get the individual service names from the svcchk command.

Username : admin
To restart all the services, run the command svcrestart. It will ask you to confirm. Enter "y" (yes) to continue:

Username : admin

You can also use svcstop and svcstart commands to stop and start all the container services, respectively.

How to generate Move support bundle from cliTo generate the Move support bundle from CLI, run the command "root@move on ~ $ /opt/xtract-vm/bin/support-bundle [--dump-path <directory>]" as shown below. This example dumps the support bundle under the location /opt/xtract-vm/supportdumps. You can select your own directory. If you leave it blank, it will generate the bundle in the /root directory.

Username : admin

How to configure static IP on Move VMIn case you deleted the Move NIC or want to give a static IP again to the Move VM, follow the procedure of Assigning a Static IP Address to Nutanix Move.

Firewall Ports requirements for MoveVerify the ports between Move VM and ESX and Target AHV is open. If the environment has different subnets, refer to Port Requirements of Move.

Testing connectivity between Move and ESXi/vCenter/Hyper-V

Move must be able to communicate with vCenter Server on port 443, ESXi hosts on 902 and 443, and AHV on 9440. You can run the following commands from Move VM to verify the connectivity:

Username : admin
Testing connectivity on port 902 between Move and ESXi hostThe port connectivity for port 902 could not be tested by nc -zv <ESXi Host IP> 902.Reason: Starting from move 3.0 onwards, iptables is handling the routing for the docker containers residing in the move appliance hence, one needs to run the test through a docker container, please find the steps below:

List all running docker images :
	Username : admin
	Example:

	Username : admin
	Test connectivity from within the docker container named srcagent using the following command :
	Username : admin
	Example:

	Username : admin

Common user permissions required on the Windows OS if using the domain administrator or local administrator accounts Any user account on a Windows VM must possess three properties to be used as a login account by the Move:
The user must have "Logon as Batch Job" enabled.UAC must be disabled (for local administrator, it is usually disabled).The users must have "Restore files and directories" rights.

Move is stuck at 0%, while "Seeding Data"Possible Causes

The customer uses third-party solutions that rely on snapshots.Third-party snapshots may cause stale snapshots to reside in the VM folder on the VMware datastore.Old snapshots that have been forgotten.
Possible Solutions

Leverage the Consolidate option in VMware.
	This can be done through the Web or vSphere client by right-clicking the VM and selecting "Consolidate".This may fail to work, and the snapshot files may still appear in the VM folder on the datastore.The Consolidate option may be greyed out and not available.
	For Hyper-V, delete checkpoints Create a new snapshot, and then delete it.Storage vMotion the VM from one datastore to another:
	The VM must move from one VMware datastore to a different VMware datastore to ensure consolidation. Verify the absence of stale snapshots in the new datastore location.If Migrate option is greyed out:
		Try powering off the VM to see if the option becomes available. There are many reasons a Storage vMotion may not be possible while the VM is running.Check for registration issues. Follow VMware KB 2044369. Remove the VM from the inventory and re-add it. Ensure you know which datastore the .vmx file lives in before removing it. In some cases, you may need to power the VM off to remove it from inventory.If none of these options works and you need to leverage this KB, it should only be done with the assistance of VMware, as it involves modifying the DB.

Move VM migration failure due to missing snapshotMove migration may fail due to missing snapshots. Consolidate snapshots from the VMware Snapshot Manager to resolve.

A common signature would be found in Move mgmtserver.log:

Username : admin

Check the Source VM Snapshot Manager, and you will be unable to find the snapshot the logs refer to.

Consolidate all snapshots of the source VM from VMware Snapshot Manager. Once finished, recreate a new job in the Move for migration.

What happens if Move upgrade is stuck?If the Move upgrade is stuck, check the updater.log on the Move VM. If you cannot figure out the exact reason, collect the logs, and share them with Nutanix Support.

Proceed with verification of the status for all the services (the expected state is UP and not crashing) by running the command svcchk.

To roll back the Move appliance to a previous version, run the script:

Username : admin

Are there any logs created on the VM by Move? Yes, logs are created on the VMs when Move triggers VirtIO installation and IP address retention scripts.On Windows, the logs can be found in:

Username : admin

On Linux, they can be found in:

Username : admin

If there is a backup schedule, will it affect Move?Yes, it will be difficult to take the number of changes into account on the disk when there is a backup job in progress. So, it would be best to perform the migration outside of backup hours or disable the backup until the migration is completed.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.


### Human: Can you tell about Move: Update of a storage container may fail if this container has a container filesystem whitelist configured. ### Assistant: Update of a storage container may fail after 5 minutes with "stream timeout" or "Http Request to endpoint 127.0.0.1:9080 failed with error.Response status:1" if this container has a container filesystem whitelist configured.. Update of a storage container that has container level filesystem (NFS) whitelist may fail with one of the following messages appearing after 5 minutes:

Oops - Server error

Or:

Oops - Server error

Errors observed

In Prism Element, when updating the container, an error “stream timeout” or “Http Request to endpoint 127.0.0.1:9080 failed with error. Response status:1” appears.

	Or:

	nCLI command takes several minutes to execute, followed by the error:
	Oops - Server error
	Nutanix Move appliance may fail to configure AOS filesystem whitelist with the following error:
	Oops - Server error
	The following errors can also be found in ~/data/logs/prism_gateway.log on the Prism leader:
	To identify the Prism leader CVM, see KB 1841.

	Oops - Server error

Workflow of modifying "Filesystem Whitelists" of a container

Prism Gateway makes an API call to genesis to update the whitelists of the container when the container "Filesystem Whitelists" is configured to add Move IP aa.bb.cc.dd. The request will be logged to prism_gateway.log. The following example is to add Move IP aa.bb.cc.dd to "Filesystem Whitelists" of a container.

Oops - Server error

Genesis internally makes a call to modify Firewall. Events are logged to genesis.out

Oops - Server error
 . Nutanix Engineering is aware of the issue and is working on a fix in a future release.

Workaround 1To work around the issue, you need to delete a per-container filesystem whitelist.

Note: If you have active external clients that still need access to the container over NFS/SMB (e.g. Move appliance), make sure that you update Global Filesystem with all IP addresses from the container filesystem whitelist before proceeding with the workaround. When a container filesystem whitelist is removed, a global AOS filesystem whitelist will be used and some clients may lose access to the container if they are not whitelisted on the cluster level. Refer to the documentation for more information on Filesystem Whitelists.

In Prism Element, select Storage from the pull-down main menu (upper left of screen) and then select the Table and Storage Container tabs.Select the storage container in question and then click the Update button.Find Filesystem Whitelist in the Advanced Settings of the container.Empty Filesystem Whitelist text box and click the Save button.Once the task completes, you can retry the operation that was failing before.

In case the steps described in this article do not resolve the issue, consider engaging Nutanix Support for further investigation.. ### Human: Now explain this as a Nutanix support agent summarizing the answer step by step. In case, you don't know the answer say - I don't know.