HPE CloudPhysics Virtual Appliance Network Troubleshooting Guide

The HPE CloudPhysics Virtual appliance is an on-premise data collection tool that requires both local and internet access to operate. This short list of troubleshooting scenarios will assist you in identifying where your network issues may be originating and how to correct them.

First, some requirements/limitations of the Virtual Appliance.

  • All network setting for the virtual appliance are defined in the VMware vCenter interface during the OVA / OVF deployment.
  • The HPE CloudPhysics virtual appliances uses IPv4 to connect to the network
  • The HPE CloudPhysics virtual appliance requires DNS resolution of internet names
  • The HPE CloudPhysics virtual appliance must be able to reach the VMware vCenter
  • The HPE CloudPhysics virtual appliance must be able to communicate with the public internet either directly or through a proxy configured in the appliance
  • The HPE CloudPhysics virtual appliance has a single network adapter. Note: a special CloudPhysics appliance with dual network adapters can be request for a VMware vCenter on restricted networks to allow for dual network connectivity (private and public).
  • The HPE CloudPhysics virtual appliance does not support VLAN tagging
  • The HPE CloudPhysics virtual appliance does not support IPv6

Troubleshooting Steps

How the Applaince Works

The HPE CloudPhysics virtual appliance is a very simple device. The appliance uses a REST API (HTTPS port 443) connection to an internal VMware vCenter and an outbound connection (HTTPS Port 443) to HPE CloudPhysics in the cloud (https://entanglement.cloudphysics.com:443). The appliance will receive its IP address either dynamically from DHCP or a static address that is defined in VMware vCenter. The appliance is required to be placed in a network segment with a DNS to resolve the names of the vCenter and the public destination.


After the appliance logs in for the first time and you are presented with a network summary and optional proxy setting, the appliance will attempt to PING the HPE CloudPhysics of entanglement.cloudphysics.com:443. This name must be resolvable by the local DNS. If the appliance is unable to reach this destination, you will receive a network error message indicating the destination is unreachable.

Things you cannot do with the HPE CloudPhysics virtual appliance:

  • Log into the appliance console

  • Specify your own certificates

  • Modify the IP address from within the appliance. This is done in VMware vCenter.

  • Connect one appliance to multiple VMware vCenters. The appliance is designed for one appliance per VMware vCenter.

  • Clone or create a template of the appliance. The appliance should be deployed from VMware vCenter from the original OVA/OVF to ensure it gets the correct network setup.

  • Use the appliance in a network or organization without Internet access. A connection to HPE CloudPhysics (https://entanglement.cloudphysics.com:443) is required.

  • Connect to a VMware ESXi Host in place of a VMware vCenter. The appliance only supports the API for VMware vCenter v5.x and above.

The Basics

If you cannot get the applaince to connect to the network, the first steps are to ensure:

  1. the correct IP Address was provided and the address was not a duplicate

  2. check network addresses for typos

  3. the network segment has internet access

  4. the DNS can resolve the target network name

  5. no firewall rules are restricting access

  6. a proxy server is defined if required

  7. a proxy server policy is created if a poxy is required.

  8. the most recent version of the appliance was deployed from an OVA/OVF and not from a template or clone of another appliance
  9. verify certificate authority and certificate is valid
Review all of the provided details in the vCenter appliance import interface were correct. Typos are common.

Review all values in the CloudPhysics Appliance console screen to confirm they were correct.

Note your appliance version at the bottom of the Blue Text and ensure you’re using the latest appliance available from https://download.cloudphysics.com. You can open the .OVF files in any text editor to view the version number. To view the appliance version, search for “Version” in the OVF file. OVA files can be renamed to .ZIP and extracted to view the configuration files.

Example:
<Version>v2.6.4</Version>
<FullVersion> v2.6.4 </FullVersion>

First, some VMware Virtual Appliance fundamentals. The VMware vCenter passes the network values to the virtual appliance as assigned during the OVA/OVF import. The CloudPhysics virtual appliance gets its values from vCenter and they are not defined within the appliance. If the values are incorrect or missing, then there is likely an issue with the VMware vCenter version/build you are using preventing these values from being passed to the appliance . This was a known issue for vCenter 6.5 and 6.7 for some builds. If everything you try fails with VMware vCenter, try to deploy using the ESXi Host web interface. Browse to your desired ESXi host using a browser, login in with a vCenter admin credential, and attempt to deploy from here.

I see a valid IP Address, DNS, and Gateway, but fail to connect to CloudPhysics

This is the most common issue we encounter. The issue typically lies in a Firewall Rule, Proxy policy, or 3rd party network vSwitch or network manager (ie: Palo Alto Network). If the console shows a valid IP Address, DNS, Gateway, and netmask, then the appliance is configured correcting and working as designed and the issue lies outside of the virtual appliance in your network. Communicate with your network security team to learn if unique firewall rules, DNS restrictions, or restricted ports exist in your environment. The CloudPhysics appliance requires Internet Access to send data to the cloud for processing. Any policy blocking this connection will prevent the appliance connecting to
https://entanglement.cloudphysics.com:443

If you have valid values for all the network settings, then the issue is in your network. The CloudPhysics virtual appliance is a very simply device that gets all of its network settings from vCenter. If the values are correct, then the appliance is configured correctly. This tells us the issue lies someplace in the network layer between the VMware Distributed Switch/vDS port and the network firewall/proxy server.

Speak with a network administrator to see if your environment restricts access to the internet. It is common for networks admins to:

  • Have firewall policies in place that limit workloads in the management network from connecting to the public internet
  • Have proxy servers policies preventing access
  • Actively manage port connections to a virtual switch
  • Restrict network segments from public internet access
  • Prevent name resolution for external networks
  • Require security tokens or access to HTTPS keys when accessing the internet

I don’t have a Proxy Server but I getting an error that I cannot connect to CloudPhysics

A proxy server is not required and entirely optional. Leave these setting blank if you do not have a network proxy requirement. If you cannot get past this screen, and you have a valid IP Address, DNS, Network Gateway, and netmask, then the issue is within your local network, firewall, or virtual switch port configurations blocking the appliance from reaching the internet.

When in doubt, start with a clean OVA/OVF install

First, when all else fails, simply delete the virtual appliance and redeploy a current appliance from a new download from the web (https://download.cloudphysics.com). This will ensure that all attributes in the OVA / OVF deployment process were specified correctly. The HPE CloudPhysics appliance is stateless and be deleted and redeployed at any time you feel you need to make a change to the appliance definition or are uncertain about an appliance configuration.

External Connectivity Firewall Rules and Proxy Settings

Some networks restrict outbound access to secure subnets to a limited set of destinations and protocols or through a proxy server. If this is a requirement for your organization, you will need to enable a rule to allow the HPE CloudPhysics virtual appliance to communicate outbound from the network segment to the following destination:

https://entanglement.cloudphysics.com:443

Your network administrator may be required to validate and complete these rules.

Do not create network rules based on IP Address as this address is not guaranteed to be static due to being a cloud-based Software as a Service (SaaS) platform and may change in the future. If you must configure an IP address-based rule, ping the network destination to resolve the current IP Address. Be aware that you may receive an email notice in the future that the appliance has disconnected and will need to be reconnected to the network. If you receive such a notice, validate the target IP address of the cloud endpoint, and change your network rules accordingly.

VMware vCenter 6.5 and 6.7 HTML Client

Note, VMware vCenter 6.5 and 6.7 had some issues defining network attributed during import. OVF Support was not yet integrated in HTML 5 client and the Flash based client is no longer supported. If you find that you are unable to view, change, or specify network attributes in VMware vCenter, you can use the alternate method of deploying the VM from the ESXi server web interface. You can deploy directly from ESXi by browsing to the ESXi Server IP Address and logging in with the local admin account to deploy appliances via the local web interface. This will resolve some of the know VMware vCenter issues. If you are having issues deploying the appliance, deploy OVF files using the command line. For more information, see the OVF Tool Documentation. https://www.vmware.com/support/developer/ovf/

Defining Static or DHCP Addresses in vCenter

When deploying the OVA / OVF, you will have the option to specify a Static IP Address or take advantage of DHCP for a dynamic address for the appliance. Note that vCenter will require all field to be either left blank for DHCP or all fields to be provided for Static IP Address. You cannot provide some data and leave other part blank.

For a Static IP Address, you will need to specify the following:

Default Gateway, DNS, Management IP Address (VM network IP), Management Network (subnet mask).

Solution: If any if these values are specified in the OVA/OVF import, VMware vCenter will default to a Static IP Address and not use any DHCP values. Always define all values if you intend to use a static IP Address or leave all values blank if you intend to use DHCP.

Appliance Startup Timeouts and No IP Address

If you open the console immediately after starting the virtual appliance, you will see the OS system check and startup process. After a few seconds you will see:

Welcome to the Observer.
Please wait while the configuration UI loads.
Observer login:

If the observer remains on this screen for more than 20 seconds, then the issue is likely that the appliance is not getting an IP Address from vCenter. If the appliance is waiting for more than 15 to 20 seconds at the login prompt, this indicates that the network stack is waiting for a network configuration from VMware vCenter. If an address is not assigned, the startup process will time-out between 30 and 60 seconds and continue to a Terms of Services screen without assigning the static or DHCP address. Instead, the OS will use a self-assigned address that will not be reachable on your network.

If this situation occurs, confirm the IP address in the VMware vCenter console is displayed and correct.

Solution: The most common cause of this problem is when a network requires a VM to be registered with DHCP or other network management tools by MAC address to assign an IP Address. If this situation occurs, reach out to the network management team to determine if there are sufficient IP Addresses in the subnet or that an IP Address has accurately been assigned to the VM.

Validate VM has an IP Address

The most basic validation will be to view the VM Summary in vCenter and confirm the presence of a DNS Name and IP Address. If DHCP or a Static IP address was assigned successfully, it will be reflected in VMware vCenter. If no address is found on the VM Summary, the appliance is either deployed in a network segment without DHCP or our network IP management system my not have a valid configuration applied to the appliance MAC address. The IP address is assigned by VMware vCenter, so if no address has associated with the VM, the error is occurring between VMware vCenter and your network definition or management resources.

DNS Value is incorrect or not specified

If you experience an issue where the DNS is either BLANK or reports back a value of NO_RESOLVECTL_RESULTS, then there was an issue passing the DNS value form the VMware vCenter to the virtual appliance. Redeploy the appliance and check your network configuration to ensure no typos were made. If you have two DNS’ specified, ensure they are separated by a comma. This issue is created by VMware vCenter not passing a valid value to the appliance.

Ping the VM from a VM in the same network segment

This is a direct follow-up to the last step of ensure the IP address is visible. Ensure you can ping the IP address of the virtual appliance from another VM in the same network segment or from someplace else on the network. Ideally, this should originate in the same network segment as the appliance or vCenter to ensure connectivity.

If you are unable to ping the appliance, ensure the network segment is not restricted or other network rules are not blocking the packets.

Is the VMware vCenter PSC is reachable by Name One of the first steps in the appliance setup is to specify the primary VMware vCenter Platform Service Controller (PSC). The first connection attempt should be with a Fully Qualified Domain Name (FQDN). If the VM can connect to the PSC, we now know that the appliance has both a valid IP Address and valid DNS. If the name cannot be resolved, it is likely that the DNS is not configured correctly, or the network connection is not present. If we can communicate with the VMware vCenter by FQDN, we have confirmed that the network address is valid and the VM is connected to the network. Any further restrictions would be related to network access ACLS/Rules or firewall/proxy filters. Continue to network proxy configuration and firewall rules.

Is the VMware vCenter PSC is reachable by IP Address

If we are unable to connect to the PSC by a FQDN, we next want to try and connect to the PSC by the IP Address. If the IP Address can reach the PSC, then we have an invalid DNS that will restrict access to the HPE CloudPhysics cloud API interface. If the IP address fails to resolve as well, but we can ping the IP Address, then we have a network segmentation issue. If we can communicate with the VMware vCenter by IP address, we have confirmed that the network address is valid and the VM is connected to the network. Any further restrictions would be related to network access ACLS/Rules or firewall/proxy filters.

I can connect to VMware vCenter, but the outbound proxy server setting is not working

If you can connect to the PSC, but you are getting a notice that you cannot connect to CloudPhysics, you are now at the point where the Proxy configuration may be incorrect or there is a network rule restricting outbound access. First, validate that you are providing a Proxy server address or name and port in the proper format. Example:

Proxy.mycompany.local:8080

Also validate that a proper user credentials is used if required. This credential may be in the format of username@domain.com or domain\username in the input field. Try both formats.

If Proxy is not required, but I cannot connect to CloudPhysics

At this point, we have confirmed we can connect to the VMware vCenter, we have a proper IP Address, and we have a valid DNS setting. If the communications is blocked at this point, the block is usually occurring at the network gateway, firewall, or proxy server. Check with the network administrator that the IP address of the virtual appliance is allowed outbound communications on port 443 to https://entanglement.cloudphysics.com.

Verify Certificate and Certificate Authority is Valid.

Verify that no security services between the virtual appliance and the HPE CloudPhysics cloud upload destination have not tampered with certificates like ZScaler and other security tools.

We have seen instances where the returned SSL Certificate is replaced by the certificate of the security appliance. This has happened with older security devices where the Certificate Authority Let's Encrypt is not added to the top-level certificate authorities.

To verify this, deploy a Microsoft Windows or Linux guest OS in the same network. Open a command prompt in the Guest OS and type the following command:

curl -vvI https://entanglement.cloudphysics.com:443

You should get back a result like the text below, but if you do not, and you see an Issuer other than Let's Encrypt (issuer: C=US; O=Let's Encrypt; CN=R3) or Google Cloud, then your appliance will not connect as it cannot trust the internet connection to CloudPhysics.

curl -vvi https://entanglement.cloudphysics.com:443

* Trying 34.98.108.83:443...
* Connected to entanglement.cloudphysics.com (34.98.108.83) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* CAfile: /etc/ssl/cert.pem
* CApath: none
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Client hello (1):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Server hello (2):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Unknown (8):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Certificate (11):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, CERT verify (15):
* [CONN-0-0][CF-SSL] (304) (IN), TLS handshake, Finished (20):
* [CONN-0-0][CF-SSL] (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=entanglement.cloudphysics.com
* start date: May 25 23:14:01 2023 GMT
* expire date: Aug 24 00:05:53 2023 GMT
* subjectAltName: host "entanglement.cloudphysics.com" matched cert's "entanglement.cloudphysics.com"
* issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
* SSL certificate verify ok.

* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: entanglement.cloudphysics.com]
* h2h3 [user-agent: curl/7.87.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7fbdd680bc00)
> GET / HTTP/2
> Host: entanglement.cloudphysics.com
> user-agent: curl/7.87.0
> accept: */*

Another way to test is to open a web browser and confirm the certificate of the URL.
1. Open the following URL: https://entanglement.cloudphysics.com:443
2. Check the Certificate or lock icon in the browser URL line.
3. Verify the Certificate was issued by Let's Encrypt (R3)

If the certificate cannot be validated or trusted, the appliance will not be allowed to connect.

If All Else Fails:

For additional support, please contact cloudphysicssupport@hpe.com. Please provide us your organization name, some background to the problem, and the steps you have tried to resolve the issue. We will set up a time to work with you to assist evaluating and resolving the problem.

Recap of Troubleshooting Steps

  • Defined IP Address as Static or DHCP. All or nothing specific in OVA/OVF import.
  • When the appliance powers up, verify an address is assigned in vCenter Summary view.
  • Verify how long the appliance takes to get to the Terms of Service screen.
  • Does the appliance have an IP Address assigned in the vCenter VM Details view? If yes, the VM likely is on the network.
  • Ping the IP Address of the appliance. If it does not reply, is it segmented, firewalled, or may have a port not connected to a vSwitch.
  • When you start the Appliance config, can you resolve the vCenter PSC name in the first select a vCenter screen? If not via FQDN, try IP address. If IP Address, then DNS is incorrect.
  • If you can connect to vCenter, but get hung up connecting to CloudPhysics, the issue typically is DNS, Firewall, or Proxy rules.
  • Ensure policy or rules for access to https://entanglement.cloudphysics.com:443
  • When all else fails, delete the appliance and redeploy. It is stateless, so no data will be lost. When deploying, ensure all network fields are blank is using DHCP or ALL FIELDS are provided if using a Static IP Address.
  • When you have completed the setup and all the settings are correct, you will be presented with the following CloudPhysics Observer Configuration Status Console Screen:
  • Validate the Certificate Authority ("Let's Encrypt") is authorized and the SSL certifiate is valid. The certificate should not be replaced by a "Man-in-the-middle" security resource/gateway.

 

HPE Technical Support: cloudphysicssupport@hpe.com
Use this email address for technical issues with HPE CloudPhysics Observer, Account issues, and technical issues with the portal.