A step-by-step guide on how to collect configuration, performance, and billing details from AWS for assessment and optimization.

CloudPhysics now offers Amazon Web Services Analytics and Bill Analysis.  To enable the integration of CloudPhysics and your account, follow these simple steps and connect your reporting and billing details to CloudPhysics for deep service and cost analysis.

Summary of steps for setup

The AWS Account Setup summary is as follows:

  1. Create CloudPhysics Read Only Policy in AWS IAM
  2. Create CloudPhysics AWS Role
    • You need the CloudPhysics AWS Account ID:  863002038009
    • You need an External App ID located at on your AWS Connection Status Page
  3. AWS Cost and Usage Report Bill Collection (Optional)
  4. Provide Bill details to CloudPhysics

Follow these steps on how to grant CloudPhysics access to Amazon Web Services for
EC2, CloudWatch, and Billing.

If you have completed all the steps and are experiencing trouble or not seeing your AWS EC2 Data or Cost and Usage Report (CUR) data load into the CloudPhysics platform, please visit the AWS Billing Setup Troubleshooting Guide to assist in your problem resolution. The most common account setup and bill collection configuration issues are addressed within this guide.


VIDEO: How to setup an AWS Account to connect to CloudPhysics:

Create CloudPhysics Read Only Policy in AWS IAM

  1. In AWS IAM Console, login and create CloudPhysics Read Only Policy.
  2. If you’ve already created a policy, search for it on this page and select it. Otherwise, complete the following steps to create a new one.
  3. Select the Policies option from the sidebar navigation.
  1. Click Create Policy.
  2. Select the JSON tab.
  1. Copy the restricted CloudPhysics JSON policy provided below and paste this into your JSON text box in AWS IAM Console.
Full JSON - Minimal Credentials

Download the current JSON text here or copy the full text below.

{ "Version": "2012-10-17", "Statement": [ { "Sid": "VisualEditor0", "Effect": "Allow", "Action": [ "autoscaling:DescribeAccountLimits", "autoscaling:DescribeAdjustmentTypes", "autoscaling:DescribeAutoScalingGroups", "autoscaling:DescribeAutoScalingInstances", "autoscaling:DescribeAutoScalingNotificationTypes", "autoscaling:DescribeLaunchConfigurations", "autoscaling:DescribeLifecycleHooks", "autoscaling:DescribeLifecycleHookTypes", "autoscaling:DescribeLoadBalancers", "autoscaling:DescribeLoadBalancerTargetGroups", "autoscaling:DescribeMetricCollectionTypes", "autoscaling:DescribeNotificationConfigurations", "autoscaling:DescribePolicies", "autoscaling:DescribeScalingActivities", "autoscaling:DescribeScalingProcessTypes", "autoscaling:DescribeScheduledActions", "autoscaling:DescribeTags", "autoscaling:DescribeTerminationPolicyTypes", "aws-portal:ViewAccount", "aws-portal:ViewBilling", "aws-portal:ViewPaymentMethods", "aws-portal:ViewUsage", "cloudwatch:DescribeAlarmHistory", "cloudwatch:DescribeAlarms", "cloudwatch:DescribeAlarmsForMetric", "cloudwatch:GetDashboard", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListDashboards", "cloudwatch:ListMetrics", "ec2:DescribeAccountAttributes", "ec2:DescribeAddresses", "ec2:DescribeAvailabilityZones", "ec2:DescribeBundleTasks", "ec2:DescribeClassicLinkInstances", "ec2:DescribeConversionTasks", "ec2:DescribeCustomerGateways", "ec2:DescribeDhcpOptions", "ec2:DescribeEgressOnlyInternetGateways", "ec2:DescribeElasticGpus", "ec2:DescribeExportTasks", "ec2:DescribeFlowLogs", "ec2:DescribeFpgaImageAttribute", "ec2:DescribeFpgaImages", "ec2:DescribeHostReservationOfferings", "ec2:DescribeHostReservations", "ec2:DescribeHosts", "ec2:DescribeIamInstanceProfileAssociations", "ec2:DescribeIdentityIdFormat", "ec2:DescribeIdFormat", "ec2:DescribeImageAttribute", "ec2:DescribeImages", "ec2:DescribeImportImageTasks", "ec2:DescribeImportSnapshotTasks", "ec2:DescribeInstanceAttribute", "ec2:DescribeInstanceCreditSpecifications", "ec2:DescribeInstances", "ec2:DescribeInstanceStatus", "ec2:DescribeInternetGateways", "ec2:DescribeKeyPairs", "ec2:DescribeLaunchTemplates", "ec2:DescribeLaunchTemplateVersions", "ec2:DescribeMovingAddresses", "ec2:DescribeNatGateways", "ec2:DescribeNetworkAcls", "ec2:DescribeNetworkInterfaceAttribute", "ec2:DescribeNetworkInterfacePermissions", "ec2:DescribeNetworkInterfaces", "ec2:DescribePlacementGroups", "ec2:DescribePrefixLists", "ec2:DescribeRegions", "ec2:DescribeReservedInstances", "ec2:DescribeReservedInstancesListings", "ec2:DescribeReservedInstancesModifications", "ec2:DescribeReservedInstancesOfferings", "ec2:DescribeRouteTables", "ec2:DescribeScheduledInstanceAvailability", "ec2:DescribeScheduledInstances", "ec2:DescribeSecurityGroupReferences", "ec2:DescribeSecurityGroups", "ec2:DescribeSnapshotAttribute", "ec2:DescribeSnapshots", "ec2:DescribeSpotDatafeedSubscription", "ec2:DescribeSpotFleetInstances", "ec2:DescribeSpotFleetRequestHistory", "ec2:DescribeSpotFleetRequests", "ec2:DescribeSpotInstanceRequests", "ec2:DescribeSpotPriceHistory", "ec2:DescribeStaleSecurityGroups", "ec2:DescribeSubnets", "ec2:DescribeTags", "ec2:DescribeVolumeAttribute", "ec2:DescribeVolumes", "ec2:DescribeVolumesModifications", "ec2:DescribeVolumeStatus", "ec2:DescribeVpcAttribute", "ec2:DescribeVpcClassicLink", "ec2:DescribeVpcClassicLinkDnsSupport", "ec2:DescribeVpcEndpointConnectionNotifications", "ec2:DescribeVpcEndpointConnections", "ec2:DescribeVpcEndpoints", "ec2:DescribeVpcEndpointServiceConfigurations", "ec2:DescribeVpcEndpointServicePermissions", "ec2:DescribeVpcEndpointServices", "ec2:DescribeVpcPeeringConnections", "ec2:DescribeVpcs", "ec2:DescribeVpnConnections", "ec2:DescribeVpnGateways", "ec2:GetHostReservationPurchasePreview", "ec2:GetLaunchTemplateData", "ec2:GetReservedInstancesExchangeQuote" ], "Resource": "*" } ] }

  1. Click Review Policy.
  2. Provide a policy name such as “CloudPhysicsReadOnlyPolicy” to be applied to your future role and a brief description to facilitate future searching.
  1. Click the Create Policy button to finish the process of defining a new policy for CloudPhysics.

Create CloudPhysics AWS Role

  1. Create a new role in the AWS IAM Console.
  1. Click Create Role.
  2. Select Another AWS account.
  3. Use 863002038009 in the Account ID field in the AWS IAM Console. This will grant CloudPhysics access to your AWS configuration data.
  4. Check Require external ID (Best practice when a third party will assume this role).
  5. Copy the AWS External ID presented to you in the CloudPhysics Observer Status Page located on the AWS Account Tab and paste it in the External ID field in the AWS IAM Console. The Observer Status PAge can be located by clicking on the Observer Status icon in the top menu bar. Partners: Visit the “My Accounts” button on the Partner Portal then navigate to the Observer Status Page to access your own AWS External ID.
EXTERNAL ID
NOTE: This External ID is a token generated by CloudPhysics that is unique to your account. This External ID is used to identify which applications are connecting to your account and adds a second level of security to your data access. You must use the External ID generated by CloudPhysics on the CloudPhysics AWS Account Tab to complete the AWS Account setup process. You cannot proceed or connect without your unique External ID. You view your Exteranl App ID by logging in to CloudPhysics and selecting the AWS Account Setup button for new users or choosing the Observer Status Icon in the top menu bar. This External App ID is not the same as an organization token used by the CloudPhysics Observer. Your External Application Id can only be found on the CloudPhysics Status Page for AWS.

 

 

or

Partners: Visit the “My Accounts” button on the Partner Portal then navigate to the Observer Status Page to access your own AWS External ID.

  1. Click Next: Permissions.
  2. Filter your policies by searching for the policy you created earlier.
  1. Select the policy you created earlier.
  2. Click Next: Review
  3. Provide a role name such as “CloudPhysicsReadOnlyRole” and a brief description to facilitate future searching.
  4. Click Create Role.

Configure Your New AWS IAM Account in CloudPhysics

  1. After completion of the previous step to create an ARN Role, click the name of your new role to view the role summary.
  2. Copy the Role ARN from the role summary and paste it in the AWS Role ARN field on the right.
    Example:  arn:aws:iam::123456789012:role/CloudPhysicsReadOnly.
  1. Enter your AWS Account ID. You can find your AWS Account ID in your Account Summary page on AWS. This value should match the account ID used in your ARN Role created above.
  2. Provide an Account Name such as “Prod-US-West” to identify your account if multiple accounts are configured.
  3. If you intend to collect billing details, jump ahead to Preparing AWS to Share Detailed Billing with CloudPhysics and do not submit your CloudPhysics account form yet.
  4. If not adding a Billing Report access, Click Add Account.
  5. Congratulations! Your AWS account is now configured with CloudPhysics.

AWS Cost and Usage Report Bill Collection

CloudPhysics collects billing and usage data from Amazon Web services from a shared AWS S3 bucket, in which Amazon Billing and AWS Partners will create cost and usage reports or detailed billing reports. Your organization will need to establish an S3 bucket for these reports if they do not already exist and share this resource with CloudPhysics.  In this process, you will enable AWS to write reports to the S3 Bucket and define a user with whose credentials to collect the data by CloudPhysics. For this process, you will apply this policy to your existing CloudPhysics AWS Collection account. This user needs to be granted access to the bucket. Additional permissions are required if utilization and analytics data is to be retrieved together with the usage data and costs by means of CloudWatch.

CloudPhysics prefers to use the new Amazon Web Services Cost and Usage Report over the legacy Detailed Billing Report. The steps for setting up your Cost and Usage Report are detailed below.  If you already have these services enabled, you can jump ahead to granting permissions to CloudPhysics to read from your S3 Bucket and setting up the CloudPhysics collection process.

Steps to Share Cost and Usage Data

  • Create an AWS S3 Bucket for billing reports storage
  • Create a Cost and Usage Report directed to the new S3 Bucket
  • Apply Resource Permissions to Bucket for AWS
  • Create IAM Policy for S3 Bucket Read-Only Access
  • Apply policy to CloudPhysics Role
  • Provide Cost and Usage Report details to CloudPhysics

Setting up your AWS S3 Bucket for Bill Storage

These steps pertain to payer accounts (master accounts) in AWS and are used for the storage of the primary payer bill in an Amazon Web Services S3 Bucket. Additional steps for linked accounts (member accounts) are described in Configuring Linked Accounts for AWS Billing later in this document. For more details, refer to the AWS documentation.  Note: Detailed billing is only available for payer accounts (master accounts).

  1. Log in to the AWS console at https://console.aws.amazon.com with your Amazon account and username.

Log into the AWS Console

Create an S3 Bucket for bill storage

Next, we need an S3 Bucket to hold the Cost and Usage Report.

  1. Visit AWS Management Console of the S3 (Storage at https://s3.console.aws.amazon.com/s3/ ) AWS service.
  1. Click Create Bucket
  1. Specify a globally unique bucket name for your report and select a region of your choice.Example: myorg-billing-bucket
  2. Write your unique bucket name down. You will need to provide this to CloudPhyscis for use in the Cost and Usage Report Setup and later in the IAM Policy Setup.

Setup Cost and Usage Report

If you have not already enabled Cost and Usage Report generation on AWS, these steps will guide your through the process of enabling the creation of Cost and Usage Reports in a format that offers the greatest granularity and visibility into your services and operations.
  1. Visit AWS Billing & Cost Management Dashboard Page at https://console.aws.amazon.com/billing/home
  2. Choose Reports from the menu
  1. On the AWS Cost and Usage Report Page, choose the “Create Report” button.
  1. On the Report Content Page, provide a unique report name for your report.
    Example: my-cost-and-usage-report.
    Write this value down. You will need to provide this to CloudPhysics. 
  2. Check the box to include resource ID’s. This will enable the ability to view costing per EC2 Instance and other resources individually. We will use this to identify dedicated cost per instance.
  3. Press the NEXT button.

 

Point Cost and Usage Report to S3 Bucket

  1. On the Delivery Options page, provide the name of your S3 bucket.
  1. Select Verify.  If the Verification fails, chances are that Amazon does not have access to write data into your new S3 Bucket.

 

How to grant AWS access to write a report in your S3 Bucket

With a new S3 Bucket, AWS will need to be granted access to write your report into your S3 bucket. If during the process to verify your S3 Bucket name you are told the bucket name you provided is invalid, it is most likely that AWS does not have access to the bucket. Follow these steps to grant AWS permissions to write your cost and usage report into your bucket:

  1. Click on “Sample Policy” in the instructions at the top of the Delivery Options page.
  1. Copy the sample policy provided by AWS and save it to a clipboard or notepad. Note, the policy ID will be unique to your account.
  1. Open a new window and return to the AWS S3 Bucket page at https://s3.console.aws.amazon.com/s3/home
  1. Select your S3 Bucket for your bill
  2. Select the Permissions tab
  1. Choose the Bucket Policy button to expose the Policy JSON window
  2. Paste in your sample AWS Policy JSON and press SAVE button.
  3. Return to your Cost and Usage Report screen to verify access to your bucket.

    1. Specify a unique storage path to keep your reports. This will be your “Path Prefix”.Write this value down. You will need to provide this value to CloudPhysics.The report engine will create lots of files. By specifying a storage path prefix, you can keep multiple cost and usage reports within the same S3 bucket and track accounts individually. Path Prefix will be presented differently if you View a Report Config compared to edit a report. If you Review a Report, not that the Path Prefix displays two items, the path and report name. It may be best to choose EDIT to get the correct Path Prefix. It will typically be a simple pathname and will not contain a “/”. If you see a “/” in the Path Prefix, it is likely the Path Prefix and the Report Name.Example: my-cost-and-usage-report-path

       

       

 

 

    1. Check the radio button for Hourly Granularity. This will allow you to see changes within a day vs operations on a daily basis.
    2. Keep default for Report Versioning as “Create New Report Version”
    3. Do not check the option to export to Amazon Data Services.
    4. Keep default for Compression Type as GZip.
    5. Press the next button
    6. Review your settings
    7. Press the Review and Complete button to finish Cost and Usage Report Generation setup.

 

 

 

 

 

 

 

Create IAM Policy for S3 Bucket and Objects Read Access

Next, we need to grant CloudPhysics access to your S3 bucket to collect your Cost and Usage Reports.  This is done in the IAM section of the AWS Console.

  1. Return to the AWS IAM page at https://console.aws.amazon.com/iam/home
  2. Select Policies from the left hand menu, then press Create Policy button.
  1. Choose the JSON tab and paste in the following JSON
Allow Bucket Read Access Policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::Insert-your-bucket-name-here",
                "arn:aws:s3:::Insert-your-bucket-name-here/*"
            ]
        }
    ]
}

  1. Replace the <Insert-your-bucket-name-here> with the name of your S3 Bucket created earlier.  Be sure the second bucket item ends in /* in order to grant access to the files contained within the bucket.
  2. Press Review Policy
  1. Provide a Policy Name such as “My-Cost-and-Usage-Report-S3-Readonly”
  2. Provide a description.
  3. Press Create Policy button to complete.

Apply the new IAM Policy to the CloudPhysics AWS Role

  1. Select ROLE from the IAM Menu
  1. Search for your CloudPhysics AWS ROLE to grant it access to your S3 Bucket.  When you find your ROLE, click on the role name to modify its policies.
  1. Press Attach Policy button.
  1. Search for your new S3 Bucket policy created earlier and check the box next to the policy name.
  1. Press the Attach Policy button on the bottom to attach the new policy to the role.
  1. You will now have a confirmation of the policy attachment.  This completes the setup in AWS.

Provide Bill details to CloudPhysics

  1. Return to the CloudPhysics AWS Account setup page to add your AWS Cost and Usage Report details
  1. In the Account Setup, provide the following fields in addition to the existing AWS Account ID, Account ARN, and Account Friendly Name used to connect your account above.
    • Bucket Name – The name of the S3 bucket you created in AWS. Use just the bucket name. Not the S3 or ARN values are required. Example: myorg-billing-bucket
    • If only a Detailed Billing Report is available, check the Use DBE checkbox.
    • Report Path Prefix – The prefix of the Cost and Usage report as specified in the AWS delivery options form.
      Example: my-cost-and-usage-report-pathPLEASE NOTE: Only provide the Path Prefix you specified in AWS form earlier, not the full URL provided in the AWS Confimration Screen. Provide only the my-cost-and-usage-report-path you created. Do not use a path prefix like my-cost-and-usage-report-path/myorg-report-name/date-range which is a combined name that contains both the path prefix and report details.
    • Report Name – The name of the Cost and Usage report as specified in AWS when creating the report. Example: my-cost-and-usage-report
  1. Press Save or Update.
  1. You are now setup to collect the AWS Cost and Usage Report billing details for analysis by CloudPhysics

Data Collection

After all the configuration steps have been completed successfully, CloudPhysics is ready to collect the usage data and costs from AWS.  AWS writes cost and usage reports or billing reports to the S3 bucket several times per day. CloudPhysics will checks for updates daily in the reports. If there are any, they are retrieved and aggregated in CloudPhysics. It may thus take some hours (about 4 to the maximum of 24) before the first data becomes visible in CloudPhysics.  CloudPhysics will collect the previous month’s bill. If the updates to the previous month’s bill become available, CloudPhysics may collect those updates to ensure the accuracy of the billing data.

Collection Schedule:
Performance Data: Hourly
Configuration Data: Hourly
Bill Data: Daily + previous month once.

Resources and Links:

CloudPhysics AWS Connection Status:CloudPhysics Premium Users Observer Status Page within Cloud Physics.

Observer Status Icon

Partners should use the “My Account” button in the partner portal and then navigate to AWS Collection Setup page.

CloudPhysics Policy for AWS JSON File:/static/uploads/2018/09/cphyaws-08212018-json.txt
AWS IAM Dashboard:https://console.aws.amazon.com/iam/home#/policies
YouTube Vide on Setting Up AWS with CloudPhysicshttps://youtu.be/aYjYVrUgE3E
CloudPhysics Support Emailsupport@cloudphysics.com