RHEDcloud Monitoring Setup Guide

Background

Via Terraform, Implementing the DataDog log monitoring and synthetic transaction monitoring at a new site.

Estimated Time to complete: ~4 hrs.

 

The following Datadog services are implemented via terraform.

  1. synthetic transaction monitoring to confirm backend services are working

  2. log monitoring and alerts to detect underlying error conditions in logs

  3. infrastructure monitoring to detect emergent events in the infrastructure that supports applications and services

A. Terraform: In order to configure services via the Datadog api, we’ll use Terraform to interact and build our monitoring and alerts.

  1. Download the approporiate terraform version for your workstation (Mac,PC, Linux) and configure (Version as of this writing is v0.12.26)

2. Terraform is provided as a single binary. Once downloaded, unzip it and move it to a folder included in your workstation’s PATH.

B. Source code for Datadog build

1. Run the following to clone the rhedcloud-service-monitoring repository to get the necessary terraform folder to configure Datadog services via Terraform.

git clone username@bitbucket.org/rhedcloud/rhedcloud-service-monitoring.git

C. Datadog Authentication

1. Prepare the Datadog API & APP Key

a. We can find the Datadog API & APP key under the intergations tab, under APIs selection
Once you have credentials, create a file in the root of the terraform repo we cloned:

rhedcloud-service-monitoring/creds.auto.tfvars

b. The contents of the file will be in the following format:

1 2 api_key = "dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx” app_key = "2xxxxxxxxxxxxxxxxxxxxxxxxxxx"

D. Check connectivity via terraform

We should be able to run a terraform commands against the cloned repo to be able to modify and update Datadog configurations.

 

1. run terraform init to build any needed resources for the terraform configs

a. terraform init rhedcloud-service-monitoring

A successful run of terraform init should produce output similar to:

1 2 3 4 5 6 7 Initializing modules... Initializing the backend... Initializing provider plugins... ... Terraform has been successfully initialized!

 

2. Next, run terraform plan to check for pending changes

b. terraform plan rhedcloud-service-monitoring

A successful run of terraform plan should produce output similar to:

1 2 3 4 Plan: 112 to add, 0 to change, 0 to destroy. Note: You didn't specify an "-out" parameter to save this plan, so Terraform can't guarantee that exactly these actions will be performed if"terraform apply" is subsequently run.

 

3. Run apply to apply the list of pending changes

c. terraform apply rhedcloud-service-monitoring

A successful run of terraform apply should produce an output along the lines of:

1 No changes. Infrastructure is up-to-date.

or

Apply complete! Resources: 112 added, 0 changed, 0 destroyed.

 

E. Bitbucket repo

  1. Fork the rhedcloud-service-monitoring repo for the new organizational construct it pertains to.

 

F. Datadog Setup

  1. Configure a private location in Datadog.

    1. Navigate to https://app.datadoghq.com/synthetics/settings and start the process for creating a private location for the Datadog organizational tests. Fill out the fields and choose an appropriate API key.

 

2. Once you’ve completed filling out the form, click the “save location” button.

3. You’ll be presented a screen that contains Datadog connectivity info for the AWS ec2 instance we’ll be creating.

a. Expand the window, copy and paste the connectivity script to a safe place as we’ll need this later within our ec2 instance.

b Once you’ve secured the connectivity info, click the “view installation instructions” button.

c. Here you will find the command line instructions needed to setup the docker instance, you can choose either docker or docker compose.

 

G. Create an AWS ec2 instance for the datadog private location

  1. We’ll need to create an ec2 instance for our Datadog private location. Admins should have an idea of what is needed but here are some notes about the instance that should help.

  • The instance will have a private subnet

  • Be sure to include a security group for inbound ssh

  • The instance needs internet access

  • A t3.small instance can get you started (remember that these have limited bandwidth)

  • We’ll need to install docker and optionally docker compose

  • Finally, run the Datadog instructions on the ec2 instance

H. Once the ec2 configuration is complete, Verify the private location is up and running by checking the “health status” in Datadog.

Find the private location you created here: https://app.datadoghq.com/synthetics/settings

 

I. Editing the repo for the new Datadog environment

  1. Within the URL https://app.datadoghq.com/synthetics/settings, in the newly created synthetic test private location, click on the configuration “sprocket / cogwheel”.

  2. Once on the next page, look at your browser URL bar, note the URL portion of the page that looks like “pl:cimp-170-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”. Copy it and keep it somehwere for reference, we’ll need it.

  3. Open the locals.tf file in your repo.

  4. Enter the pl:cimp-170-xxxx value in the appropriate variable :

synthetics_privatelocation = "pl:cimp-170xxxxxx”

5. To update the email address within the Datadog configuration:

Open the synthetics-api.tf file in your repo.

6. Enter the appropriate email value :

<ema1:EmailAddress>user@domain.com</ema1:EmailAddress>

J. Running the terraform apply to activate the Datadog configuration.

From here, you should be able to run

terraform plan

then

terraform apply

And have terraform take care of the rest.