Via Terraform, Implementing the DataDog log monitoring and synthetic transaction monitoring at a new site.
Estimated Time to complete: ~4 hrs.
The following Datadog services are implemented via terraform.
synthetic transaction monitoring to confirm backend services are working
log monitoring and alerts to detect underlying error conditions in logs
infrastructure monitoring to detect emergent events in the infrastructure that supports applications and services
A. Terraform: In order to configure services via the Datadog api, we’ll use Terraform to interact and build our monitoring and alerts.
Download the approporiate terraform version for your workstation (Mac,PC, Linux) and configure (Version as of this writing is v0.12.26)
2. Terraform is provided as a single binary. Once downloaded, unzip it and move it to a folder included in your workstation’s PATH.
B. Source code for Datadog build
1. Run the following to clone the
rhedcloud-service-monitoring repository to get the necessary terraform folder to configure Datadog services via Terraform.
git clone email@example.com/rhedcloud/rhedcloud-service-monitoring.git
C. Datadog Authentication
1. Prepare the Datadog API & APP Key
a. We can find the Datadog API & APP key under the
tab, under intergations selection APIs Once you have credentials, create a file in the root of the terraform repo we cloned:
b. The contents of the file will be in the following format:
api_key = "dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”
app_key = "2xxxxxxxxxxxxxxxxxxxxxxxxxxx"
D. Check connectivity via terraform
We should be able to run a terraform commands against the cloned repo to be able to modify and update Datadog configurations.
1. run terraform init to build any needed resources for the terraform configs
terraform init rhedcloud-service-monitoring
A successful run of
terraform init should produce output similar to : 1
Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!
2. Next, run terraform plan to check for pending changes
terraform plan rhedcloud-service-monitoring
A successful run of
terraform plan should produce output similar to: 1
Plan: 112 to add, 0 to change, 0 to destroy.
Note: You didn't specify an "-out" parameter to save this plan,
so Terraform can't guarantee that exactly these actions will
be performed if"terraform apply" is subsequently run.
3. Run apply to apply the list of pending changes
terraform apply rhedcloud-service-
A successful run of
should produce an output along the lines of: terraform apply 1
No changes. Infrastructure is up-to-date.
Apply complete! Resources: 112 added, 0 changed, 0 destroyed.
E. Bitbucket repo
Fork the rhedcloud-service-monitoring repo for the new organizational construct it pertains to.
F. Datadog Setup
Configure a private location in Datadog.
https://app.datadoghq.com/synthetics/settings and start the process for creating a private location for the Datadog organizational tests. Fill out the fields and choose an appropriate API key.
2. Once you’ve completed filling out the form, click the “
save location” button.
3. You’ll be presented a screen that contains Datadog connectivity info for the AWS ec2 instance we’ll be creating.
a. Expand the window, copy and paste the connectivity script to a safe place as we’ll need this later within our ec2 instance.
b Once you’ve secured the connectivity info, click the “
view installation instructions” button.
c. Here you will find the command line instructions needed to setup the docker instance, you can choose either docker or docker compose.
G. Create an AWS ec2 instance for the datadog private location
We’ll need to create an ec2 instance for our Datadog private location. Admins should have an idea of what is needed but here are some notes about the instance that should help.
The instance will have a private subnet
Be sure to include a security group for inbound ssh
The instance needs internet access
A t3.small instance can get you started (remember that these have limited bandwidth)
We’ll need to install docker and optionally docker compose
Finally, run the Datadog instructions on the ec2 instance
H. Once the ec2 configuration is complete, Verify the private location is up and running by checking the “health status” in Datadog.
I. Editing the repo for the new Datadog environment
Within the URL
https://app.datadoghq.com/synthetics/settings, in the newly created synthetic test private location, click on the configuration “sprocket / cogwheel”.
Once on the next page, look at your browser URL bar, note the URL portion of the page that looks like “pl:cimp-170-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx”. Copy it and keep it somehwere for reference, we’ll need it.
locals.tf file in your repo.
Enter the pl:cimp-170-xxxx value in the appropriate variable :
synthetics_privatelocation = "pl:cimp-170xxxxxx”
5. To update the email address within the Datadog configuration:
6. Enter the appropriate email
J. Running the terraform apply to activate the Datadog configuration.
From here, you should be able to run
And have terraform take care of the rest.