Webinars and Demonstrations
The detailed demonstration covers the following topics:
- Emory AWS Landing Page
- Provisioning an account
- All features of the VPCP Console
- Accessing the AWS Console
- Creating an EC2 instance and accessing it
- API/Command-line access with the Temporary Key Issuance (TKI) Client
- Security policy enforcement with IAM policies, SCP policies, and the Security Risk Detection Service
- Demo and description of how SRDs work
- Service-oriented architecture that enables these applications and automation
- Desire to create a collaboration with other organizations and how this might be deployed at other sites
Emory Amazon Web Services (AWS) is Emory University’s preferred and recommended Cloud service for computational infrastructure needs, such as high performance computing and big data analytics. The service provides access to Amazon’s Cloud Computing services, including computing, storage, database, etc., with the exception of services that have been blocked for security purposes or those that have not been assessed for risks. The Emory AWS Service is a tripartite mission platform that can facilitate the advancement of science, education, and service across the University.
Below are some examples and use cases of projects using this service.
Yerkes National Primate Research Center
Amit Upadhyay, PhD, a researcher in the Yerkes Primate Center has successfully reconstructed B-cell antibody sequences from single cell RNA-Seq data with AWS. The process included creating a bioinformatics Amazon Machine Image (AMI), uploading the data to S3 using the AWS Command-line Interface, and then launching the assembly and the down-stream analyses in EC2 instances with the AMI. Using this process, analyses was carried out for approximate 700 single cells. The work was performed in collaboration with Gregory Tharp, and Steven Bosinger, PhD.
Upadhyay, Amit A., Robert C. Kauffman, Amber N. Wolabaugh, Alice Cho, Nirav B. Patel, Samantha M. Reiss, Colin Havenar-Daughton, Reem A. Dawoud, Gregory K. Tharp, Iñaki Sanz, Bali Pulendran, Shane Crotty, F. Eun-Hyung Lee, Jens Wrammert and Steven E. Bosinger, "BALDR: a computational pipeline for paired heavy and light chain immunoglobulin reconstruction in single-cell RNA-seq data." Genome medicine 10, no. 1 (2018): 20.
Emory Clinical Biomarkers Laboratory
The Emory Clinical Biomarkers Laboratory, directed by Dr. Dean P. Jones, operates a high-resolution metabolomics platform to analyze clinical samples, e.g. plasma and tissue samples from HIV patients, for identification of metabolic biomarkers and to improve the understanding of disease mechanisms. Under the technical guidance of Karan Uppal, PhD, Director of Computational Metabolomics of the laboratory, AWS is used to support various data processing workflows including: 1) data extraction from instrument files using R packages apLCMS, xcms, and xMSanalyzer; 2) quality evaluation using xMSanalyzer; 3) annotation using xMSannotator; 4) statistical, network and pathway analysis using R and Python based tools. The lab generally uses AWS EC2 spot instances (with low risk of out-bidding) for computing to maximize cost saving and uses S3 for data transfer and sharing. Overall, the lab is enjoying the flexibility and scalability of AWS for most of their computation needs.
Emory Integrated Computational Core
Rich Johnston, PhD, director of the Emory Integrated Computation Core (EICC) has recently mapped and called 1027 genomes in less than a week with AWS using PEMapper and PECaller [https://www.ncbi.nlm.nih.gov/pubmed/28223510]. Each of these 30x genomes was about 60 to 100 GB compressed in size, and the mapping would typically take 12 hours on a 64-core workstation. While the on-premises Emory Department of Genetics’ HPC compute cluster could only handle 2 genomes per day, Rich was able to launch 200 AWS EC2 instances in parallel to process 200 genomes per day. The whole workflow was scripted and took advantage of AWS APIs to enable the automation. S3 was used to stage the genomes and handles the transfer of the results. In short, AWS’s scalability allows Rich to meet his project deadline fast. For additional information, Rich is available to share his AWS experience. You can contact him at email@example.com.
The Emory AWS Service is available to any active user within the Emory community when used for the purpose of conducting Emory business and used within the appropriate guidelines and terms of service. Because the service requires a financial commitment, only Emory faculty and staff can request the creation of an account, for which they become the account owner. Once created, an account can be extended to add other Emory community roles, such as students. Collaborators under the context of an academic collaboration may also be granted access to an account with the use of an Emory sponsored identity account. Please note, the account owner has accountability for the AWS usage costs accrued within their account.
For more information on initiating an Emory AWS Service account, please see the Account Creation page.
The Emory AWS Service supports two classes of accounts: (1) “HIPAA-designated” accounts that can be used to store and utilize electronic protected health information (ePHI) as part of Emory’s covered entity; (2) standard accounts that should not be used to store or use ePHI data. Since Emory may only use services designed to be HIPAA eligible by Amazon, the HIPAA-designated accounts have a subset of AWS services available. Beyond the difference in the number of available services, the Emory-specified controls are very similar for HIPAA and standard accounts, thereby ensuring the highest level of security for both types of accounts.
As with most Cloud providers, Amazon uses a ‘pay-as-you-go’ model, charging only for the services utilized. Emory does not charge any overhead for these native AWS services, but rather passes the costs directly to the account owner. There may be some instances when a faculty member may wish to engage LITS, a research core or a consulting firm for additional help in managing a service, which may incur additional costs. To help estimate base costs of utilization of each AWS service, Amazon provides a calculator: https://calculator.s3.amazonaws.com/index.html
In addition to the ‘pay-as-you-go’ model, please be aware that Amazon has two additional costs models for its EC2 (compute) service. The three pricing models are: on demand, reserved, and spot. As pricing can vary depending on the cost model selected, we recommend learning a bit more about these pricing models before spinning up an EC2 service.
To help simplify the billing process, you must use a valid 10-digit Emory SpeedType to pay for these services.
As part of the account creation process, the faculty or staff member must enter a valid 10-digit SpeedType. The process verifies that the SpeedType is an active SpeedType in Compass and requests explicit confirmation before proceeding to ensure the appropriate SpeedType is entered.
Each account has a single associated SpeedType. The Emory AWS Service is not supporting splitting expenses among multiple SpeedType.
Current charges can be accessed through the reports in the AWS console. Historic charges are also kept within the Emory AWS management application (VPCP app) and are visible in Compass and associated EBI reports.
As long as the AWS services directly support a specific research award and have clear benefit to the research, the service may be charged as direct expenses to research grants. To help investigators within including AWS services in their grants, investigators can contact the LITS group for free consultation on their design and cost estimation. We also have template language explaining the service which may be included in your grant narrative.
Based on use case discovery work with faculty and their teams, the Emory AWS Service supports two types of Virtual Private Clouds (VPC) in AWS accounts:
- Type 1 which extends Emory’s network into the Cloud with no direct access to the Internet without traversing back through Emory’s network and
- Type 2 which extends Emory’s network into the Cloud with direct access to the Internet from AWS.
Most Emory use cases defined to date can be supported with a Type 1 VPC. However, public-facing applications with a large user base may warrant a Type 2.
Each VPC will automatically be configured to have a pair of 3 network zones, a public, a private, and a management, across 2 availability zones. For a pictorial representation, please see below:
Currently, we have AWS region, US N Virginia or us-east-1 offered for server-based products e.g. EC2.
For more details on the technical architecture, please see the AWS Technical Architecture design pages.
Designated AWS services, when used through the Emory AWS Service HIPAA-designated accounts, comply with Emory’s HIPAA policy and procedures and may be used with electronic protected health information (ePHI) and individually identified health information (IIHI) with the appropriate compliance authorization and user-based practices. For the current list of these services, please refer to the following updated list of Emory AWS Services for use with HIPAA workloads.
Use of an AWS HIPAA eligible service with a non Emory AWS service account, such as a personal account, is not considered compliant with Emory’s HIPAA policies and should not be used with ePHI or IIHI.
Any sensitive data used or stored in this service must go through the appropriate compliance authorization process. Identifiable data originating from the Atlanta VA, Children’s Healthcare of Atlanta, and Grady may have additional steps before they may be used in this service. Data that requires infrastructure compliant with PCI or FISMA may not currently be used in the service. Please contact us for other avenues.
In order to assist with the appropriate use of these services and the protection of sensitive data loads, the Emory AWS Service deploys Risk Detectors that check to ensure an account is not misconfigured in a way that may introduce significant risk. These detectors run frequently and will notify the account owners of changes needed to remediate these risks. A listing of all AWS services offered and their security measures and the impacts of these measures can be viewed in the Emory AWS management application.
Additional information on information security can be found on the Information Security page.
In addition to traditional AWS console and command-line access, Emory users also have access to an Emory AWS management application, known as the VPCP application, to manage Emory-specific aspects of their accounts, such as which Emory people can manage their accounts, Emory networking information, Emory firewall rules, Emory IP addresses, and Emory information security notifications.
Some of software licenses that Emory has purchased for use may have provisions that prohibit their use on the Cloud. Please see the following page for more information on the site license software and their use in the Cloud.
There are a number of resources to help interested users move forward with the Emory AWS Service. Introductory training to the service and initial consultations provided by LITS staff are available to Emory users and groups and can be initiated through a Service Desk ticket. Other avenues include reaching out to local support, Research Core facilities or calling the University service desk.
More in-depth information on particular services can be found with AWS support services. Training videos, FAQs, message boards, and white papers are all available.
Faculty member may also wish to engage LITS, a research core, or a consulting firm for additional help in configuring or managing a set of services to implement a particular storage, computing or development project on AWS. Please see this list of partners.
In addition, LITS hosts Ask-the-Expert sessions with AWS Architects, more specific training sessions and is convening a Center of Excellence group for Cloud Computing. If you are interested, please sign up through by emailing to firstname.lastname@example.org.
Requesting the Service
The launch page for the Emory AWS Service is http://aws.emory.edu. Note, the content at this location prior to initial service launch looks nothing like the intended landing page design. The non-production environments of this page are what one should reference at:
The purpose of the Emory AWS Landing Page is to provide an easy-to-reference launch page and entry point to request the service, login to the service, and explore service details. It is analogous to the AWS Landing Page that AWS itself maintains at http://aws.amazon.com.
Initial AWS Account and VPC provisioning as well as the provisioning of additional VPCs into existing accounts has been automated in a 48-step provisioning process that integrates the AWS accounts and VPCs into Emory's network, identity, and security infrastructure. The provisioning process presently takes about 10 minutes and users can watch the status of the provisioning process interactively or request the service and be notified when the automatic provisioning is complete. The current steps of the provisioning process are listed in the AWS Account Service Technical Design Document. The following are the key results of the provisioning process, which can be verified through observation and tests:
|1||Authorize the requestor||Emory Shared Data|
|2||Assign AWS account e-mail address||Office365|
|3||Create new AWS account or authorize re-use of existing account||AWS|
|4||Delete or verify there are no default VPCs in the account||AWS|
|5||Create/update the account||AWS|
|6||Create the account alias||AWS|
|7||Create and configure the account SAML provider||AWS|
|8||Create LDS organizational unit||LDS|
|9||Create LDS groups||LDS|
|10||Create/update roles for the AWS account||IDM|
|11||Create/update account metadata||Emory AWS Account Service|
|12||Create the requested VPC||AWS|
|13||Create metadata for the requested VPC||Emory AWS Account Service|
|14||Create a site-to-site VPN connection||Network Ops Service|
|15||Place the account in the correct AWS organization||AWS|
|16||Request BAA coverage for HIPAA accounts||AWS|
|17||Execute production readiness test||AWS/All Emory Components|
To request the service users will visit the landing page mentioned above and click on the "Create Account/VPC" button, which will take them into the ServiceNow request form for the Emory AWS Service. The Emory AWS Request form allows users to request a new VPC account or add to an existing VPC account by adding the relevant information to successfully provision the account in AWS. Once users enter the form, they can choose to add a new account or add to an existing account.
If they choose to add a new account, they are prompted to add a compliance class, account admin(s) NetID, account owner's NetID, type of VPC, description of the purpose of the account, 10-digit Speedtype, any security compliance regimes, and agree to the terms of service. One field in particular–the 10-digit Speedtype field–allows the user to validate their entry by sending their value to PeopleSoft that returns a message from the service to let the user know that their Speedtype is either valid, invalid, or has a warning along with both a description of what is at issue if a warning or invalid value is returned and descriptive fields related to their Speedtype account. After the message is returned from PeopleSoft, the user is prompted with a confirm message asking them if they are sure they want to use the Speedtype they have entered and then another confirm message that asks them to acknowledge that their response has been logged by the system.
After all of the required fields are completed and the Speedtype is successfully validated, a user clicks submit at the bottom left of the form. Once they submit, a SOAP message is created with all of the user's values included and is sent to ESB to kick of the provisioning process. What is transparent to the user is that as soon as they submit the form, their netID has been recorded as the authorized requestor, and that NetID will be sent along with the rest of the data in the outgoing provisioning envelop. Additionally, an info bar at the top of the screen appears, and the user can click on the link contained within the bar to view the status of their request in real-time. Each step is outlined within this page individually, and as each step is completed in the provisioning process, the row turns green. If there is an error in the provisioning process, the row will turn red. If a user has multiple AWS provisioning requests, they can also view all of their requests by returning to their main ServiceNow portal page and click on "View all" at the bottom right of their screen under "My Open Service Requests." Once they click "View all," they can see all of their outstanding requests (including but not limited to AWS requests), and for AWS Requests, they will see a indicator bar next to their request showing the progress of their provision that includes a green (no errors) or red (errors) indicator color as well as the steps completed, e.g., 15/21, in the middle of the indicator bar.
Central Administrators may use the VPCP application in development, test, staging, or production environment to generate new accounts or VPCs for testing, development, or on behalf of end user. To create accounts or new VPCs in existing accounts, users in the Central Administrator role can log into the VPCP application from the AWS landing page and select the VPC Provisioning tab. Similar to the ServiceNow request form central administrators provide all of the required inputs to generate an account and VPC or a new VPC in an existing account. Then they press the Generate button to initiate account or VPC provisioning. Central administrators may watch the progress of the provisioning process interactively if desired.
To request an additional VPC in an existing account users will visit the landing page mentioned above and click on the "Create Account/VPC" button, which will take them into the ServiceNow request form for the Emory AWS Service. The Emory AWS Request form for adding an additional VPC in an existing accounts works similar to adding a new account with a minor difference. If they choose to add to an existing account, they are prompted to input the existing VPC account ID they would like to add an additional account to. In it's current form, the VPC number they entered is sent as a VPCP Query through the AWS Account Service via the ESB. ServiceNow parses through the response and records two things for the user. First, it compares the end user's netID (which is collected transparent to the user) with the authenticated requestor netID that is returned from the query. If they match, it lets them continue. If not, it returns an error message letting them know they are not authorized to add to the VPC account number they entered. And transparent to user, it also records if the account they are adding to is HIPAA or standard without them having to choose the compliance class again as this value is needed when the form is submitted and sent through the AWS Service for VPCP Generate. All other required fields remain the same as if they were adding a new account, and clicking submit performs the same operations as described above when adding a new VPC account.
Central Administrators may use the VPCP application in development, test, staging, or production environment to add additional VPCs to existing accounts for testing, development, or on behalf of end user. To create VPCs in existing accounts, users in the Central Administrator role can log into the VPCP application from the AWS landing page and select the "VPC Provisioning" tab ...[Tod describe next steps in detail]
Accessing the Service
General Access Requirements
Most applications and features of the Emory AWS Service are accessible from the internet without any requirement to connect to the Emory VPN or Emory network. Users authenticate with their Emory NetID and Duo authentication and access Emory AWS Services using a web browser or command line interface.
Users must connect to the Emory VPN in order to access resources deployed into an Emory Type 1 VPC. The Emory VPN can also be used for administrative access to resources deployed into Emory Type 2 VPCs, but public access to these resources can be exposed to the internet through AWS internet gateways and Emory firewalls deployed in AWS.
The following is a summary of what is required to access each major aspect of the Emory AWS Service:
|Emory AWS Landing Page|
|AWS Console||Button on AWS Landing Page||internet connectivity, Emory NetID/password, Duo authentication|
|VPCP Console||Button on the AWS Landing Page||internet connectivity, Emory NetID/password, Duo authentication|
|Resources deployed in Emory VPCs||Extension of the Emory Network|
Emory VPN access, which also requires internet connectivity, Emory NetID/password, Duo authentication
Note: like any other resources on the Emory network, firewall rules can be adjusted to expose these resources to the broader Emory network or to the internet
Requesting Access for Emory People
All access for Emory people is automated by the provisioning process for users specified in the provisioning request and by a user provisioning orchestration at the time users are added to roles from within the VPCP application. Important note: presently this provisioning cannot be triggered by adding the user to the role through the backend IDM service. In order to do that IDM would need to either invoke the user provisioning service at that point of service or we would need to implement a polling connector to look for new users added into roles.
The following access is verified or granted for Emory people during the user provisioning process:
|1||Basic VPN access (VPNAllow)||Emory SSL VPN|
|2||AWS VPN access (AWSAllow)||Emory SSL VPN|
|3||Requested Emory AWS Account role||IDM|
Requesting Access for Non-Emory People
Many Emory AWS account owners and admins will need to grant access to their account for non-Emory people. Access for non-Emory people can only be provisioned automatically after the non-Emory people have Emory sponsored accounts. The process for requesting Emory sponsored accounts is:
|1||Collect consultants name, email, dob, and phone|
|2||Fill out and submit SN request. Use "Remote" for "Campus Location"|
|3||Ask Sriram Chari to expedite request|
|4||Wait to receive the “UPDATE: An Item from Request REQ99999 has been Updated” message that contains the directions for the consultant to change their password and establish their security questions.|
|5||Forward the message from step (4) to the consultant|
|6||Consultant resets password, establishes security questions|
This process can take 1-4 days depending on whether or not the request is escalated. After the user has an Emory sponsored account, they can be added to the Emory AWS Account roles as part of a provisioning request or through the VPCP application and all other access required to access the account and VPC will be granted or verified automatically as part of that request.
To access the AWS Console for Emory AWS accounts, begin at the Emory AWS Landing page http://aws.emory.edu (or appropriate non-production environment URL indicated above) and select the "AWS Console" button to log into the AWS Console. This will take the user to Emory Login where they provide their Emory NetID and password. Then it will prompt for Duo two-factor authentication if the user is not on the Emory network. Important note: it will prompt for Duo two-factor authentication even if the user is VPN'd to the Emory network, because Emory Login seems to pass the original source IP to Duo and not the VPN pool IP address, so Duo believes VPN users are not on the Emory network.
Once the user successfully authenticates, if they have only one account and one role they will be immediately logged into the AWS console for that account and role. If the user has more than one account or role, the user will be presented first with a role assumption choice asking them which account and role they want to assume in the console. After the user makes a selection they will be logged into the selected account and role.
AWS Command Line
To access an Emory AWS Account from the command line, Emory requires the use of temporary API keys to reduce the incidence of accounts compromised by disclosure of persistent API keys. Temporary keys can be challenging to manage, so Emory had provided a command-line utility and a backed service to simplify the process of requesting and using temporary keys. These are called the Temporary Key Issuance Client (TKI Client) and the Temporary Key Issuance Service (TKI Service). Users will download and install the TKI Client on each workstation from which they would like to acces the account. A link to download the TKI Client can be found on the Emory AWS Landing Page at http://aws.emory.edu (or at its appropriate non-production URL listed above). After downloading the appropirate TKI Client package for your operations system, execute the TKI Client installer and follow the installer instructions.
To obtain temporary keys with the TKI Client, run the TKI Client with the platform-specific instructions provided by the install process and respond to its prompts for:
- Emory NetID
- Emory Password
- If username and password authentication is successful, the user will be prompted to select a Duo authentication method
- If Duo authentication is successful and if the user is associated with more than one account or role, the user will be prompted to select an account and role to assume
- The user will be issued temporary keys and the TKI Client will write them to the user's AWS profile and return the profile name of the new temporary credentials
The user can then interact with the account from the command line using the profile name that contains the temporary credentials. Temporary credentials are valid for 12 hours. Once the credentials expire users must re-execture the TKI Client to obtain new temporary credentials.
AWS Service Accounts
Some long-running applications will require service accounts for AWS. In many cases for applications running on AWS, AWS or LITS Support can advise on how to run applications as a role that will authorize all access they need without services accounts and API credentials. However, there are legitimate use cases in which applications will need API keys.
To request a service account and API keys the end user will fill out the Emory AWS Service Account/API request form. (This form doesn't exist yet and would need to be created.) A link to this ServiceNow form is available in the VPCP Application. The form requires the end user to enter required important information such as the AWS account ID, exact IAM Policy that is desired for each API key, provide a detailed description of how the Service Account/API keys will be used, and why any alternative secure methods cannot be used. At some point in time, the AWS account owner should be determined and have to provide an approval. The requestor should also be identified as a user of the specified AWS account. When the request form is submitted it will eventually be sent to the Enterprise Security team to approve the specified use case. Once the Enterprise Security team has approved the request, it will be forwarded to LITS Systems team to create a service account user and key pair for the customer. The key pair information will be distributed to the customer through a secure method of transport. Then the LITS Systems team will close the request.
The current policy and practice for service account credential rotation will be managed by a SRD/SRR that checks for IAM Users (including long-term key pairs) and deactivating resources that have been inactive for 30 days.
Administering the Service
Customers administer their accounts by being specified as administrators at the time the account is provisioned or by being added as an administrator by another person who already holds the administrator role. The administrator role is restricted in three major ways. First, it has restricted access to structures in the AWS account and VPCs in the account, which is implemented with IAM policies. These restrictions prevent customer administrators from changing the basic network topology, security, and authorization mechanisms of the account and VPC. Second, customer administrators (and also central LITS administrators) are restricted by Service Control Policies (SCP), which control access to specific AWS services and features of services and apply to all principals in all accounts within an organizational unit or AWS Organizations. So, all Emory HIPAA accounts have the same service control policies and all Emory standard accounts have the same service control policies implemented by the service control policies associated to their respective organizational units. Finally, Emory implements Security Risk Detectors (SRDs) and Security Risk Remediators (SRRs) for important measures that cannot be implemented declaratively by IAM policies or service control policies. These SRDs and SRRs run on a schedule or are trigger by events and detect unacceptable risks, perform remediation, and notify users. An example of these SRDs/SRRs is the detection and remediation of public S3 buckets. Emory policy states that users may not share data publicly without authentication and there is no policy mechanism to prevent users from publicly sharing S3 buckets. So an SRD detects public buckets and an SRR make public buckets private.
Customer administrators (and LITS administrators) will not be able to perform all operations that a root account holder could. The specific set of actions that are prohibited or allowed at any time is defined by the union of IAM policies, service control policies, and security risk detectors and remediators.
LITS Administrators have access to all customer accounts via a pre-defined IAM role which allows an elevated level of access required for maintenance, request fulfillment, and troubleshooting. LITS Administrators include members from across LITS including Enterprise Security, Identity Management, Middleware, Product Management, Solution Architects, and Systems. Membership of the LITS Administrators role is controlled through a LDS group /group and logins through the service console or temporary key issuance service.
LITS Administrators are restricted by SCPs and SRDs/SRRs too.
Emory AWS Landing Page
The AWS Landing Page is an integral part of the VPCP Console and AWS Console login sequence at Emory. This is articulated and deployed as a separate application, because it is an unauthenticated landing page that users can hit to explore the Emory AWS Service, order the service, and login to the service. This page is essentially a replacement from the landing page AWS provides at http://aws.amazon.com, which Emory users cannot use, because Emory has customized and restricted AWS services, implemented its own request form and provisioning process, and implemented it own login process with Emory Login. Users will go to http://aws.emory.edu or the appropriate non-production environment URL listed above to perform the following four main activities:
- Order the Emory AWS Service by clicking on the "Create Account/VPC" button
- Log into the AWS Console by clicking on the "AWS Console" button
- Log into the VPCP Console by clicking on the "VPCP Console" button
- Explore the AWS services offered in the Emory AWS Service
- Get general information and announcements about the service and link to more detailed documentation
It is important to note that this landing page is not a one-time or rarely-used user interface in the Emory AWS Service. Users will launch from this page every time they use the service and for many they will hit it every time they logout and switch accounts or access AWS services in a new browser window. For this reason, the Emory AWS Landing Page, the VPCP Console, and the AWS Console have been designed to look and behave similarly to create a consistent experience and not jar users as they move from account to account and console to console.
Virtual Private Cloud Provisioning (VPCP) Console
The VPCP Console is intended to provide Emory AWS Service users with a Single Pane of Glass (SPOG) for managing all Emory-specific aspects of their Emory AWS accounts. As part of integrating AWS accounts into the Emory network, identity, and security infrastructure Emory needed to restrict or change the approach to certain AWS functions such as:
- identity and access management
- network topology
- public IP addresses
- elastic IP addresses
Additionally, there are some account ownership and tracking data that cannot be managed in the AWS console such as:
- owner of the account
- purpose of the account
- compliance class of the account
- Emory security alerts and notifications
In order to serve Emory AWS Service users Emory complied all of this data and these functions into the VPCP Console application in one place. Emory anticipates improving this application and adding more features based on user feedback and new requirements for account provisioning, tracking, and security.
To manage account metadata launch the VPCP Console from http://aws.emory.edu or the appropriate non-production, environment-specific URL listed above. Click on the "VPCP Console" button, authenticate with your Emory NetID and password. Upon login you will see a list of Emory AWS Accounts with which you are affiliated and the roles you have for those accounts. Click over to the "AWS Account Maintenance" tab and you will see a paginated listing of the AWS accounts you are affiliated with.
To view or manage detailed account metadata, select an account from the list and click the actions button to select the action you would like to perform. Account administrators and auditors can "View/Maintain Account" which will present a detailed information page. Account administrators may edit much of this data. Account auditors may only view this data. Central administrators (LITS) may edit the same information as account administrators. The primary functions account administrators and central administrators perform from the detailed account metadata page are:
- Change the 10-digit SpeedType to which the Emory AWS account is billed
- Change the owner of the account (presently only the owner of the account can specify a new owner of the account or central administrators from LITS)
- Manage role assignments
- Manage e-mail addresses (central administrators from LITS only)
In addition to these features, central administrators (LITS) can also delete account metadata or view billing integration data from the account listing tab.
Manage Emory Elastic IP (Static NAT)
Users of the Emory AWS Service cannot make use of AWS Public IP addresses or Elastic IP addresses to expose their EC2 instances to the internet. This is one of the consequences of removing internet gateways from Emory AWS Service customer VPCs. Instead Emory routes network traffic through the Emory on-premises network (Type 1 VPC) or through a transit VPC with Emory-controlled firewalls in AWS (Type 2 VPC).
To replace the AWS Elastic IP feature, Emory has implemented Emory Elastic IP, which implements automated static NAT on-demand for customers from within the VPCP Console. To request an Emory Elastic IP, log into the VPCP Console and select the VPC you would like to work with on the VPC Maintenance tab. Click the Actions button and select View/Maintain VPC to see the VPC details view. From there click on the Elastic IP Assignments tab to view the Emory Elastic IPs assigned to your VPC. To allocate a new address, click the Allocate New Address button. To associate and existing Emory Elastic IP with a private IP address, select the Emory Elastic IP from the list and click on the Actions button and select Associate IP Address. Similarly to disassociate the address with a private IP select Disassociate Address. To release the Elastic IP address from your VPC select the Release Address action.
In order to expose an EC2 instance to the internet in addition to an Emory Elastic IP one must also place attendant firewall rule exception requests to allow traffic to traverse the Emory firewall and reach instances in the VPC.
Mange Emory Firewall Rule Exceptions
Users of the Emory AWS Service may place request for firewall rule exceptions to allow access to their assets in their VPCs to the Emory network or the Internet. Often there is a need to request that EC2 instances be allowed to access other resources on the Emory network. This can also be accomplished by placing an Emory firewall rule exception request. There are two ways that you can place Emory firewall rule exception requests: from the VPCP Console and from ServiceNow.
Placing Firewall Rule Exception Requests from the VPCP Console
Placing firewall rule exception requests from the VPCP console has several advantages.
- Users can view all existing firewall rule exceptions for their VPCs
- Users can select an existing firewall rule exception and essentially clone it with the Create New Firewall Exception Request Like This feature
- Users can select an existing firewall rule exception and delete it
To access the firewall rule exception features of the VPCP Console, select the VPC Maintenance tab and then select the VPC to administer. Select View/Maintain VPC from the Actions button. From the VPC Detail View, select the Firewall Rules tab.
Placing Firewall Rule Exception Requests from ServiceNow
Firewall rule exceptions for Emory AWS VPCs can also be placed using the traditional ServiceNow request form. Log into ServiceNow an use the Add or Remove Firewall Exception form. This form supports requesting AWS VPC firewall rule exceptions and those can also be viewed and managed from the VPCP Console if users specify the appropriate AWS VPC ID on this form.
When Emory AWS Service users receive notifications from automations or administrators they are alerted with an indicator at the bell icon at the top of the VPCP Console. Click on the bell icon to see current alerts. On this view users can specify if they want to receive e-mail notifications in addition to in-app notifications. Users can also select a notification to view details, mark it as read, or delete it. Notifications are never really deleted, because many notify users of critical security issues. Deleting a notification just removes it from user view and not from the database. Notifications informs users of critical events or issues with their account.
View Emory AWS Service Details
View Emory Site-to-Site VPN Connection Status
The status of the site-to-site VPN connection is visible from the Manage VPC view. To see the VPN connection status, click on the VPC Maintenance tab, select a VPC to manage and choose the View/Maintain VPC action. On this page one can view the details of the VPN connection as reported by the Emory Cisco ASR routers. These results are populated by sending a VPN connection query request to the Network Ops Service, which interrogates the routers to determine the overall status of the connection.
Site-to-Site VPN Connection
The site-to-site VPN connection allows for resiliant private connectivity to AWS VPCs via public transport i.e. Internet and Internet2. All VPCs, Type1 and 2 alike, are auto-provisioned with a pair of IPSEC VPN tunnels. One tunnel terminates in the 555 Asbury Cir (COX) facility and the other terminates in the 1100 White Street (WS) facility. Each tunnel is serviced by one of a redundant pair of Cisco ASR1K routers. Each router is licensed currently for 8Gbps of throughput which is shared across all VPNs for AWS and non-AWS use. Note that AWS limits their VGW bandwidth to 1.5Gbps which in turn limits the VPN tunnels since they are all serviced by the VGW. Also keep in mind that ALL AWS VPN connectivity rides over a single 10Gbps link to Internet2 which is shared by the enterprise. During normal operations, both IPSEC tunnels are up, but only one tunnel bears traffic at any given time.
Type1 VPCs utilize the IPSEC tunnels for ALL connectivity to-and-from the VPC. Type1's backhaul all traffic to the Emory network. This includes Type1 inter-VPC traffic. Note that this traffic does not traverse the Emory AWS firewall, so inter-VPC traffic is currently blocked via AWS NACLs. Type2 VPCs utilize the IPSEC tunnels ONLY for Emory traffic. Emory traffic is defined by any traffic to-or-from the VPC and the following CIDR prefixes:
All other traffic for Type2's is routed via the Firewall VPC and IGW resident there. This includes Type2 inter-VPC traffic, so all inter-VPC traffic traverses the firewall.
Monitoring Tunnel Status
AWS side provisioning and VPN tunnel status can be viewed in the Emory AWS console under the VPC. Click "VPN Connections" and you will see two VPNs: EmoryVpnConnection1 and EmoryVpnConnection2. Once provisioning is complete on the AWS side, both VPN Connections should show a State of "available". Select each VPN Connection one at a time and select the "Tunnel Details" tab at the bottom. Each VPN connection has TWO tunnels, but we are only using one tunnel out of each pair. So, a healthy VPN Connection will show one tunnel with a Status of "UP" and Details of "6 BGP Routes" and the other tunnel with a Status of "DOWN" and Details of "IPSEC IS DOWN". Each VPN Connection will have one tunnel UP and one tunnel DOWN for a total of two functioning tunnels.
Emory side provisioning and VPN tunnel status can be viewed in the Emory VPCP app. Note that if the AWS side shows two healthy VPN Connections with one tunnel UP in each, then the Emory side is guaranteed to be provisioned and UP as well.
Note that for the VPN tunnels to come up, automated provisioning must be completed on both the Emory side and the AWS side.
Reporting VPN Connection Problems
If one tunnel in each VPN Connection does not come up within 5 minutes of the completion of automated provisioning on BOTH sides, please open a Service Now incident or call the Emory Network Operations Center.
VPN performance between Emory and an AWS VPC has been benchmarked at 1.5Gbps. This was observed under optimal conditions between a single correctly sized AWS instance and a VM on the Emory network. This is the maximum throughput that can be expected. Actual performance may be significantly less depending on many factors. Some of these factors are specific to our environment
- 8Gbps VPN throughput of the ASR1K routers shared by entire enterprise
- 10Gbps Internet2 circuit shared by entire enterprise
- 1.5BGbps per VPC limit (imposed by AWS VGW bandwidth limit)
- General infrastructure utilization based on time of day
and some are more general
- Host capabilities to push data (CPU/NIC/etc.)
- Host network tuning i.e. MTU/MSS
- Application being used
AWS has a good article on troubleshooting poor VPN performance here
Please keep in mind that this is production infrastructure shared across the entire enterprise. Any tests intended to stress the network will impact the amount of resources available to the entire enterprise and not just the VPC or instance being tested. For this reason, please keep any throughput testing brief (ideally 1 minute tests) and off peak utilization times (6AM - 5PM). This will minimize the risk of a skewed test due to general high network utilization during the day. Additionally, this will ensure network resources are available to the enterprise during peak times.
If there is in fact a legitimate network related VPN performance issue, it will likely impact all VPN users. If you feel this to be the case, please open a Service Now incident or call the Emory Network Operations Center.
AWS Enterprise Support for Emory
All accounts provisioned as part of Emory Amazon Web Services will include AWS Enterprise Support. With AWS Enterprise Support, Customers will be able to open tickets directly with AWS for general questions and technical issues. One of the benefits of this service is tickets will be routed directly to an AWS Senior Cloud Support engineer for quicker analysis and better responses.
Accounts must be added to Emory’s Enterprise Support by Emory’s AWS support team. Emory’s account provisioning process will automatically email the account team to request Enterprise Support when a new account is created.
Support for Emory AWS Patterns and Practices
The LITS Research Solutions (RS) Team maintains Amazon Machine Images (AMIs) installed with common bioinformatics packages and data science tools such as R for use by the Emory Research Community. In addition, the RS team provides scripts and templates to enable researchers to perform common tasks in AWS, as well as free consultation on research solutions design and cost estimation. The RS Team and the Product Management Team also jointly conduct basic AWS trainings. For questions and inquires about these aids and trainings, please email email@example.com
Emory AWS Security
- RHEDcloudVpcFlowLogRolePolicy: The RHEDcloudVpcFlowLogRolePolicy is intended to allow the RHEDcloudVpcFlowLogRole the ability to write VPC Flow Logs to CloudWatch. Not attached to an assumable IAM Role for customer usage.
- RHEDcloudSecurityRiskDetectionServiceRolePolicy: The RHEDcloudSecurityRiskDetectionServiceRolePolicy allows security checker applications to run in this account with access required to implement the security checker functions. Not attached to an assumable IAM Role for customer usage.
- RHEDcloudAdministratorRoleHipaaPolicy: The RHEDcloudAdministratorRoleHipaaPolicy is intended to prevent customer administrators performing operations on specific resources that does not meet HIPAA requirements, which cannot be implemented as service control policies. Created if the AWS account is identified as HIPAA, attached to the assumable IAM Role "RHEDcloudAdministratorRole"
- RHEDcloudAdministratorRolePolicy: The RHEDcloudAdministratorRolePolicy is intended to prevent customer administrators performing operations on specific resources, which cannot be implemented as service control policies. This policy also restricts access to the account Cloud Trail logs and the bucket in which they are stored to allow customer administrators only to read these logs and not modify or delete them. Attached to the assumable IAM Role "RHEDcloudAdministratorRole".
- *-RHEDcloudManagementSubnetPolicy (Type 1 and Type 2 Links): The one or more (due to multiple VPCs in a single AWS account) ManagementSubnetPolicies is intended to prevent customer administrator role from deploying infrastructure in the management subnets in all VPCs. Attached to the assumable IAM Role "RHEDcloudAdministratorRole.
Links to repos that all IAM policies reside within: RS, Type 1, and Type 2
Service Control Policies
- rhedcloud-aws-org-standard-scp: The rhedcloud-aws-org-standard-scp SCP policy blocks AWS services that have not been released to the customer due to any reason. There are numerous reasons why an AWS service may reside on this list to be blocked. Some reasons to why an AWS service may be blocked may be due to: The AWS service has not been approved by Security, scheduled to be released in the future, may require assistance from Support/Security to setup and configure, etc.
- rhedcloud-aws-org-hipaa-scp: The rhedcloud-aws-org-hipaa-scp SCP policy blocks AWS services that are not HIPAA compliant by AWS. This SCP is only in effect on AWS accounts that are identified as HIPAA AWS accounts and will carry HIPAA data.
Security Risk Detector Service
The Security Risk Detector Service handles requests for SecurityRiskDetections that are requested by a scheduled application, a web application like the VPCP application, or triggered in response to an event. If a detector is configured for remediation, it executes the corresponding remediator for the detector. Detection events are stored in DynamoDB in AWS for short-term and long-term reference. Many trigger notifications which can be viewed within the VPCP application.
Security Risk Detectors
Security Risk Detectors (SRDs) implement a SecurityRiskDetection and return results. A comprehensive list of Security Risk Detectors is available in the Security Risk Detector Service Technical Design Document. Presently, when a detector runs it executes the check for the account in all regions in which Emory assets may be deployed for a service. Specifically, those regions are: ap-northeast-1, ap-northeast-2, ap-south-1, ap-southeast-1, ap-southeast-2, ca-central-1, cn-north-1, cn-northwest-1, eu-central-1, eu-west-1, eu-west-2, eu-west-3, sa-east-1, us-east-1, us-east-2, us-west-1, us-west-2, and us-gov-west-1.
Security Risk Detectors (SRRs) implement a remediation steps for detected security risks. A comprehensive list of Security Risk Remediators is available in the Security Risk Detector Service Technical Design Document.
Billing for Emory AWS Accounts
Viewing Bills in Emory AWS Accounts
Every month LITS will bill AWS expenses to the SpeedType associated with the AWS account through Compass. You may view that billing information directly in Compass, through a custom EBI report, or in the AWS console.
View Bills in Compass
- Log into Emory Compass at https://compass-login.emory.edu/ and select ‘Query Viewer’.
- Search for ‘EU_ALLTRANS_DETAIL’ and choose ‘Run to HTML’. You may also save this search to ‘Favorites’ future access.
- Enter the following: Business Unit (Required): ‘EMUNV’, Starting Department, Ending Department, Project ID, Fiscal Year (Required), Starting Period (Required), Ending Period (Required), Starting Account, and Ending Account.
- In the resulting query output, most of the fields are self-explanatory, but you will want to pay particular attention to the fields with these headings:
- Date - This is the date for the AWS Billing Cycle (should be the last day of the month)
- Ref - This is the AWS Account Number for your account.
- Amount (first Amount column) - Price for AWS service consumed.
- Ship To:
- Amount consumed
- Product consumed (ie EC2)
- Price of product consumed
- Multiplying amount x price should equal ‘Amount’
View Bills in EBI
EBI reports is a custom reporting tool available to administrators. Because of the custom nature, there is no template or typical style for these reports. For more information about EBI (including how to obtain access if you do not currently have it), please visit http://ebi.emory.edu/
AWS billing information can be found in the following fields:
- Journal Date – date for the AWS Billing Cycle.
- Journal Line Reference – AWS account Number for your account.
- Accounting Line Description – contains the following:
- Amount of service consumed
- Product consumed (i.e. EC2)
- Price of service consumed
- Multiplying amount x price could equal ‘Monetary Amount’.
- Monetary Amount – price for AWS consumed.
- There will be multiple line items for your charges based on your usage of AWS for the month.
View Bills in AWS
You can view six months of billing information from the AWS Console with the ability to print bills from the Bills page. There is line item detail available similar to Compass and EBI reporting. Note that the ‘Download CSV’ function will not work for the Emory consolidated account.
- To view billing information log into the Emory AWS Console at http://console.aws.emory.edu/ and navigate to ‘My Billing Dashboard’.
- Select the bill you would like to view.
Settling Emory AWS Bills
Your Emory AWS Service charges will be billed directly to your SpeedType in Compass.
As the user of the Emory AWS Service, you are responsible for keeping your SpeedType up to date. You also have full responsibility for the charges incurred on the account, including those created by those to whom you provide access (such as co-workers, co-investigators, lab assistants, etc.).
Integration with Emory Compass
Every month the AWS Bill Detector in each Emory AWS Service environment detects the presence of the master payer account bill CSV file in an S3 bucket, downloads the file, and executes a job to create the bill in the AWS Account Service. This persists the detail of each, monthly master payer account bill in the Emory AWS Account Metadata Store within the AWS Account Service and it also published an AWS Bill.Create-Sync message that is processed by the AWS Account Service to create the Compass flat-file feed. The AWS Account Service then delivers this feed to the designated Compass dropzone for scheduled processing to import the bill into Compass.
Closing Emory AWS Accounts
To request termination of an Emory AWS Account, select the account you administer in the VPCP application and select "Close Account." You will be presented with a dialog to create a request to LITS administrators to terminate your account. Closing the account is not immediate, but is a manual process in which LITS administrators will contact the account owners and confirm the desire to terminate the account and all activity in the account. This process is intended to ensure that no account is accidentally terminated and that the accountable account owner truly intends to terminate the account.
Note that after AWS accounts are terminated, they remain accessible from the console for a period of time until all charges are settled. AWS charges are billed and settled monthly in arrears, so whenever an account is terminated there will be pending charges to be settled at the beginning of the next month.
Initially LITS will not be deprovisioning accounts in either a manual or automated fashion, but just closing the account and quiescing the site-to-site VPN tunnel when account termination is requested. Deprovisinoing will occur after deprovisioning steps have been defined and automated, which is presently targeted for Q4 2018.
LITS Central Administration Practices
Maintaining CloudFormation Templates with Structures and IAM Policies
CloudFormation templates will be maintained by the LITS Systems team. The CloudFormation templates will kept in Bitbucket for version control. Changes to a CloudFormation template will invoke a "pipeline" where the new template will be deployed into a test account and evaluated with a series of test scripts. A CloudFormation template is ready for deployment when it has successfully completed a "pipeline" run.
The process for deploying CloudFormation template changes to existing accounts has not been decided. The two approaches that are being investigated are AWS CloudFormation Stack Sets and/or custom built management scripts. CloudFormation Stack Sets is a tool AWS provides for managing CloudFormation templates across many accounts. While this tool seems to do most of what we need, there are some downsides to using it. One of the downsides is its default mode is to update all accounts in the same window. While this works fine with one or two accounts, there are open questions for how long it will take to update larger amounts of accounts. When the automated provisioning process is working, we will test a Stack Set update across a more representative number of accounts (30-60). Then we will decide whether we can use CloudFormation Stack Sets or not.
Maintaining Service Control Policies
Similar to the CloudFormation templates, Service Control Polices will be maintained by the LITS Systems team, kept in BitBucket, and changes to the Service Control Policies will be deployed to a test account and evaluated with a series of scripts in an automated pipeline process. A new version of the Service Control Policy will be deployable when it successfully passes a pipeline run.
Service Control Policies changes will be deployed by uploading the new policy to the master account. The new policy will take effect immediately for accounts in the Organization Unit(s) where the new Service Control Policy is attached.
Adding, Removing, or Reconfiguring SRDs and SRRs
SRDs and SRRs are very specific Java components developed within our frameworks and run within Emory ESB/web services. To add, remove, or reconfigure SRDs and SRRs, configuration changes must be made to the config doc of the Security Risk Detector Service and the service must be restarted. This happens in a rolling fashion, so that there can always be instances of the Security Risk Detector Service running to handle the requests for detection.