已过SAA认证，贴出知识点总结

EC2

IAM

Applications that run on an EC2 instance must include AWS credentials in the AWS API requests. you can and should use an IAM role to manage temporary credentials for applications that run on an EC2 instance. the role supplies temporary permissions that applications can use when they make calls to other AWS resources. When you launch an EC2 instance, you specify an IAM role to associate with the instance. Applications that run on the instance can then use the role-supplied temporary credentials to sign API requests.

ec2iam.jpg

EC2 Hibernate

• Supported instance families - C3, C4, C5, M3, M4, M5, R3, R4, and R5.
• Instance RAM size - must be less than 150 GB.
• Instance size - not supported for bare metal instances.
• AMI: Amazon Linux 2, Linux AMI, Ubuntu & Windows…
• Root Volume: must be EBS, encrypted, not instance store, and large
• Available for On-Demand and Reserved Instances
• An instance cannot be hibernated more than 60 days
• retains its private IPv4 addresses and any IPv6 addresses and associated Elastic IP addresses.

When an EC2 instance is in the Hibernate state, you pay only for the EBS volumes and Elastic IP Addresses attached to it.

Purchasing Options

• On-Demand Instances: short workload, predictable pricing

• Reserved: (MINIMUM 1 year)

• Reserved Instances: long workloads

• Convertible Reserved Instances: long workloads with flexible instances

• Scheduled Reserved Instances: example – every Thursday between 3 and 6 pm

• Spot Instances: short workloads, cheap, can lose instances (less reliable)

• Dedicated Hosts: book an entire physical server, control instance placement --Allocated for your account for a 3-year period reservation

• Dedicated Instances: no other customers will share your hardware

EC2 Spot Instance Requests

• max spot price
  • current spot price < max
• Spot Block
• “block” spot instance during a specified time frame (1 to 6 hours) without interruptions

Spot Fleets

Strategy

• lowestPrice: from the pool with the lowest price (cost optimization, short workload)
• diversified: distributed across all pools (great for availability, long workloads)
• capacityOptimized: pool with the optimal capacity for the number of instances

Placement Groups

• Cluster—clusters instances into a low-latency group in a single Availability Zone

  • use case:
    • Big Data job that needs to complete fast
    • Application that needs extremely low latency and high network throughput

• Spread—spreads instances across underlying hardware (max 7 instances per group per AZ)

  • user case:
    • Critical Applications where each instance must be isolated from failure from each other
    • Application that needs to maximize high availability

• Partition—spreads instances across many different partitions (which rely on different sets of racks) within an AZ. Scales to 100s of EC2 instances per group (Hadoop, Cassandra, Kafka)

  • Up to 7 partitions per AZ
  • Can span across multiple AZs in the same region
  • use case:
    • HDFS, HBase, Cassandra,Kafka

EC2 Instance Storage

EBS Volume

• only be mounted to one instance at a time
• They are bound to a specific availability zone

EBS Volume Types

• gp2 / gp3 (SSD): General purpose SSD volume that balances price and performance for

  a wide variety of workloads

• io1 / io2 (SSD): Highest-performance SSD volume for mission-critical low-latency or

  high-throughput workloads

• st1 (HDD): Low cost HDD volume designed for frequently accessed, throughput-

  intensive workloads

• sc1 (HDD): Lowest cost HDD volume designed for less frequently accessed workloads

Only gp2/gp3 and io1/io2 can be used as boot volumes

• General Purpose SSD
  • System boot volumes, Virtual desktops, Development and test environments
• Provisioned IOPS (PIOPS) SSD
  • Great for databases workloads (sensitive to storage perf and consistency)
  • Supports EBS Multi-attach
• Hard Disk Drives (HDD)
  • Cannot be a boot volume
    • Throughput Optimized HDD (st1)
      • Big Data, Data Warehouses, Log Processing
      • Max throughput 500 MiB/s – max IOPS 500
    • Cold HDD (sc1)
      • Scenarios where lowest cost is important
      • For data that is infrequently accessed
      • Max throughput 250 MiB/s – max IOPS 250

EFS – Elastic File System

• Managed NFS (network file system) that can be mounted on many EC2
  • mount helper: To mount your Amazon EFS file system with the mount helper with encryption of data in transit enabled
• EFS works with EC2 instances in multi-AZ
• Uses security group to control access to EFS
• Compatible with Linux based AMI
• Only for Linux Instances POSIX file system (~Linux) that has a standard file API
• File system scales automatically, pay-per-use, no capacity planning

Performance

• EFS Scale
  • 1000s of concurrent NFS clients, 10 GB+ /s throughput
  • Grow to Petabyte-scale network file system, automatically
• Performance mode (set at EFS creation time)
  • General purpose (default): latency-sensitive use cases (web server, CMS, etc…)
  • Max I/O – higher latency, throughput, highly parallel (big data, media processing)
• Throughput mode
  • Bursting (1 TB = 50MiB/s + burst of up to 100MiB/s)
  • Provisioned: set your throughput regardless of storage size, ex: 1 GiB/s for 1 TB storage
• Storage Tiers (lifecycle management feature – move file after N days)
  • Standard: for frequently accessed files
  • Infrequent access (EFS-IA): cost to retrieve files, lower price to store

EC2 budgets

• Cost budgets – Plan how much you want to spend on a service.
• Usage budgets – Plan how much you want to use one or more services.
• RI utilization budgets – Define a utilization threshold and receive alerts when your RI usage falls below that threshold. This lets you see if your RIs are unused or under-utilized.
• RI coverage budgets – Define a coverage threshold and receive alerts when the number of your instance hours that are covered by RIs fall below that threshold. This lets you see how much of your instance usage is covered by a reservation.
• Savings Plans utilization budgets – Define a utilization threshold and receive alerts when the usage of your Savings Plans falls below that threshold. This lets you see if your Savings Plans are unused or under-utilized.
• Savings Plans coverage budgets – Define a coverage threshold and receive alerts when your Savings Plans eligible usage that is covered by Savings Plans fall below that threshold. This lets you see how much of your instance usage is covered by Savings Plans.

Load Balancer

Sticky Sessions

• Cookie Names:Don’t use AWSALB, AWSALBAPP, or AWSALBTG

SSL Certificates

• using ACM (AWS Certificate Manager)

• You can create upload your own certificates alternatively

• uses an X.509 certificate (SSL/TLS server certificate)

• HTTPS listener:

  • You must specify a default certificate

  SSL – Server Name Indication (SNI)

  loading multiple SSL certificates onto one web server (to serve multiple websites)

Connection Draining(CLB) & Deregistration Delay(ALB & NLB)

• Time to complete “in-flight requests” while the instance is de-registering or unhealthy
• Stops sending new requests to the EC2 instance which is de-registering
• Between 1 to 3600 seconds (default: 300 seconds)
• Can be disabled (set value to 0)

Auto Scaling Group(ASG)

• Scale out (add EC2 instances) to match an increased load
• Scale in (remove EC2 instances) to match a decreased load
• Ensure we have a minimum and a maximum number of machines running
• Automatically Register new instances to a load balancer

attributes

• A launch configuration or Launch Templates (newer)
  • Launch Configuration:
    • Must be re-created every time
  • Launch Templates
    • Can have multiple versions
    • Provision using both On-Demand and Spot instances (or a mix)
• Min Size / Max Size / Initial Capacity
• Network + Subnets Information
• Load Balancer Information
• Scaling Policies
  • Dynamic Scaling Policies
    • Target Tracking Scaling
      • like average ASG CPU to stay at around 40%
    • Simple / Step Scaling
      • When a CloudWatch alarm is triggered (example CPU > 70%), then add 2 units
      • When a CloudWatch alarm is triggered (example CPU < 30%), then remove 1
    • Scheduled Actions
      • Anticipate a scaling based on known usage patterns
      • Example: increase the min capacity to 10 at 5 pm on Fridays
  • Predictive Scaling

Scaling Cooldowns

After a scaling activity happens, you are in the cooldown period (default 300 seconds),During the cooldown period, the ASG will not launch or terminate additional instances

Lifecycle Hooks

You have the ability to perform extra steps before the instance goes in service (Pending state) You have the ability to perform some actions before

the instance is terminated (Terminating state)

ASG main point

• Scaling policies can be on CPU, Network… and can even be on custom metrics or based on a schedule (if you know your visitors patterns)
• ASGs use Launch configurations or Launch Templates (newer)
  • Launch Configuration (legacy):
    • Must be re-created every time
  • Launch Template (newer)
    • Can have multiple versions
    • Provision using both On-Demand and Spot instances (or a mix)
    • Create parameters subsets (partial configuration for re-use and inheritance) -- 继承参数
    • Can use T2 unlimited burst feature
• To update an ASG, you must provide a new launch configuration / launch template
• IAM roles attached to an ASG will get assigned to EC2 instances
• ASG are free.
• if they get terminated for whatever reason, the ASG will automatically create new ones as a replacement.
• ASG can terminate instances marked as unhealthy by an LB (and hence replace them)
• ASG tries the balance the number of instances across AZ by default
  • Find the AZ which has the most number of instances
  • If there are multiple instances in the AZ to choose from, delete the one with the oldest launch configuration

AWS RDS

• Postgres
• MySQL
• MariaDB
• Oracle
• Microsoft SQL Server
• Aurora

BackUp

• Backups are automatically enabled in RDS
  • Daily full backup of the database (during the maintenance window) in storage volume snapshot
  • Transaction logs are backed-up by RDS every 5 minutes to Amazon S3
• 7 days retention (can be increased to 35 days)

Storage Auto Scaling

• When RDS detects you are running out of free database storage, it scales automatically
• You have to set Maximum Storage Threshold

RDS Read Replicas

• Up to 5 Read Replicas (Within AZ, Cross AZ or Cross Region)
• Replication is ASYNC ,so reads are eventually consistent (异步并最终一致)
• Replicas can be promoted to their own DB
• Applications must update the connection string to leverage read replicas

RDS Multi AZ (Disaster Recovery)

• SYNC replication

• One DNS name – automatic app failover to standby

• Not used for scaling for Disaster Recovery （DR)

• Multi-AZ replication is free

• From Single-AZ to Multi-AZ

  • Zero downtime operation
  • Just click on “modify” for the database

RDS Security

Encryption

• At rest encryption
  • Encryption has to be defined at launch time
  • If the master is not encrypted, the read replicas t cannot be encrypted
  • Transparent Data Encryption (TDE) available for Oracle and SQL Server
• In-flight encryption
• SSL certificates to encrypt data to RDS in flight
• for MySQL/PostgreSQL

Network Security

RDS security works by leveraging security groups

Access Management

• IAM policies help control who can manage AWS RDS (through the RDS API)
• Traditional Username and Password can be used to n login into the database
• IAM-based authentication can be used to login into RDS MySQL & PostgreSQL (Auth token has a lifetime of 15 minutes)

RDS parameter

You manage your DB engine configuration through the use of parameters in a DB parameter group. DB parameter groups act as a container for engine configuration values that are applied to one or more DB instances.

If you want to use your own DB parameter group, you simply create a new DB parameter group, modify the desired parameters, and modify your DB instance to use the new DB parameter group.

Amazon Aurora

• HA (High Availability) native.
• Master + up to 15 Aurora Read Replicas serve reads( 16 nodes)
• automatically grows in increments of 10GB, up to 128 TB.
• Support for Cross Region Replication

Security

• authenticate using IAM token
• security groups
• Encryption in flight using SSL && Encryption at rest using KMS
• Automated backups, snapshots and replicas are also encrypted

Aurora Serverless

Good for infrequent,intermittent or unpredictable workloads

Multi-Master

Every node does R/W immediate failover for write node

Global Aurora

• 1 Primary Region (read / write)
• Up to 5 secondary (read-only) regions, replication lag is less than 1 second
• Up to 16 Read Replicas per secondary region
• Helps for decreasing latency
• Promoting another region (for disaster recovery) has an RTO of < 1 minute

Aurora Machine Learning

• Supported services
  • Amazon SageMaker (use with any ML model)
  • Amazon Comprehend (for sentiment analysis)

Amazon ElastiCache

Route 53

• A highly available, scalable, fully managed and Authoritative DNS(the customer can update the DNS records)
• Domain Registrar
• Ability to check the health of your resources
• The only AWS service which provides 100% availability Service Level Agreement（ SLA ）

Record Types

• A – maps a hostname to IPv4
• AAAA – maps a hostname to IPv6
• CNAME – maps a hostname to another hostname
  • The target is a domain name which must have an A or AAAA record
  • Can’t create a CNAME record for the top node of a DNS namespace （example.com） you can www.example.com
• NS – Name Servers for the Hosted Zone
  • Control how traffic is routed for a domain
  • Hosted Zone: A container for records that define how to route traffic to a domain and its subdomains
  • Public Hosted Zones
  • Private Hosted Zones
• Alias: Points a hostname to an AWS Resource
  • route to AWS Resources

Records TTL

• High TTL – – e.g., 24 hr
• Low TTL – – e.g., 60 sec.
• Except for Alias records, TTL is mandatory for each DNS record

Routing Policies

DNS does not route any traffic, it only responds to the DNS queries

• Simple
  • route traffic to a single resource
  • Can specify multiple values in the same record
  • If multiple values are returned, a random one is chosen by the client
  • When Alias enabled, specify only one AWS resource
• Weighted
  • load balancing between regions, testing new application versions…
  • Assign a weight of 0 to a record to stop sending traffic to a resource
  • If all records have weight of 0, then all records will be returned equally
• Latency-based
  • Redirect to the resource that has the least latency close to us
• Geolocation
  • This routing is based on user location
  • Should create a “Default” record
• Multi-Value Answer
  • Use when routing traffic to multiple resources
  • Route 53 return multiple values/resources
  • Up to 8 healthy records are returned for each Multi-Value query
  • Multi- - Value is not a substitute for having an ELB
• Geoproximity
  • Route traffic to your resources based on the geographic location of users and resources
  • Ability to shift more traffic to resources based on the defined bias
  • You must use Route 53 Traffic Flow to use this feature

Elastic Beanstalk VS CloudFormation

CloudFormation

AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources you want (like Amazon EC2 instances or Amazon RDS DB instances). AWS CloudFormation takes care of provisioning and configuring those resources for you.

Benefits

• Infrastructure as code
  • No resources are manually created, which is excellent for control
  • The code can be version controlled for example using git
  • Changes to the infrastructure are reviewed through code
• Cost
  • Each resources within the stack is tagged with an identifier so you can easily see how much a stack costs you
  • You can estimate(估算) the costs of your resources using the CloudFormation template
  • Savings strategy: In Dev, you could automation deletion of templates at 5 PM and recreated at 8 AM, safely
  • Separation of concern: create many stacks for many apps, and many layers.

How CloudFormation Works

• Templates have to be uploaded in S3 and then referenced in CloudFormation
• To update a template, we can’t edit previous ones. We have to re-upload a new version of the template to AWS
• Deleting a stack deletes every single artifact that was created by CloudFormation.

Templates

1. Resources: your AWS resources declared in the template (MANDATORY)
2. Parameters: the dynamic inputs for your template
3. Mappings: the static variables for your template
4. Outputs: References to what has been created
5. Conditionals: List of conditions to perform resource creation
6. Metadata

StackSets

so anytime you see in the exam something around deploying a CloudFormation stack globally or across accounts,think no more than just StackSets.

Elastic Beanstalk

Elastic Beanstalk is a PaaS layer on top of AWS's IaaS services which abstracts away the underlying EC2 instances, Elastic Load Balancers, Auto Scaling groups, host Docker containers,etc. This makes it a lot easier for developers, who don't want to deal with all the systems stuff, to get their application quickly deployed on AWS. With Elastic Beanstalk, you don't need to understand how any of the underlying magic works.

Elastic Beanstalk is intended to make developers' lives easier. CloudFormation is intended to make systems engineers' lives easier.

S3

• Amazon S3 allows people to store objects (files) in “buckets” (directories)
  • Object values are the content of the body
    • Max Object Size is 5TB (5000GB)
    • If uploading more than 5GB, must use “multi-part upload”,recommended for files > 100MB,
  • Metadata (list of text key / value pairs – system or user metadata)
  • Tags (Unicode key / value pair – up to 10) – useful for security / lifecycle
  • Version ID (if versioning is enabled)
• Buckets are defined at the region level
• Versioning
  • enabled at the bucket level
  • Protect against unintended deletes (ability to restore a version)
  • Any file that is not versioned prior to enabling versioning will have version “null”
  • Suspending versioning does not delete the previous versions

S3 Encryption for Objects

• server side
  • SSE-S3:
    • encryption using keys handled & managed by Amazon S3
    • Must set header: x-amz-erver-side-encryption:AES256
  • SSE-KMS:
    • encryption using keys handled & managed by KMS
    • Must set header: x-amz-erver-side-encryption:aws:kms
  • SSE-C:
  • server-side encryption using data keys fully managed by the customer outside of AWS
  • HTTPS must be used
  • Amazon S3 does not store the encryption key you provide
• Client Side

S3 Security

• User based
  • IAM policies - which API calls should be allowed for a specific user from IAM console
  • MFA Delete: MFA (multi factor authentication) can be required in versioned buckets to delete objects
  • Pre-Signed URLs: URLs that are valid only for a limited time (ex: premium video service for logged in users)
• Resource Based
  • Bucket Policies - bucket wide rules from the S3 console - allows cross account
  • Object Access Control List (ACL) – finer grain
  • Bucket Access Control List (ACL) – less common

S3 Storage Classes

• S3 Standard –General Purpose
  • Use Cases: Big Data analytics, mobile & gaming applications, content distribution…
• S3 Standard –Infrequent Access (IA)
  • Use Cases: As a data store for disaster recovery, backups…
• S3 One Zone - Infrequent Access (IA)
  • Use Cases: Storing secondary backup copies of on-premise data, or storing data you can recreate
• S3 Intelligent Tiering
  • Automatically moves objects between two access tiers based on changing access patterns
• Amazon Glacier
  • Expedited (1 to 5 minutes)
  • Standard (3 to 5 hours)
  • Bulk (5 to 12 hours)
  • Minimum storage duration of 90 days
• Amazon Glacier Deep Archive
  • Standard (12 hours)
  • Bulk (48 hours)
  • Minimum storage duration of 180 days

S3 Lifecycle Rules

Rules can be created for a certain prefix (ex - s3://mybucket/mp3/*)

Rules can be created for certain objects tags (ex - Department: Finance)

• Transition actions : It defines when objects are transitioned to another storage
  class.
  • Move objects to Standard IA class 60 days after creation
  • Move to Glacier for archiving after 6 months
• Expiration actions: configure objects to expire (delete) after some time
  • Access log files can be set to delete after a 365 days
  • Can be used to delete old versions of files (if versioning is enabled)
  • Can be used to delete incomplete multi-part uploads

S3 Analytics – Storage Class Analysis

• You can setup S3 Analytics to help determine when to transition objects from Standard to Standard_IA
• Does not work for ONEZONE_IA or GLACIER

S3 Transfer Acceleration

Increase transfer(upload and download) speed by transferring file to an AWS edge location which will forward the data to the S3 bucket in the target region

S3 Byte-Range Fetches

• Parallelize GETs by requesting specific byte ranges
• Better resilience in case of failures
• Can be used to speed up downloads
• Can be used to retrieve only partial data (for example the head of a file)

S3 Select & Glacier Select

• Retrieve less data using SQL by performing server side filtering
• Can filter by rows & columns (simple SQL statements)
• Less network transfer, less CPU cost client-side
• CSV, JSON, or Apache Parquet format. GZIP & BZIP2 compression is supported with CSV or JSON format with server-side encryption.

S3 – Requester Pays

With Requester Pays buckets , the requester instead of the bucket owner pays the cost of the request and the data download from the bucket

The requester must be authenticated in AWS (cannot be anonymous)

Amazon Athena

• Serverless query service to t perform analytics against S3 objects
• Uses standard SQL language to query the files
• Supports CSV, JSON, ORC, Avro, and Parquet (built on Presto)
• Use compressed or columnar data for cost-savings (less scan)

Exam Tip: analyze data in S3 using serverless SQL, use Athena

Lock

• Adopt a WORM (Write Once Read Many) model
  • Glacier Vault Lock:Lock the policy for future edits (can no longer be changed)
  • S3 Object Lock (versioning must be enabled)
    • Block an object version deletion for a specified amount of time
      • Retention Period : specifies a fixed period
      • Legal Hold: same protection, no expiry date
    • Governance mode:users can't overwrite or delete an object version or alter its lock
      settings unless they have special permissions
    • Compliance mode : a protected object version can't be overwritten or deleted by
      any user, including the root user in your AWS account. When an object is locked in
      compliance mode, its retention mode can't be changed, and its retention period
      can't be shortened.

AWS CloudFront

• Content Delivery Network (CDN)：Improves read performance, content is cached at the edge
• DDoS protection, integration with Shield, AWS Web Application Firewall
• Can expose external HTTPS and can talk to internal HTTPS backends

CloudFront – Origins

• S3 bucket
  • Enhanced security with CloudFront Origin Access Identity (OAI)
  • For distributing files and caching them at the edge
  • CloudFront can be used as an ingress (to upload files to S3)
• Custom Origin (HTTP)
  • Application Load Balancer(Must be Public)
  • EC2 instance（Must be Public）
  • Any HTTP backend you want
  • S3 website (must first enable the bucket as a static S3 website)

CloudFront Geo Restriction

• using a 3rd party Geo-IP database
  • Whitelist
  • Blacklist

CloudFront vs S3 Cross Region Replication

• CloudFront
  • Global Edge network
  • Files are cached for a TTL (maybe a day)
  • Great for static content that must be available everywhere
• S3 Cross Region Replication
  • Must be setup for each region you want replication to happen
  • Files are updated in near real-time
  • Great for dynamic content that needs to be available at low-latency in few regions

CloudFront Signed URL / Signed Cookies

• Signed URL = access to individual files (one signed URL per file)

• Signed Cookies = access to multiple files (one signed cookie for many files)

CloudFront Signed URL vs S3 Pre-Signed URL

• CloudFront Signed URL
  • Allow access to a path, no matter the origin
  • Account wide key-pair, only the root can manage it
  • Can filter by IP, path, date, expiration
  • Can leverage caching features
• S3 Pre-Signed URL
  • Issue a request as the person who pre-signed the URL
  • Uses the IAM key of the signing IAM principal

AWS Global Accelerator

• 2 Anycast IP are created for your application

AWS Snow Family

AWS OpsHub

use AWS OpsHub (a software you install on your computer / laptop) to manage your Snow Family Device

Snowball into Glacier

• Snowball cannot import to Glacier directly
• You must use Amazon S3 first, in combination with an S3 lifecycle policy

Amazon FSx

• Amazon FSx for Windows
  • a fully managed Windows file system share drive
  • Supports SMB protocol & Windows NTFS
  • up to 10s of GB/s, millions of IOPS, 100s PB of data
  • supports access across VPCs, accounts, and Regions via Direct Connect or VPN (on-premises) and VPC Peering or AWS Transit Gateway.
• Amazon FSx for Lustre(Linux)
  • up to 100s GB/s, millions of IOPS, sub-ms latencies
  • Seamless integration with S3
• Scratch File System(Temporary storage)
• Persistent File System(Long-term storage)

AWS Storage Gateway

• Bridge between on-premises data and cloud data in S3
• File Gateway
  • using the NFS and SMB protocol
  • Supports S3 standard, S3 IA, S3 One Zone IA
  • Most recently used data is cached in the file gateway
  • Bucket access using IAM roles for each File Gateway
  • Can be mounted on many servers
  • Integrated with Active Directory (AD) for user authentication
• Volume Gateway
  • Block storage using iSCSI protocol backed by S3
  • Backed by EBS snapshots which can help restore on-premises volumes
  • Cached volumes：low latency access to most recent data
  • Stored volumes： entire dataset is on premise, scheduled backups to S3
• Tape Gateway
  • Back up data using existing tape-based processes (and iSCSI interface)
• Storage Gateway – Hardware appliance
  • Works with File Gateway, Volume Gateway, Tape Gateway
• Amazon FSx File Gateway
  • Native access to Amazon FSx for Windows File Server
  • Local cache for frequently accessed data

AWS Transfer Family

• A fully-managed service for file transfers into and out of Amazon S3 or Amazon EFS using the FTP protocol
• ntegrate with existing authentication systems (Microsoft Active Directory, LDAP, Okta, Amazon Cognito, custom)

Amazon SQS

• Unlimited throughput, unlimited number of messages in queue
• Default retention of messages: 4 days, maximum of 14 days
• Limitation of 256KB per message sent
• Poll SQS for messages (receive up to 10 messages at a time)
• CloudWatch Metric – Queue Length --- ApproximateNumberOfMessages
• Message Visibility Timeout
• Dead Letter Queue--Useful for debugging!
• Delay Queue--up to 15 minutes
• Long Polling--LongPolling decreases the number of API calls made to SQS while increasing the efficiency and latency of your application.
• SQS Temporary Queue Client
• FIFO Queue
  • Limited throughput: 300 msg/s without batching, 3000 msg/s with

Security

• Encryption
  • In-flight encryption using HTTPS API
  • At-rest encryption using KMS keys
  • Client-side encryption if the client wants to perform encryption/decryption itself
• Access Controls;IAM policies to regulate access to the SQS API
• SQS Access Policies
  • Useful for cross-account access to SQS queues
  • Useful for allowing other services (SNS, S3…) to write to an SQS queue

Amazon SNS

• Each subscriber to the topic will get all the messages
• Up to 10,000,000 subscriptions per topic
• 100,000 topics limit
• FIFO
  • Can only have SQS FIFO queues as subscribers
  • Limited throughput (same throughput as SQS FIFO)
• Message Filtering
• Security same as SQS

Kinesis

• collect, process, and analyze streaming data in real-time such as Application logs, Metrics, Website clickstreams, IoT telemetry data…

Kinesis Data Streams

• Billing is per shard provisioned, can have as many shards as you want
• Retention between 1 day (default) to 365 days
• Ability to reprocess (replay) data
• Once data is inserted in Kinesis, it can’t be deleted (immutability) (不变性)
• Data that shares the same partition goes to the same shard (ordering)
• Consumers
  • Write your own: Kinesis Client Library (KCL), AWS SDK
  • Managed: AWS Lambda, Kinesis Data Firehose, Kinesis Data Analytics

Kinesis Data Firehose

• Supports many data formats, conversions, transformations, compression
• Supports custom data transformations using AWS Lambda
• Can send failed or all data to a backup S3 bucket
• Consumers
  • AWS: Redshift / Amazon S3 / ElasticSearch
    • Amazon Kinesis Data Firehose cannot directly send data logs to Amazon Redshift but needs to first store in the Amazon S3 bucket & then it copies data to Amazon Redshift.
  • 3rd party partner: Splunk / MongoDB / DataDog / NewRelic / …
  • Custom: send to any HTTP endpoint

Kinesis Data Analytics (SQL application)

• Perform real-time analytics on Kinesis Streams using SQL

Amazon MQ

When migrating to the cloud, instead of re-engineering the application to use SQS and SNS, we can use Amazon MQ

Container

ECS

• Launch Docker containers on AWS
• You must provision & maintain the infrastructure (the EC2 instances)
• ECS agent
  • Makes API calls to ECS service
  • Reference sensitive data in Secrets Manager or SSM Parameter Store
  • Send container logs to CloudWatch Logs
• Task Role
• Allow each task to have a specific role
• Use different roles for the different ECS Services you run
• ECS Data Volumes – EFS File Systems

Fargate

• Launch Docker containers on AWS
• Serverless offering -- You do not provision the infrastructure
• ECS agent
  • Makes API calls to ECS service
  • Reference sensitive data in Secrets Manager or SSM Parameter Store
  • Send container logs to CloudWatch Logs
• Task Role
• Allow each task to have a specific role
• Use different roles for the different ECS Services you run

EKS

AWS K8S (ECS and Fargate)

Lambda

• per region
• Maximum execution time: 900 seconds (15 minutes)
• Environment variables (4 KB)
• Concurrency executions: 1000 (can be increased)
• Memory allocation: 128 MB – 10GB (1 MB increments)
• Disk capacity in the “function container” (in /tmp): 512 MB

DynamoDB

• Maximum size of an item is 400KB
• DynamoDB Accelerator
  • Help solve read congestion by caching
  • Doesn’t require application logic modification (compatible with existing DynamoDB APIs)
  • 5 minutes TTL for cache (default)
• DynamoDB Streams(Ordered stream)
  • Sent to Kinesis Data Streams
  • Read by AWS Lambda
  • Read by Kinesis Client Library applications
  • Data Retention for up to 24 hours
• DynamoDB Global Tables
• Time To Live (TTL):Automatically delete items after an expiry timestamp

Lambda@Edge

• wanted to run a global AWS Lambda alongside
• implement request filtering before reaching your application
• Use Cases
  • Website Security and Privacy
  • Dynamic Web Application at the Edge
  • Search Engine Optimization
  • Intelligently Route Across Origins and Data Centers
  • Bot Mitigation at the Edge
  • Real-time Image Transformation
  • A/B Testing
  • User Authentication and Authorization
  • User Prioritization
  • User Tracking and Analytics

AWS API Gateway

• AWS Lambda + API Gateway
• Security: Lambda Authorizer

AWS SAM - Serverless Application Model

Framework for developing and deploying serverless applications

Redshift

• Redshift is based on PostgreSQL
• Redshift Enhanced VPC Routing: COPY / UNLOAD goes through VPC
  • no need a cluster security group
• Data is loaded from S3, DynamoDB, DMS, other DBs…
• Redshift has no “Multi-AZ” mode You can restore a snapshot into a new cluster
• To enable access to the cluster from SQL client tools via JDBC or ODBC
• Redshift Spectrum
  • Must have a Redshift cluster available to start the query
  • Query data that is already in S3 without loading it

AWS Glue

• Fully serverless service Managed extract, transform, and load (ETL) service
• Glue Data Catalog: catalog of datasets

Neptune

Fully managed graph database

AWS CloudWatch

• Up to 10 dimensions per metric (最多十个维度)
• EC2 Detailed monitoring
  • EC2 instance metrics have metrics “every 5 minutes” With detailed monitoring (for a cost), you get data “every 1 minute”
• Custom Metrics：send your own custom metrics to CloudWatch
  • Metric resolution
    • Standard: 1 minute (60 seconds)
    • High Resolution: 1/5/10/30 second(s) – Higher cost
• CloudWatch Dashboards
  • Dashboards are global
  • Dashboards can include graphs from different AWS accounts and regions

CloudWatch Logs

• Metric Filter ：can use filter expressions For example, find a specific IP inside of a log Or count occurrences of “ERROR” in your logs
• Metric filters can be used to trigger CloudWatch alarms
• CloudWatch Logs Insights
• CloudWatch Logs Agent： For virtual servers (EC2 instances, on-premise servers…) send logs to CloudWatch Logs
  • CloudWatch Logs Agent
  • CloudWatch Unified Agent
    • Centralized configuration using SSM Parameter Store
    • Collect additional system-level metrics such as RAM, processes, etc…

CloudWatch Alarms

• Alarms are used to trigger notifications for any metric
• Period
  • Length of time in seconds to evaluate the metric
  • High resolution custom metrics: 10 sec, 30 sec or multiples of 60 sec
• Targets
  • EC2
  • ECS Auto Scaling
  • Amazon SNS

CloudWatch Events

• Event Pattern: Intercept events from AWS services (Sources) -- Example sources: EC2 Instance Start, CodeBuild Failure, S3, Trusted Advisor
• Schedule or Cron

Amazon EventBridge

• Default event bus: generated by AWS services (CloudWatch Events)

• Partner event bus: receive events from SaaS service or applications (Zendesk, DataDog, Segment, Auth0…)

• Custom Event buses: for your own applications

• Schema Registry (versioned)

AWS CloudTrail

• Provides governance, compliance and audit for your AWS Account
• A trail can be applied to All Regions (default) or a single Region.
• CloudTrail Insights
  • Enable CloudTrail Insights to detect unusual activity in your account
  • continuously analyzes write events to detect unusual patterns
• CloudTrail Events
  • Management Events:Operations that are performed on resources in your AWS account
  • Data Events: By default, data events are not logged can activity Amazon S3 object-level
  • CloudTrail Insights Events
  • Events are stored for 90 days in CloudTrail

AWS Config

• auditing and recording compliance of your AWS resources
• record configurations and changes over time
• AWS Config is a per-region service,Can be aggregated across regions and accounts
• Rules
  • custom config rules (must be defined in AWS Lambda)
  • evaluate if each EBS disk is of type gp2
  • evaluate if each EC2 instance is t2.micro
  • Rules can be evaluated / triggered
    • For each config change
  • AWS Config Rules does not prevent actions from happening (no deny)
  • Remediations(补救措施)
    • Automate remediation of non-compliant resources using SSM Automation Documents
    • Use AWS-Managed Automation Documents or create custom Automation Documents
  • Notifications
    • Use EventBridge to trigger notifications
      • ex:resources are non-compliant,configuration changes,compliance state

AWS STS – Security Token Service

• Allows to grant limited and temporary access to AWS resources.
• AssumeRole
• AssumeRoleWithSAML
• AssumeRoleWithWebIdentity （网络身份）
• GetSessionToken

AWS Directory Services

• AWS Managed Microsoft AD
  • Create your own AD in AWS, manage users locally, supports MFA
  • Establish “trust” connections with your on-premise AD
• AD Connector
  • Directory Gateway (proxy) to redirect to on-premise AD
  • Users are managed on the on-premise AD
• Simple AD
  • AD-compatible managed directory on AWS
  • Cannot be joined with on-premise AD

AWS Organizations

• Global service
• Allows to manage multiple AWS accounts
• Member accounts can only be part of one organization
• sharing resources
  • enable sharing with AWS Organizations
  • if disable sharing with AWS Organizations
    • resources are shared within accounts as external accounts & an invitation needs to be accepted between these accounts to start resource sharing.

Service Control Policies (SCP)

• SCP is applied to all the Users and Roles of the Account, including Root user
• SCP must have an explicit Allow

IAM Permission Boundaries

• IAM Permission Boundaries are supported for users and roles
• Advanced feature to use a managed policy to set the maximum permissions an IAM entity can get.

AWS Resource Access Manager (RAM)

• Share AWS resources that you own with other AWS accounts

AWS Single Sign-On (SSO)

• Centrally manage Single Sign-On to access multiple accounts and 3rd -party business applications.
• Integrated with AWS Organizations
• Supports SAML 2.0 markup
• Integration with on-premise Active Directory
• Permission sets can control the time duration for user login to the AWS Console by setting session duration. The Default Session duration is 1 hour, while the maximum can be set to 12 hours.

AWS KMS (Key Management Service)

• KMS can only help in encrypting up to 4KB of data per call
• If data > 4 KB, use envelope encryption
• Automatic Key Rotation
  • For Customer- - managed CMK (not AWS managed CMK)
  • If enabled: automatic key rotation happens every 1 year
• KMS Alias：Better to use aliases in this case (to hide the change of key for the application)

SSM Parameter Store

• Secure storage for configuration and secrets
• Parameters Policies
  • Allow to assign a TTL to a parameter (expiration date) to force updating or deleting sensitive data such as passwords
  • Can assign multiple policies at a time

AWS Secrets Manager

• Newer service, meant for storing secrets
• Capability to force rotation of secrets every X days
• Integration with Amazon RDS (MySQL, PostgreSQL, Aurora)
• Automate generation of secrets on rotation
• Mostly meant for RDS integration

AWS Shield

• AWS Shield Standard
  • Provides protection from attacks such as SYN/UDP Floods, Reflection attacks and other layer 3/layer 4 attacks
• AWS Shield Advanced
  • Optional DDoS mitigation service
  • Protect against more sophisticated

CloudHSM

• CloudHSM => AWS provisions encryption hardware
• Redshift supports CloudHSM for database encryption and key management
• Good option to use with SSE-C encryption
• CloudHSM clusters are spread across Multi AZ (HA)

AWS WAF –Web Application Firewall

• Protects your web applications from common web exploits (Layer 7)
• Deploy on Application Load Balancer , API Gateway, CloudFront

Amazon GuardDuty

• Intelligent Threat discovery to Protect AWS Account
• Input data includes
  • CloudTrail Logs: unusual API calls, unauthorized deployments
  • VPC Flow Logs: unusual internal traffic, unusual IP address
  • DNS Logs: compromised EC2 instances sending encoded data within DNS queries

Amazon Inspector

• Automated Security Assessments for EC2 instances （EC2安全化评估）
• AWS Inspector Agent must be installed on OS in EC2 instances
• Analyze the running OS against known vulnerabilities
• Analyze against unintended network accessibility

Amazon Macie

• uses machine learning and pattern matching to discover and protect your sensitive data in AWS.

CICD

• push the code to a code repository often (CodeCommit -- GitHub)
• A testing / build server checks the code as soon as it’s pushed(CodeBuild -- Jenkins CI)

AWS Step Functions VS AWS SWF – Simple Workflow Service

Amazon EMR

• EMR stands for “Elastic MapReduce”
• EMR helps creating Hadoop clusters (Big Data) to analyze and process vast amount of data
• Auto-scaling and integrated with Spot instances
• Also supports Apache Spark, HBase, Presto, Flink…
• The clusters can be made of hundreds of EC2 instances

AWS Opsworks

• It’s an alternative to AWS SSM
• Chef & Puppet needed => AWS Opsworks

AWS WorkSpaces

• Managed, Secure Cloud Desktop
• Great to eliminate management of on-premise VDI (Virtual Desktop Infrastructure)

AWS AppSync

• Store and sync data across mobile and web apps in real-time
• Integrations with DynamoDB / Lambda
• Makes use of GraphQL

Cost Explorer

• Visualize, understand, and manage your AWS costs and usage over time
• Forecast usage up to 12 months based on previous usage
• Choose an optimal Savings Plan

DMS – Database Migration Service

Quickly and securely migrate databases to AWS, resilient, self healing

AWS Schema Conversion Tool (SCT)

Convert your Database’s Schema from one engine to another

AWS Server Migration Service (SMS)

Incremental replication of on-premise live servers to AWS

AWS DataSync

• Move large amount of data from on-premise to AWS --Amazon S3 (any storage classes – – including Glacier） , Amazon EFS , Amazon x FSx for Windows
• Move data from your NAS or file system via NFS or SMB

QuickSight

sight.jpg

Amazon QuickSight is a business analytics service you can use to build visualizations, charts, perform ad hoc analysis, and get business insights from your data. It can automatically discover AWS data sources and also works with your data sources.

AWS Backup

• Centrally manage and automate backups across AWS services
• Supports cross-region backups
• Supports cross-account backups
• Supports PITR for supported services
• On-Demand and Scheduled backups
• You create backup policies known as Backup Plans
  • Backup frequency (every 12 hours, daily, weekly, monthly, cron expression)
  • Backup window
  • Transition to Cold Storage (Never, Days, Weeks, Months, Years)
  • Retention Period (Always, Days, Weeks, Months, Years)

Amazon Data Lifecycle Manager

Amazon Data Lifecycle Manager can be used for creation, retention & deletion of EBS snapshots. It protects critical data by initiating backup of Amazon EBS volumes at selected intervals, along with storing & deletion of old snapshots to save storage space & cost.

Disaster Recovery

• Backup and Restore
  • Backup and restore is a suitable approach for mitigating against data loss or corruption.This approach can also be used to mitigate against a regional disaster by replicating data to other AWS Regions, or to mitigate lack of redundancy for workloads deployed to a single Availability Zone.
• Pilot Light
  • With the pilot light approach, you replicate your data from one Region to another and provision a copy of your core workload infrastructure. Resources required to support data replication and backup, such as databases and object storage, are always on. A pilot light approach minimizes the ongoing cost of disaster recovery by minimizing the active resources, and simplifies recovery at the time of a disaster because the core infrastructure requirements are all in place.
• Warm Standby
  • The warm standby approach involves ensuring that there is a scaled down, but fully functional, copy of your production environment in another Region.
• Hot Site / Multi Site Approach
  • You can run your workload simultaneously in multiple Regions as part of a multi-site active/active or hot standby active/passive* strategy. Multi-site active/active serves traffic from all regions to which it is deployed, whereas hot standby serves traffic only from a single region, and the other Region(s) are only used for disaster recovery. With a multi-site active/active approach, users are able to access your workload in any of the Regions in which it is deployed.

Elastic Network Adapter (ENA)

• EC2 Enhanced Networking
• Higher bandwidth, higher PPS (packet per second), lower latency

Elastic Fabric Adapter (EFA)

• Improved ENA for HPC, only works for Linux
• Great for inter-node communications, tightly coupled workloads

VPC

IPV4

• Max. CIDR per VPC is 5
  • Min. size is /28 (16 IP addresses)
  • Max. size is /16 (65536 IP addresses)
• You can have multiple VPCs in an AWS region (max. 5 per region – soft limit)
• VPC – Subnet
  • AWS reserves 5 IP addresses (first 4 & last 1)
    • Example: if CIDR block 10.0.0.0/24, then reserved IP addresses are:
      • 10.0.0.0 – Network Address
      • 10.0.0.1 – reserved by AWS for the VPC router
      • 10.0.0.2 – reserved by AWS for mapping to Amazon-provided DNS
      • 10.0.0.3 – reserved by AWS for future use
      • 10.0.0.255 – Network Broadcast Address. AWS does not support broadcast in a VPC, therefore the address is reserved

Internet Gateway (IGW)

• One VPC can only be attached to one IGW and vice versa
• Internet Gateways on their own do not allow Internet access…Route tables must also be edited!
• It scales horizontally and is highly available and redundant

NAT Instance

• Allows EC2 instances in private subnets to connect to the Internet
• Must be launched in a public subnet
• Must have Elastic IP attached to it
• Not highly available / resilient

NAT Gateway

• NATGW is created in a specific Availability Zone, uses an Elastic IP
• AWS-managed NAT, higher bandwidth, high availability, no administration
• NAT Gateway is resilient within a single Availability Zone
• Must create multiple NAT Gateways in multiple AZs for fault-tolerance

Bastion Hosts

• The bastion is in the public subnet which is then connected to all other private subnets
• Bastion Host security group must be tightened

DNS Resolution in VPC

• DNS Resolution (enableDnsSupport)

  • Decides if DNS resolution from Route 53 Resolver server is supported for the VPC

• DNS Hostnames (enableDnsHostnames)

  • By default,

    • True => default VPC
    • False => newly created VPCs

  • Won’t do anything unless enableDnsSupport=true

  • If True, assigns public hostname to EC2 instance if it has a public IPv4

VPC_DNSenable.png

VPC Security Groups & NACLs

Reachability Analyzer

troubleshoots network connectivity between two endpoints in your VPC(s)

VPC Peering

• Privately connect two VPCs using AWS network(different AWS accounts/regions)
• You must update route tables in each VPC’s subnets to ensure EC2 instances can communicate with each other
• You can reference a security group in a peered VPC(Account ID in same region)

VPC Endpoints

• allows you to connect to AWS services using a private network instead of using the public Internet
• Interface Endpoints
  • Supports most AWS services Provisions an ENI (private IP address) as an entry point (must attach a Security Group)
• Gateway Endpoints
  • Provisions a gateway and must be used as a target in a route table
  • Supports both S3 and DynamoDB

VPC Flow Logs

Query VPC flow logs using Athena on S3 or CloudWatch Logs Insights

AWS Site-to-Site VPN

• Virtual Private Gateway (VGW)

  • VPN concentrator on the AWS side of the VPN connection

  • connection has two tunnels, with each tunnel using a unique virtual private gateway public IP address

vgateway.jpg

• Important step: enable Route Propagationn for the Virtual Private Gateway in the route table that is associated with your subnets

• Customer Gateway (CGW)

  • Software application or physical device on customer side of the VPN connection

AWS VPN CloudHub

• Create a single virtual private gateway.
• Create multiple customer gateways, each with the public IP address of the gateway. You must use a unique Border Gateway Protocol (BGP) Autonomous System Number (ASN) for each customer gateway.
• Create a dynamically routed Site-to-Site VPN connection from each customer gateway to the common virtual private gateway.

Direct Connect (DX)

• need to setup a Virtual Private Gateway on your VPC
• Access public resources (S3) and private (EC2) on same connection
• Supports both IPv4 and IPv6
• Direct Connect Gateway
  • If you want to setup a Direct Connect to one or more VPC in many different regions (same account), you must use a Direct Connect Gateway
• Connection Types
  • Dedicated Connections
  • Hosted Connections
• longer than 1 month to establish a new connection
• Encryption
  • AWS Direct Connect + VPN provides an IPsec-encrypted private connection

AWS PrivateLink (VPC Endpoint Services)

• Requires a network load balancer (Service VPC) and ENI (Customer VPC) or GWLB
• If the NLB is in multiple AZ, and the ENIs in multiple AZ, the solution is fault tolerant!
• Most secure & scalable way to expose a service to 1000s of VPC (own or other accounts)

Transit Gateway

• For having transitive peering between thousands of VPC and on-premises, hub-and-spoke (star) connection
• Regional resource, can work cross-region
• Share cross-account using Resource Access Manager (RAM)
• Site-to-Site VPN ECMP
  • Use case: create multiple Site-to-Site VPN connections to increase the bandwidth of your connection to AWS

IPV6

• You can enable IPv6 (they’re public IP addresses)
• Your EC2 instances will get at least a private internal IPv4 and a public IPv6
• They can communicate using either IPv4 or IPv6 to the internet through an Internet Gateway

Egress-only Internet Gateway

• Used for IPv6 only (NAT Gateway for Ipv4)

DR

dr.jpg

Compute Savings Plans && EC2 Instance Savings Plans

1. Compute Savings Plans 的灵活性最高，最高可帮助您节省 66% 的费用。这些计划会自动应用于 EC2 实例用量，不分实例系列、大小、可用区、区域、操作系统或租期，并且还适用于 Fargate 和 Lambda。例如，注册 Compute Savings Plans 后，您可以随时从 C4 实例更改为 M5 实例，将工作负载从欧洲（爱尔兰）区域转移到欧洲（伦敦）区域，或者将工作负载从 EC2 迁移到 Fargate 或 Lambda，并继续自动支付 Savings Plans 价格。
2. EC2 Instance Savings Plans 可提供最低的价格，最高可享受 72% 的折扣，以换取在单个区域内使用单个实例系列的承诺（例如在弗吉尼亚北部区域使用 M5 实例）。这会自动降低您在该区域的选定实例系列成本，不分可用区、实例大小、操作系统或租期。借助 EC2 Instance Savings Plans，您可以灵活地在该区域的一个实例系列中更改实例的使用情况。例如，您可以从运行 Windows 的 c5.xlarge 实例迁移到运行 Linux 的 c5.2xlarge 实例，并自动享受 Savings Plans 价格。