我的 Amazon Elastic Container Service (Amazon ECS) 學習筆記

Published:
  Lastmod: 2023-01-17
by Ernest Chiang

(圖說：巴西第一大港，Port of Santos，提供裝卸貨櫃服務。圖片來源: Photo by sergio souza on Unsplash。)

概覽摘要 Briefing

(Command/Ctrl + 點圖放大)

歷史考古 History

古往今來，縱橫脈絡。

名詞定義 Terms

這裡整理有出場的名詞清單，並註記原文全名、名詞定義與來源。

  • Amazon ECS:
    • Amazon Elastic Container Service is a highly scalable, fast, container management service that makes it easy to run, stop, and manage Docker containers on a cluster.
  • Amazon ECS Launch Types:
  • Features of Amazon ECS
    • Containers:
      • To deploy applications on Amazon ECS, your application components must be architected to run in containers. For more information about container technology, see Docker Basics for Amazon ECS.
    • Container Images:
      • Containers are created from a read-only template called an image.
    • Task Definitions:
      • A blueprint for your application. To prepare your application to run on Amazon ECS, you create a task definition. The task definition is a text file, in JSON format, that describes one or more containers, up to a maximum of ten, that form your application.
    • Tasks:
      • A task is the instantiation of a task definition within a cluster.
    • Task Scheduling:
      • The Amazon ECS task scheduler is responsible for placing tasks within your cluster. There are several different scheduling options available. For example, you can define a service that runs and maintains a specified number of tasks simultaneously.
    • Clusters:
      • When you run tasks using Amazon ECS, you place them on a cluster, which is a logical grouping of resources.
    • Amazon ECS Container Instances:
      • When using the EC2 launch type, then your clusters are a group of container instances you manage. An Amazon ECS container instance is an Amazon EC2 instance that is running the Amazon ECS container agent.
    • Container Agent:
      • The container agent runs on each infrastructure resource within an Amazon ECS cluster. It sends information about the resource’s current running tasks and resource utilization to Amazon ECS, and starts and stops tasks whenever it receives a request from Amazon ECS.

使用情境 Use Cases

  • 適用：文武百業。
  • 適用：數位轉型初階段。
    • 封裝、隔離、上雲，擺脫固定資產折舊。
  • 適用：以分鐘為粒度做 auto scaling 的場景。
  • 不適合：以秒為粒度做 auto scaling 的場景。

(2020) 案例: rt.live

rt.live, a measure of how COVID-19 is spreading for each state in the United States. 每天計算全美各州 COVID-19 Rt 散播速度值的網站服務，是前 Instagram co-founders Mike Krieger 與 Kevin Systrom 離開 Facebook 後的新作品。

Mike Krieger 於 Medium 發表了一篇文章：「Automating daily runs for rt.live’s COVID-19 data using Airflow & ECS」。

rt.live 的目標有：

  1. 每日排程執行
  2. 如果過程中遇到失敗可自動重新嘗試再執行一次
  3. 可以平行處理美國各州資料，因為各州的 ML 模型不相依存於其他州
  4. 有時需要訂正某幾州的資料，可以手動重新執行某幾州，不用整個重跑

使用的工具主要有 Apache Airflow 與 Amazon’s Elastic Constiner Service (ECS)，特別是 Fargate 模式。使用 Fargate 模式，可以建立 Docker container 做為其 Airflow worker，且可以擴展成 N 個副本而不需要建立及擔心 EC2 instances。而且每天只需要 30 分鐘，得以控制支出。

(圖片來源: 原文)

  • 要開始運算前，將 ECS service desiredCount 修改成 25，待命。
  • 各個 ECS Task on Fargate 共用存放在 Elastic File System (EFS) 的 model code。
  • 使用 S3 儲存來自於 COVIDTracking 的原始資料。
  • 一個州一個 Airflow task 放進 queue 讓 ECS task (Airflow worker) 拿去執行。運算結果同樣儲存於 S3。
  • 當天運算完成後，將 ECS service desiredCount 修改回 0。

其他延伸參考資料，可以建立類似的 pipelines：How to deploy Apache Airflow with Celery on AWS 與 Lessons learned deploying Apache Airflow to ECS。

限制條件 Limits

可調限制條件

Service quotaDescriptionDefault
ClustersThe maximum number of clusters in this account in the current Region.10,000
Container instances per clusterThe maximum number of container instances per cluster.2,000
Services per clusterThe maximum number of services per cluster.5,000
Tasks per serviceThe maximum number of tasks per service (the desired count).5,000
Tasks using the EC2 launch type per service (the desired count)The maximum number of tasks using the EC2 launch type per service (the desired count). This limit applies to both standalone tasks and tasks launched as part of a service.1,000
Tasks using the Fargate launch type or the FARGATE capacity provider, per Region, per accountThe maximum number of tasks using the Fargate launch type or the FARGATE capacity provider, per Region. This limit applies to both standalone tasks and tasks launched as part of a service.100
Fargate Spot tasks, per Region, per accountThe maximum number of tasks using the FARGATE_SPOT capacity provider, per Region.250
Public IP addresses for tasks using the Fargate launch typeThe maximum number of public IP addresses used by tasks using the Fargate launch type, per Region.100

不可調限制條件

Service quotaDescriptionDefault
Tasks launched (count) per run-taskThe maximum number of tasks that can be launched per RunTask API action.10
Container instances per start-taskThe maximum number of container instances specified in a StartTask API action.10
Revisions per task definition familyThe maximum number of revisions per task definition family. Deregistering a task definition revision does not exclude it from being included in this limit.1,000,000
Task definition size limitThe maximum size, in KiB, of a task definition.32
Task definition max containersThe maximum number of containers definitions within a a task definition.10
Subnets specified in an awsvpcConfigurationThe maximum number of subnets specified within an awsvpcConfiguration.16
Security groups specified in an awsvpcConfigurationThe maximum number of security groups specified within an awsvpcConfiguration.5
Target groups per serviceThe maximum number of target groups per service, if using an Application Load Balancer or a Network Load Balancer.5
Classic Load Balancers per serviceThe maximum number of Classic Load Balancers per service.1
Tags per resourceThe maximum number of tags per resource. This applies to tasks, services, task definitions, clusters, and container instances.50

收費結構 Pricing

請參考最新官方文件。這裡僅為速記。

  • 依照 Amazon ECS Launch Types 區分兩大類。
    • Fargate Launch Type Model
    • EC2 Launch Type Model
  • Amazon ECS on AWS Outposts 比照上述 EC2 Launch Type Model。

深入討論 Deep Dive

大方向是為圭臬，但小心地上有坑。

參考資料 Reference

Awesome Lists

快速上手

範例

文章與分享

比較

  • Scaling containers on AWS in 2020, 2020-04-15, by Vlad Ionescu (AWS Container Hero) (Reading time: about 20 minutes)
  • 技術選型: Amazon ECS Launch Types: EC2 vs. Fargate, 2020-12-04, by Ernest Chiang (AWS Community Hero), at AWS DEV DAY TAIPEI 2020
  • Scaling containers on AWS in 2021, 2021-02-15, by Vlad Ionescu (AWS Container Hero) (Watching time: about 30 minutes)
  • No Kubernetes Needed: Amazon ECS Anywhere, 2021-09-10, by Janakiram MSV
    • Amazon ECS was launched in 2014 — a year before Kubernetes became available — as a managed container orchestration platform for AWS customers. It was modeled after Docker Compose, the tool that enabled multiple containers to run as a single workload. At re:Invent 2020, Amazon announced ECS Anywhere, the service that extends ECS to on-prem and other cloud environments. What this means is that ECS got an additional execution environment beyond EC2 and Fargate.
  • Scaling containers on AWS in 2022, 2022-04-15, by Vlad Ionescu (AWS Container Hero) (Reading time: about 45 minutes. 6-month research.)
Loading comments…