tags: gcp, cloud-vpn, shared-vpc, terraform
為了更了解地端、雲端網路如何連接，用了 GCP 來做一整套簡單的實驗。
架構說明
雲端
為了模擬比較複雜的組織架構，雲端使用了 Shared VPC 的架構，有以下幾個優點：
- 減少 VPC 數量，較容易管理
- 多個子專案(service project)可以共享同一個主專案(host project)定義的網路
- 子專案可以各自分開 billing
- 劃分網路權限，子專案只能使用主專案定義的網路
要使用 Shared VPC 架構有幾個前提：
- 須要有 GCP Organization
- 主專案須設定為 shared VPC host
- 子專案須 attach 到主專案
雲端的 VPC Gateway 的實作方式採用 GCP 的 Classic VPN，也稱為 route-based VPN，採用事先定義好的靜態路由規則(static routes) 來做 routing。
GCP 還有另一套 VPN 叫 HA VPN，背後是使用 BGP 來做動態路由設定，由於架構較複雜，沒在此實驗採用。
要使用 GCP 的 Classic VPN 有以下前提：
- 兩端只能走 IPsec 協議
- 兩端都必須有 public IPv4 位址
在兩端 peering 上也要避免 IP 位址重疊。另外，若要做到高可用，可以建立多個 VPN gateway 來達成。
地端
為了模擬地端環境，另開一個 GCP 專案，也會設定一個獨立的 VPC。
地端採用一台 VM 實例來扮演 VPC Gateway，使用的工具是 strongSwan
1. 申請 GCP Organization
- 購買一組網域，例如
example.com - 建立一個信箱，例如
[email protected] - 使用此網域註冊 GCP Organization
- 登入 admin.google.com 驗證網域，有幾種方式可以驗證，例如在 domain provider 新增 TXT DNS record
- 登入 GCP，取得 $300 美金抵用額
2. 建立 Host/Service/On-premise 專案
專案也可以透過 Organization Admin Level 的帳戶配合 Terraform 去定義，但為了簡化這個流程，假設我們皆是手動建立
- 建立 Host Project
- 為 Host Project 啟用 Shared VPC
12$ gcloud compute shared-vpc enable <host-project-id>
- 建立 Service Project
-
將 Service Project 連接到 Host Project
12$ gcloud compute shared-vpc associated-projects add <service project-id> --host-project <host-project-id> - 建立 On-Premise Project
如果是用 Terraform 定義，設定 Shared VPC 的定義大致如下：
|
1 2 3 4 5 6 7 8 9 10 11 |
# Enable Shared VPC to host project. resource "google_compute_shared_vpc_host_project" "host" { project = "my-host-project" } # Attach service project to host project. resource "google_compute_shared_vpc_service_project" "service" { host_project = "my-host-project" service_project = "my-service-project" } |
3. 設定 Terraform 環境
設定 state backend 的方式有很多種，這裡就不細講，你也可以使用 GCS。為了要可以操作 GCP 資源，比較重要的是權限，建議的作法是透過 Service Account，為了可以更好的模擬權限，這裡我們為每個環境各建議一組 Service Account：
| Account | Roles | Scope |
|---|---|---|
| Host Project Admin | Project IAM Admin, Network Admin, Security Admin | Host Project |
| Service Project Developer | subnet level compute.networkUser, Instance Admin | Service Project |
| On-premise Admin | Owner | On-premise Project |
接下來我們可以使用容器來操作 Terraform，將服務帳戶 mount 進容器指定目錄即可，以下是範例 Dockerfile：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# Stage 1 FROM hashicorp/terraform:1.3.4 AS terraform-deps # Stage 2 FROM google/cloud-sdk:409.0.0-slim RUN gcloud config set core/disable_usage_reporting true && \ gcloud config set component_manager/disable_update_check true COPY --from=terraform-deps /bin/terraform /bin/terraform CMD ["gcloud", "auth", "activate-service-account", "--key-file=/creds/sa.json, "&&", "tail", "-F", "/dev/null"] |
4. 定義 Host Project 資源
Host Project 有幾個主要的資源要定義：網路、靜態 VPN。
網路部份包含一個乾淨的 VPC，以及其中切一個子網段出來給 Service Project 使用，如架構圖所示，這邊切的網段是 10.1.0.0/16：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
resource "google_compute_network" "main" { name = "main" routing_mode = "REGIONAL" auto_create_subnetworks = false delete_default_routes_on_create = true } resource "google_compute_subnetwork" "service" { name = "service" ip_cidr_range = "10.1.0.0/16" region = "asia-east1" network = google_compute_network.main.id } |
建立好之後，還要 assign 子網路權限給 Service Project 的服務帳戶：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
resource "google_compute_subnetwork_iam_policy" "service0" { subnetwork = google_compute_subnetwork.service0.name policy_data = data.google_iam_policy.service0.policy_data } data "google_iam_policy" "service0" { binding { role = "roles/compute.networkUser" members = [ "serviceAccount:<service-project-developer-sa-email>" # highlight ] } } |
接下來設定靜態 VPN，假設 gateway 取得的對外 IP 為 104.199.159.46：
|
1 2 3 4 5 6 7 8 9 |
resource "google_compute_vpn_gateway" "cloud_vpn_gw" { name = "cloud-vpn-gw" network = google_compute_network.main.id } resource "google_compute_address" "cloud_vpn" { name = "cloud-vpn" } |
由於是靜態路由，我們還不知道地端的對外 Gateway IP 以及要 peer 的地端網段為何，以下資源必須等地端網路定義出來後才能建立。
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
resource "google_compute_forwarding_rule" "fr_esp" { name = "fr-esp" ip_protocol = "ESP" ip_address = google_compute_address.cloud_vpn.address target = google_compute_vpn_gateway.cloud_vpn_gw.id } resource "google_compute_forwarding_rule" "fr_udp500" { name = "fr-udp500" ip_protocol = "UDP" port_range = "500" ip_address = google_compute_address.cloud_vpn.address target = google_compute_vpn_gateway.cloud_vpn_gw.id } resource "google_compute_forwarding_rule" "fr_udp4500" { name = "fr-udp4500" ip_protocol = "UDP" port_range = "4500" ip_address = google_compute_address.cloud_vpn.address target = google_compute_vpn_gateway.cloud_vpn_gw.id } resource "google_compute_vpn_tunnel" "on_premise" { name = "on-premise" peer_ip = "35.221.139.153" # highlight shared_secret = "Xt9KllbZqYJtUJ5ggkgAjYnATuzbgo6q" # highlight ike_version = 2 local_traffic_selector = ["10.1.0.0/16"] remote_traffic_selector = ["192.168.1.0/24"] target_vpn_gateway = google_compute_vpn_gateway.cloud_vpn_gw.id depends_on = [ google_compute_forwarding_rule.fr_esp, google_compute_forwarding_rule.fr_udp500, google_compute_forwarding_rule.fr_udp4500, ] resource "google_compute_route" "on_premise" { name = "on-premise" network = google_compute_network.main.name dest_range = "192.168.1.0/24" next_hop_vpn_tunnel = google_compute_vpn_tunnel.on_premise.id priority = 1000 } |
由於 GCP 的預設 FW 是 default deny all，來源為地端的 packets 可以進得來：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
resource "google_compute_firewall" "allow_on_premise" { name = "allow-on-premise" network = google_compute_network.main.name allow { protocol = "tcp" } allow { protocol = "udp" } allow { protocol = "icmp" } source_ranges = ["192.168.1.0/24"] priority = 1001 } |
5. 定義地端專案資源
一樣先定義網路，為了比較直覺與雲端區分，這邊子網路用 192.168.1.0/24
|
1 2 3 4 5 6 7 8 9 10 11 12 13 |
resource "google_compute_network" "main" { name = "main" routing_mode = "REGIONAL" auto_create_subnetworks = false } resource "google_compute_subnetwork" "subnet0" { name = "subnet0" ip_cidr_range = "192.168.1.0/24" region = "asia-east1" network = google_compute_network.main.id } |
地端我們起一台 VM 來充當 VPN gateway 於 192.168.1.2 並假設取得對外 IP 為 35.221.139.153，最後記得要開啟 IP Forwarding 讓它可以充當 router：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
resource "google_compute_address" "on_prem_gw" { name = "on-prem-gw" } resource "google_compute_instance" "on_prem_gw" { name = "on-prem-gw" machine_type = "f1-micro" zone = "asia-east1-a" boot_disk { initialize_params { image = "debian-11-bullseye-v20220822" } } network_interface { subnetwork = google_compute_subnetwork.subnet0.name network_ip = "192.168.1.2" access_config { nat_ip = google_compute_address.on_prem_gw.address } } can_ip_forward = true metadata_startup_script = file("./vpn_start.sh") tags = ["vpn-gateway"] } |
於此 gateway 的啟動腳本我們安裝 VPN server，自訂一組 IPSec secret 並設定好 tunnel 規則：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 |
#!/bin/bash cat << EOF > ipsec.conf conn vpn_tunnel authby=psk auto=route dpdaction=hold ike=aes256-sha1-modp2048,aes256-sha256-modp2048,aes256-sha384-modp2048,aes256-sha512-modp2048! esp=aes256-sha1-modp2048,aes256-sha256-modp2048,aes256-sha384-modp2048,aes256-sha512-modp2048! forceencaps=yes keyexchange=ikev2 mobike=no type=tunnel left=%any leftid=35.221.139.153 leftsubnet=192.168.0.0/16 leftauth=psk leftikeport=4500 right=104.199.159.46 rightsubnet=10.0.0.0/8 rightauth=psk rightikeport=4500 EOF sudo apt-get update && apt-get install strongswan -y # Patch IPSec secret echo "%any : PSK Xt9KllbZqYJtUJ5ggkgAjYnATuzbgo6q" | sudo tee /etc/ipsec.secrets > /dev/null # Turn on IP forward echo "net.ipv4.ip_forward=1" | sudo tee -a /etc/sysctl.conf > /dev/null && sudo sysctl -p # Bring up tunnels sudo cp ipsec.conf /etc && \ ipsec restart && \ ipsec up vpn_tunnel |
要讓 subnet 的 IP 可以訪問雲端的 10.0 網段，需要設定路由規則，將 next hop 設定成上面的 VPN gateway
|
1 2 3 4 5 6 7 8 |
resource "google_compute_route" "cloud_vpn" { name = "cloud-vpn" network = google_compute_network.subnet0.name dest_range = "10.1.0.0/16" next_hop_instance = google_compute_instance.on_prem_gw.id priority = 1000 } |
從雲端來的 packets 也要設定防火牆允許進入：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
resource "google_compute_firewall" "vpn_allow_ike_esp" { name = "vpn-allow-ike-esp" network = google_compute_network.vpn0.name allow { protocol = "udp" ports = ["500", "4500"] } target_tags = ["vpn-gateway"] source_ranges = ["104.199.159.46"] priority = 1001 } resource "google_compute_firewall" "allow_cloud" { name = "allow-cloud" network = google_compute_network.main.name allow { protocol = "tcp" } allow { protocol = "udp" } allow { protocol = "icmp" } source_ranges = ["10.1.0.0/16"] priority = 1001 } |
6. 在雲端 Service Project 開 VM
將 VM 指定到 Host Project 的子網段：
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 |
data "google_compute_subnetwork" "shared_subnet" { name = "shared-subnet" project = "my-host-project" region = "asia-east1" } resource "google_compute_instance" "foo" { name = "foo" machine_type = "f1-micro" boot_disk { initialize_params { image = "debian-11-bullseye-v20220822" } } network_interface { network_ip = "10.1.0.2" access_config { nat_ip = null } subnetwork = data.google_compute_subnetwork.shared_subnet.self_link } } |
7. 在地端開 VM
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
resource "google_compute_instance" "bar" { name = "bar" machine_type = "f1-micro" zone = "asia-east1-a" boot_disk { initialize_params { image = "debian-11-bullseye-v20220822" } } network_interface { network_ip = "192.168.1.3" subnetwork = google_compute_subnetwork.subnet0.name access_config { nat_ip = null } } } |
8. 測試連線
作到這裡，預期地端的機器 192.168.1.3 可以透過地端 VPN gateway 192.168.1.2 通到雲端機器 10.1.0.2。另外，若你要在 local SSH 到 VM，還需要額外的設定，例如透過 self-managed SSH key 或 OS Login。這在上面範例都刻意省略了。
最後，如果連線失敗的話，再檢查看看。當然，也有可能是我漏寫了什麼或改名稱沒改好。