Devops

深入理解 Nginx 讀書筆記 (第一部份)

2020 年 9 月 25 日2020 年 9 月 25 日
Devops, Web Develop

為什麼選擇 Nginx

更快： 1)單次請求更快響應 2) 在高峰期比其他服務器更快響應
高擴展性： 1)由耦合度極低模塊組成 2)模塊皆嵌入到2進制文件中執行
高可靠性： 1)模塊穩定 2)進程相對獨立 3)worker出錯可快速輪替
低內存消耗： 1)10,000個非活躍 HTTP Keep-Alive 連接僅消耗 2.5 MB
高併發： 1)單機支援 100,000 以上連接
熱部署： 1)基於 master 與 worker 進程分離 2)服務不間斷下，進行升級可執行元件、配置及更換日誌
BSD 許可協議

開發準備工作

必要

Linux 內核版本 2.6 以上（須靠 epoll 處理高併發）
GCC 編譯器編譯 C 語言

非必要

G++，用來編譯 C++ 以編寫 HTTP 模塊
PCRE（Perl 兼容正則表達式），用來在配置文件中使用正則表達式，pcre-devel 是使用 PCRE 做二次開發所需
zlib，用來對 HTTP 內容做 gzip 壓縮，減少網路傳輸量
OpenSSL，支持 SSL 協議，或想使用 MD5 或 SHA 雜湊

目錄結構

源代碼目錄
編譯中間文件（置於源碼目錄底下，命名為objs）
部署目錄（莫認為 /usr/local/nginx）
日誌目錄

Linux 內核參數優化

須要修改內核參數，使得 Nginx 可以擁有更高的性能
通常根據業務特性進行調整，作為內容服務器、反向代理，或是提供縮圖用的服務器，會做不同調整

通用高併發 /etc/sysctl.cnf 設置如下，執行 systcl -p 套用：

fs.file-max = 999999

net.ipv4.tcp_tw_reuse = 1

net.ipv4.tcp_keepalive_time = 600

net.ipv4.tcp_fin_timeout = 30

net.ipv4.tcp_max_tw_bucket = 5000

net.ipv4.ip_local_port_range = 1024 61000

net.ipv4.tcp_rmem = 4096 32768 262142

net.ipv4.tcp_wmem = 4096 32768 262142

net.core.netdev_max_back = 8096

net.core.rmem_default = 262144

net.core.wmem_default = 262144

net.core.rmem_max = 2097152

net.core.wmem_max = 2097152

net.ipv4.tcp_syncookies = 1

net.ipv4.tcp_max_syn.backlog = 1024

每個TCP連接都會為了維持滑動窗口而消耗內存（滑動窗口，接收方通過通告發送方自己的窗口大小，從而控制發送方的發送速度），參數 rmem_default/wmem_default/rmem_max/wmem_max 須要衡量物理內存的總大小及最大連接數量 (workerprocess * workerconnections) 來決定，滑動窗口過大可能導致 OOM，過小會影響大數據量的傳輸速度

編譯安裝 Nginx

下載源碼後，最簡單的安裝方式為執行以下3行指令：

./configure： 1) 檢測系統內核及軟件 2) 解析參數 3) 生成中間目錄 4) 根據參數生成 C 源碼文件及 Makefile 文件
make: 1)使用生成的 Makefile 進行編譯 2) 生成目標文件及最終2進制文件
make install: 1) 根據參數將 nginx 部署到指定的安裝目錄 2) 複製二進制文件及配置文件

補充

以下是我練習使用的 dockerfile，按照書上指式的步驟在 centos 系統上安裝 nginx：

# ~/mynginx-dev/Dockerfile

FROM centos:centos7

# Install requires

RUN yum install -y gcc && \

yum install -y gcc-c++ && \

yum install -y pcre pcre-devel && \

yum install -y zlib zlib-devel && \

yum install -y openssl openssl-devel

# Download source code

RUN yum install -y wget && \

wget https://nginx.org/download/nginx-1.0.15.tar.gz && \

tar -zxvf nginx-1.0.15.tar.gz

WORKDIR /nginx-1.0.15

# Compile Nginx

RUN yum install -y make && \

./configure && \

make && \

make install

CMD ['bash']

你可以在目錄下執行以下指令來進入安裝 nginx 後的環境

$ docker build -t mynginx-dev .

$ docker run -it mynginx-dev bash

[root@de646f5219ee nginx-1.0.15]#

編譯完後檢視編譯中間文件

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

[root@db37b79e9661 nginx-1.0.15]# tree /nginx-1.0.15/objs

objs

|-- Makefile

|-- autoconf.err

|-- nginx

|-- nginx.8

|-- ngx_auto_config.h

|-- ngx_auto_headers.h

|-- ngx_modules.c # 定義運行時所有模組的優先順序，非常重要

|-- ngx_modules.o

`-- src # 編譯產生的目標文件

|-- core

| |-- nginx.o

| |-- ngx_array.o

| |-- ngx_buf.o

| |-- ngx_conf_file.o

| |-- ngx_connection.o

| |-- ngx_cpuinfo.o

| |-- ngx_crc32.o

| |-- ngx_crypt.o

| |-- ngx_cycle.o

| |-- ngx_file.o

| |-- ngx_hash.o

| |-- ngx_inet.o

| |-- ngx_list.o

| |-- ngx_log.o

| |-- ngx_md5.o

| |-- ngx_murmurhash.o

| |-- ngx_open_file_cache.o

| |-- ngx_output_chain.o

| |-- ngx_palloc.o

| |-- ngx_parse.o

| |-- ngx_queue.o

| |-- ngx_radix_tree.o

| |-- ngx_rbtree.o

| |-- ngx_regex.o

| |-- ngx_resolver.o

| |-- ngx_shmtx.o

| |-- ngx_slab.o

| |-- ngx_spinlock.o

| |-- ngx_string.o

| `-- ngx_times.o

|-- event

| |-- modules

| | `-- ngx_epoll_module.o

| |-- ngx_event.o

| |-- ngx_event_accept.o

| |-- ngx_event_busy_lock.o

| |-- ngx_event_connect.o

| |-- ngx_event_pipe.o

| |-- ngx_event_posted.o

| `-- ngx_event_timer.o

|-- http

| |-- modules

| | |-- ngx_http_access_module.o

| | |-- ngx_http_auth_basic_module.o

| | |-- ngx_http_autoindex_module.o

| | |-- ngx_http_browser_module.o

| | |-- ngx_http_charset_filter_module.o

| | |-- ngx_http_chunked_filter_module.o

| | |-- ngx_http_empty_gif_module.o

| | |-- ngx_http_fastcgi_module.o

| | |-- ngx_http_geo_module.o

| | |-- ngx_http_gzip_filter_module.o

| | |-- ngx_http_headers_filter_module.o

| | |-- ngx_http_index_module.o

| | |-- ngx_http_limit_req_module.o

| | |-- ngx_http_limit_zone_module.o

| | |-- ngx_http_log_module.o

| | |-- ngx_http_map_module.o

| | |-- ngx_http_memcached_module.o

| | |-- ngx_http_not_modified_filter_module.o

| | |-- ngx_http_proxy_module.o

| | |-- ngx_http_range_filter_module.o

| | |-- ngx_http_referer_module.o

| | |-- ngx_http_rewrite_module.o

| | |-- ngx_http_scgi_module.o

| | |-- ngx_http_split_clients_module.o

| | |-- ngx_http_ssi_filter_module.o

| | |-- ngx_http_static_module.o

| | |-- ngx_http_upstream_ip_hash_module.o

| | |-- ngx_http_userid_filter_module.o

| | |-- ngx_http_uwsgi_module.o

| | `-- perl

| |-- ngx_http.o

| |-- ngx_http_busy_lock.o

| |-- ngx_http_copy_filter_module.o

| |-- ngx_http_core_module.o

| |-- ngx_http_file_cache.o

| |-- ngx_http_header_filter_module.o

| |-- ngx_http_parse.o

| |-- ngx_http_parse_time.o

| |-- ngx_http_postpone_filter_module.o

| |-- ngx_http_request.o

| |-- ngx_http_request_body.o

| |-- ngx_http_script.o

| |-- ngx_http_special_response.o

| |-- ngx_http_upstream.o

| |-- ngx_http_upstream_round_robin.o

| |-- ngx_http_variables.o

| `-- ngx_http_write_filter_module.o

|-- mail

|-- misc

`-- os

|-- unix

| |-- ngx_alloc.o

| |-- ngx_channel.o

| |-- ngx_daemon.o

| |-- ngx_errno.o

| |-- ngx_files.o

| |-- ngx_linux_init.o

| |-- ngx_linux_sendfile_chain.o

| |-- ngx_posix_init.o

| |-- ngx_process.o

| |-- ngx_process_cycle.o

| |-- ngx_readv_chain.o

| |-- ngx_recv.o

| |-- ngx_send.o

| |-- ngx_setproctitle.o

| |-- ngx_shmem.o

| |-- ngx_socket.o

| |-- ngx_time.o

| |-- ngx_udp_recv.o

| |-- ngx_user.o

| `-- ngx_writev_chain.o

`-- win32

安裝完後檢視部署目錄

[root@db37b79e9661 nginx-1.0.15]# tree /usr/local/nginx

/usr/local/nginx

|-- conf

| |-- fastcgi.conf

| |-- fastcgi.conf.default

| |-- fastcgi_params

| |-- fastcgi_params.default

| |-- koi-utf

| |-- koi-win

| |-- mime.types

| |-- mime.types.default

| |-- nginx.conf # 配置文件

| |-- nginx.conf.default

| |-- scgi_params

| |-- scgi_params.default

| |-- uwsgi_params

| |-- uwsgi_params.default

| `-- win-utf

|-- html

| |-- 50x.html

| `-- index.html

|-- logs

`-- sbin # 可執行文件

`-- nginx

命令行控制

默認方式啟動二進制程序，不指定 -c 路徑的情況下，會套用默認的配置文件

1 2	/usr/local/nginx/sbin/nginx

檢視程序

[root@db37b79e9661 nginx-1.0.15]# ps -ef | grep nginx

root 31 1 0 12:41 ? 00:00:00 nginx: master process /usr/local/nginx/sbin/nginx

nobody 32 31 0 12:41 ? 00:00:00 nginx: worker process

關閉服務，使用 quit 參數則等待當前請求處理完畢才停止

/usr/local/nginx/sbin/nginx -s stop

# equals

kill -s SIGTERM <master pid>

# equals

kill -s SIGINT <master pid>

/usr/local/nginx/sbin/nginx -s quit

# equals

kill -s SIGQUIT <master pid>

運行中重新加載配置，指令會先檢查配置是否有誤，若通過則以 quit 方式重啟

/usr/local/nginx/sbin/nginx -s reload

# equals

kill -s SIGHUP <master pid>

滾動升級 Nginx

發送訊號通知舊版服務

1
2

kill -s SIGUSR2 <master pid>
啟動新版本服務
以 quit 方式結束舊服務

HBase Basics

2020 年 7 月 22 日2020 年 7 月 27 日
Devops

Apache HBase is an open source, scalable, consistent, low latency, random access data store

Source from Infinite Skills

Features

Horizontally Scalable

Linear increase in servers results in linear increases in storage capacity and I/O operations

CAP Trade off

In CAP theory, Hbase is more likely a CP type of system

Consistency: ACID(atomicity, consistency, isolation, durability) garantees on rows
Availability: Response time 2-3ms from cache, 10-20ms from disk
Partition Tolerance: Failures don’t block system. It might take longer to response to maintain consistency

Dependencies

Apache ZooKeeper

Use for distributed coordination of leaders for high availability
Optimized to be highly avaiable for reads
Not designed to scale for high write throughput

Apache Hadoop HDFS

Provide data durability and reliability
Optimized for sequential reads and writes of large files
Does not provide random updates, only simple API for rando reads
Cannot scale tens of billions of small entities (less then a few hundred MB)

Both system have their strengths but do not individually provide the same properties as HBase

Random Access

Optimized for small random reads

Entities indexed for efficient random reads

Optimized for high throughput random writes

Updates without requiring read
Random writes via Log Structured Merge (LSM)

Short History

Inspired from Google’s Bigtable

Bigtable: A Distributed Storage System for Structured Data(2006)

BigTable

Datastore for Google’s Web Crawl Table

Store web page content
Web URL as key
Use MapReduce to find links and generate backlinks
Calculate page rank to build the Google index

Later, it also used as backend for Gmail, GA, Google Earth etc.

Hadoop HDFS

Inspired by Google distributed file system GFS

Timeline

Since 2009, many compaies (Yahoo, Facebook, eBay etc.) chose to use HBase for large scale production use case

In 2015, Google announced BigTable with HBase 1.0 compatible API support for its compute engine users

2017, HBase 2.0.0

2020, HBase 3.0.0

Despite being bucketed into NoSQL category of data storage, some of intresting are moving NoSQL back to SQL, by using HBase as a storage engine for SQL compliant OLTP database system.

Use case

HBase’s strengths are its ability to scale and sustain high write throughputs

Many HBase apps are:

Ports from RDBMS to HBase
New low-latency big data apps

How to Porting RDBMS to HBase?

Many RDBMS are painful to scale
Scale up is no longer pratical for massive data
Data inconsistency was not acceptable when scaling reads
Operationally gets more complicated as the number of replicas increases
Operational techniques not sufficient when scaling writes

To make it easier to scale, we need to discard the fundamental features that RDBMS provides, such as:

text search (LIKE)
joins
foreign keys and avoid constraint checks

Changing the schema, make it only contains denormalized tables, we won’t incur replication IO when sharding the RDBMS

Now you’re relatively straightforward porting RDBMS to HBase

Why choosing HBase instead?

When your apps need high wirte and read throughput
When you tired of RDMS’s fragile scaling operations

Data Volumes

Entity data: information about the current state of a particular persion or thing
Event data(or time series data): Records events that are generally spaced over many time intervals

Data volume explods when we need both of them

HBase or Not

Q: Does your app expect new data to be vailable immediately after an update?

Yes: Use HBase
- When data queried, must reflect the most recent values
- Expect query responses in milliseconds
No: No need for HBase

Q: Whether your app analytical or operational?

Analytical: Not optimal for HBase
- Look for large set of data
- Often filter for particular time range
- Better choose Hadoop
Operational: Use HBase
- Look for single or small set of entities

Q: Does your app expect updates to be available immediately after an update?

Yes: Use HBase
- Frequently modified
- Pinpoint deletes
- Updates must be reflected within milliseconds
No: No need for HBase
- Data is append-only
- Deletes in bulk or never
- Updates can be ignored until the next report is run

comparison

Workload	HBase	Hadoop
Low Latency	1ms from cache 10ms from disk	1min vis MR/Spark 1s via Impala
Random Read	Rowkey is primary index	The small file problem
Short Scan	Sorted and efficient	Bespoke partitioning can help
Full Scan	Possible but non-optimal Improved pref w/MR on snapshots	Optimized with MR, Hive, Impala
Updates	Optimized	Not supported

Kubernetes Short Notes(4)

2020 年 1 月 6 日2020 年 4 月 26 日
Devops

Storage

Persistent Volume

Except storing a volume on the host, Kubernetes provide serveral type of storage solutions

NFS
GlusterFS
Flocker
Ceph
ScaleIO
AWS EBS
Azure Disk
Google Persistent Disk

Persistent Volume Claim

Administrators creates PV, and users creates PVC to use the PV, during the binding process Kubernetes tries to find the PV that has sufficient capacity as requested by the claim and any other request properties such as access modes, volume modes, storage class and selector

Note that a smaller claim may get bound to a larger volume if all the other criteria matches and there are no better options

There is a one to one relationship bewteen PV and PVC, no other claims can utilize the remaining capacity in the volume

Configure the field persistentVolumeReclaimPolicy to defined what action to perform to PV after a PVC deleted.

Retain (default)
Delete
Recycle

Networking

Networking for Linux Basics

Network Switch

A switch is a device in a computer network that connects other devices together, can only enable a communication within a network

Host A(192.168.1.10)[eth0] ↔ Switch(192.168.1.0) ↔ [eth0]Host B(192.168.1.11)

# For Network A

$ ip link

$ ip addr add 192.168.1.10/24 dev eth0 # set a ip addr for interface eth0

# For Network B

$ ip link

$ ip addr add 192.168.1.11/24 dev eth0

# Test

$ ping 192.168.1.11

Router

A router is a device/service that provides the function of routing IP packets between networks

Switch(192.168.1.0) <–> [192.168.1.1]Router[192.168.2.1] <–> Switch(192.168.2.0)

Route/Gateway

A gateway (in network terms) is a router that describes the function for connectivity

# For Network A

$ ip route add 192.168.2.0/24 via 192.168.1.1

# For Network B

$ ip route add 192.168.1.0/24 via 192.168.2.1

Default Gateway

If none of these forwarding rules in the routing table is appropriate for a given destination address, the default gateway is chosen as the default router of last resort

1 2	$ ip route show default

Forwording packets between interfaces

By default in linux, packets are not forwarded from one interface to the next, for security reasons

Explicity allow it

1 2	echo 1 > /proc/sys/net/ipv4/ip_forward

Persists the settings

1 2	net.ipv4.ip_forward = 1

DNS

Translate host name to IP address by configure the /etc/hosts

When a environment has too many entries and IP address are not persistent, we need a DNS server

$ cat /etc/resolv.conf

nameserver 192.168.1.100

The host will lookup an entry in /etc/hosts first, then lookup in the DNS. This order can be changed by configure file /etc/nsswitch.conf

$ cat /etc/nsswitch.conf

passwd: files

group: files

shadow: files

gshadow: files

hosts: files dns

networks: files

protocols: db files

services: db files

ethers: db files

rpc: db files

netgroup: nis

You can configure the DNS server to forward unknown host name to the public name server in the Internet, for example reach www.google.com

private DNS → Root DNS → .com DNS → google DNS → cache the result

When looking for a host in the same domain, we want to simple use the host name not the full name, such as using web not web.mycompany.com, therefore we specify the domain name you want to append in /etc/resolv.conf

$ cat /etc/resolv.conf

search mycompany.com

There are records stores in DNS with specific types:

A: ipv4
AAAA: ipv6
CNAME: name to name mapping

You can use tools like nslookup, dig to debug, note that nslookup only query from dns, not files

There are plenty DNS solutions, such as CoreDNS, except configure from files, CoreDNS supports other ways of configuring DNS entries through plugins like kubernetes

Network Namespace

A namespace is a way of scoping a particular set of identifiers

Linux provides namespaces for networking and processes, if a process is running within a process namespace, it can only see and communicate with other processes in the same namespace

Linux starts up with a default network namespace

Each network namespace has its own routing table and has its own set of iptables

# Create namespace

ip netns add red

# List namespace

ip netns list

# List interface

ip link

# List interface in namespace

ip netns exec red ip link

# or

ip -n red link

Connect namespaces together using a virtual Ethernet pair (or virtual cable, pipe)

# Creat veth pair

$ ip link add veth-red type veth peer name veth-blud

# Attach each interface to the appropriate namespace

$ ip link set veth-red netns red

$ ip link set veth-blue netns blue

# Assign IP to each namespaces

$ ip -n red addr add 192.168.15.1 dev veth-red

$ ip -n blue addr add 192.168.15.2 dev veth-blue

# Bring up the interface for each device within the respective namespace

$ ip -n red link set veth-red up

$ ip -n blue link set veth-blue up

# List ARP table to see neighbor

$ ip netns exec red arp

# Ping across namespace

$ ip netns exec red ping 192.168.15.2

When there more of namespaces need connected, use a virtial switch to create a virtial network. There few solutions:

Linux Bridge
Open vSwitch

# Create a virtial switch interface

ip link add v-net-0 type bridge

# Bring the interface up

ip link set dev v-net-0 up

# Delete the previous cable and create new cable to connect to the bridge

ip link add veth-red type veth peer name veth-red-br

ip link add veth-blue type veth peer name veth-blue-br

# Attach one end to the appropriate namespace

$ ip link set veth-red netns red

$ ip link set veth-blue netns blue

# Attach the other end to the bridge

$ ip link set veth-red-br master v-net-0

$ ip link set veth-blue-br master v-net-0

# Assign IP to each namespaces

$ ip -n red addr add 192.168.15.1 dev veth-red

$ ip -n blue addr add 192.168.15.2 dev veth-blue

# Bring up the interface for each device within the respective namespace

$ ip -n red link set veth-red up

$ ip -n blue link set veth-blue up

# Assign IP address to the bridge (since it’s just another interface on the host)

ip addr add 12.168.15.3/24 dev v-net-0

# Ping accross namespaces

$ ip netns exec red ping 192.168.15.2

When a private virtual network need to reach the outer network, it need a gateway, the host is the gateway

1 2	$ ip netns exec red ip route add 192.168.1.0/24 via 192.168.15.3

For destination network to response, enable NAT on host acting as a gateway.

Add a new rule in the NAT IP table in the POSTROUTING chain to masquerade or replace the from address on all packets coming from the source network 192.168.15.0 with its own IP address.

Thus anyone receiving these packets outside the network will think that they are coming from the host and not from within the namespaces

1 2	$ iptables -t nat -A POSTROUTING -s 192.168.15.0/24 -j MAS

Add a route using default gateway to outside world

1 2	$ ip netns exec red ip route add default via 192.168.15.3

For outside world to reach the namespace in a private network, add a port forwarding rule using IP tables to say any traffic coming to port 80 on the localhost is to be forwarded to port 80 on the IP assigned to the namespace

$ iptables \

-t nat \

-A PREROUTING \

--dport 80 \

--to-destination 192.168.15.1:80 \

-j DNAT

Kubernetes Short Notes(3)

2019 年 12 月 17 日2019 年 12 月 18 日
Devops

Cluster Maintainance

OS Upgrade

Pod Eviction Timeout

When the nodes was down for more than 5 minute(default) then the pods are terminated; pod will recreate if has replicaset

Drain, Cordon, Uncordon

We’re not sure the node will come back online in 5 minutes, therefore we can drain the node.

After the drained node upgraded and come back, it still unschedulable, uncordon the node to make it schedulable.

Note that the previouse pods won’t be automatically reschedule back to the nodes.

Cluster Upgrade

The core control plane components’s version can be different, but should follow certain rules:

the kube-api is the primary component, none of the other components’s version must not be higher than the kube-api
the components can be lower in 1-2 versions
- kube-api: x
- Controlloer-manager, kube-scheduler: x, x-1
- kubelet, kube-proxy: x, x-1, x-2
the kubectl can be one version higher than kube-api: x+1, x, x-1

The kubernetes support only up to the recent 3 minor versions. The recommanded approch is to update one minor version at a time.

Update the cluster depend on how you deploy them:

cloud provider: few clicks at the UI
kubeadm: using upgrade argument (you should upgrade the kubeadm first!)
the hard way from scratch: manually upgrade components by yourself

Two major steps:

upgrade master node, the control plane componets goes down, all management function are down, only the applications deploy on worker nodes keeps serving
update worker nodes, with strategies:
- upgrade all at once with downtimes
- upgrade one at a time
- create new nodes and remove the workloads, then finally remove old nodes

When you run command like kubectl get nodes, the VERSION is indicat the version of the kubelet

Backup and Restore

Master / Node DR

Cordon & drain
Provision replacement master / node

ETCD DR

Option: Backup resources

Saving objects as a copy by query the kube-api

1 2	kubectl get all --namespace=default -o yaml > default-deplayment-services.yaml

Option: Backup ETCD

Making copies of the ETCD data directory

# etcd.service

ExecStart=/user/local/bin/etcd \

--name= ${ETCD_NAME}

...

--data-dir=/var/lib/etcd

Or use the etcd command line tool

Make a snap shot

ETCDTL_API=3 etcdctl backup save etcd.db \

--endpoint=http://127.0.0.1:2379 \

--cacert=/etc/etcd/ca.crt \

--cert=/etc/etcd/etcd-server.crt \

--key=/etc/etcd/etcd

Remember to specify the certification files for authentication

Stop kube-api

1
2

service kube-apiserver stop

Restore snapshot

ETCDCTL_API=3 etcdctl snapshot restore etcd.db \

-- data-dir=/var/lib/etcd-backup-dir \

--initial-cluster master-1=https://192.169.5.11:2380,master-2=https://192.168.5.12:2380 \

--initial-cluster-token etck-cluster-1 \

--initial-advertise-peer-urls https://${INTERNAL_IP}:2380

When ETCD restore from a backup, it initialize a new cluster configuration and configures the members of ETCD as new members to a new cluster. This is to prevent a new member from accidentally joining an existing cluster.
For example, using a snapshot to provision a new etcd-cluster from testing purpose. You don’t want the members in the new test cluster to accidentally join the production cluster.

Configure the etcd.service with new data directory and new cluster token

During a restore, you must provide a new cluster token and the same initial cluster configuration
Restart ETCD service

1
2
3

systemctl daemon-reload
service etcd restart
Start kube-api

1
2

service kube-apiserver start

Persistant Volume DR

You can’t relay on kubernetes to for backing up and restore persistant volumes.

If you’re using cloud provider specific persistant volumes like EBS volumes, Azure managed disks or GCE persistent disks, you should use cloud provider snapshot APIs

Kubernetes Short Notes (2)

2019 年 10 月 6 日2019 年 10 月 6 日
Devops

Scheduling

Manual Scheduling

Bind the pod to node by nodeName property, before that, the pod stays in the Pending state

Manutal ways to bind:

specify the spec.nodeName, not updatable

create the Binding object

curl --header "Content-Type:application/json" \

--request POST --data '{"apiVersion": "v1", "kind": "Binding" ...}' \

http://$SERVER/api/v1/namespaces/default/pods/$PODNAME/binding/

Labeling

Use to group and select the objects, for example a ReplicaSet object configs:

metadata.labels sets the ReplicaSet itself
spec.template.metadata.lables sets the Pod
spec.selector.matchLabels defines how ReplicaSet to discover the Pod

Annotations

Use to record other details for intergration purpose e.g. build info, contact details

Restriction

Taint/Toleration

Limit pods without tolerations cannot get scheduled to a tainted node

Taint the nodes

1
2
3

kubectl taint nodes master node-role.kubernetes.io/master:NoSchedule # Taint
ubectl taint nodes master node-role.kubernetes.io/master:NoSchedule- # UnTaint
Set the pods’ tolerance, three behavior are avaliable if not tolerant:
- NoSchedule
PreferNoSchedule: not guaranteed

NoExecute: new pods=NoSchedule, existed pods=evicted

Note the value in tolerations keys must use double quotes

Node Selector

Limit the pod to get scheduled to one kind of node only

Lable the node
Set the nodeSelector

Note there is no OR or NOT conditions, use node affinity instead

Node Affinity

Limit the pod to get scheduled to one or more particular nodes

Lable the node
Set the nodeAffinity
operators: In, NotIn, Exists, DoesNotExist, Gt, Lt
3 types

Combines the Taint/Toleration with NodeSelector or NodeAffinity to cover the scenarios

Resources

Request

The scheduling base on the resource requests
By default, k8s assumes a pod requires 0.5 cpu and 256Mi memory

Limit

By default, k8s limit a pod to 1 cpu and 512Mi memory
When a pod try to exceed resources beyond the limit
- cpu: k8s throttles the cpu won’t kill
- memory: k8s kill the pod with OOM

Static Pods

Use in creating control plane components (kube admin tools)

Without the intervention from the kube-api server, the kubelet can manage a node independently by monitor config files in the file system, and be able to create, recreate, update and delete the POD only object

–pod-manifest-path=/etc/Kubernetes/manifest
–config=kubeconfig.yaml (staticPodPath)

While the static pod created, the kube-api only get a readable mirror and not have the ability to update/delete it

Multiple Scheduler

copy the kube-scheduler configs from /etc/kubernetes/manifests
rename the scheduler --scheduler-name
if one master nodes with multiple scheduler:
- set the --leader-elect=false
if multiple masters with multiple scheduler, only one scheduler can active at a time
- set the --leader-elect=true
- set the --lock-object-name to differentiate the custom scheduler from default if multiple master
specify the scheduler for pod by schedulerName

2019 年 9 月 27 日2020 年 1 月 6 日
Devops

Cluster Architecture

Master Node

ETCD cluster
kube-scheduler
kube-controller-manager

These components communicate via kube-api server

Worker Node

container runtime engine, e.g. Docker, Rocket, ContainerD
kubelet: agent that runs and listen for instructions from kube-api
containers

The services deploy within worker nodes communicate with each other via kube-proxy

Objectives

ETCD

a distributed reliable key-value store
client commuications on port 2379
server to server on port 2380

kube-api

primary management component
setup:
1. using kube-admin tools
  - deploy kube-api as a pod in kube-system namespace
  - the manifests is at /etc/kubernetes/manifests/kube-apiserver.yaml
  - the options is at /etc/systemd/system/kube-apiserver.service
  - search for kube-apiserver process on master node
    
    1
    2
    
    ps -aux | grep kube-apiserver
example: apply deployment using kubectl
1. authenticates user
validate the HTTP requests

the kube-scheduler monitored the changes from the kube-api, then:

retrieve the node information from kube-api

schedule the pod to some node through kube-api to kubelet

update the pod info to ETCD

kube-controller-manager

continuously monitors the state of components
the controllers packages into a single process called Kube-Controller-Manager, which includes:
1. deployment-controller, cronjob, service-account-controller …
2. namespace-controller, job-contorller, node-controller …
3. endpoint-controller, replicaset, replication-controller(replica set) …
remediate situation

kube-scheduler

decide which pod goes to which node
1. filter nodes
2. rank nodes

kubelet

follow the instruction from kube-scheduler to controll the container runtime engine (e.g. docker) that run or remove a container
using kube-admin tools to deploy cluster, the kubelet are not installed by default in worker nodes, need intstall manually

kube-proxy

runs on each nodes in the cluster
create iptables rules on each nodes to forward traffic heading to the IP of the services to the IP of the actual pods
kube-admin tool deploy kube-proxy as daemonset in each nodes

pod

the container are encapsulated into a pod
is a single instance of an application, the smallest object in k8s
containers in same pod shares storages and network namespaces, created and removed in the same time
multi-container pod is rare use case

ReplicationController

apiVersion support in v1
the process to monitor the pods
maintain the HA and specified number of pods that running on all nodes
only care about the pod which RestartPolicy is set to Always
scalable and replacable application should be managed by the controller
use cases: rolling updates, multiple release tracks (multiple replication controller replica the same pod but using different labels)

ReplicaSets

next generation of ReplicationController
api version support in apps/v1
enhance the filtering in .spec.selector (the major difference)
be aware of the non-template pod that has same lables
using Deployment as a replacement is recommended, it own and manage its ReplicaSets

Deployment

provide replication vis replicaset and other:
- rolling update
- rollout
- pause and resume

Namespace

namespaces created at cluster creation
1. kube-system
kube-public

default

each namespace can be assigned quota of resources
a DNS entry with SERVICE_NAME.NAMESPACE.svc.cluster.local format is automatically created when at service creation
1. the cluster.local is the default domain name of the cluster
permanently config the namespace

1
2

kubectl config set-context $(kubectl config current-context) --namespace=$NAMESPACE

Devops

深入理解 Nginx 讀書筆記 (第一部份)

為什麼選擇 Nginx

開發準備工作

目錄結構

Linux 內核參數優化

編譯安裝 Nginx

補充

命令行控制

HBase Basics

Features

Horizontally Scalable

CAP Trade off

Dependencies

Apache ZooKeeper

Apache Hadoop HDFS

Random Access

Short History

Use case

Kubernetes Short Notes(4)

Storage

Persistent Volume

Persistent Volume Claim

Networking

Networking for Linux Basics

Kubernetes Short Notes(3)

Cluster Maintainance

OS Upgrade

Cluster Upgrade

Backup and Restore

Kubernetes Short Notes (2)

Scheduling

Manual Scheduling

Labeling

Annotations

Restriction

Resources

Static Pods

Multiple Scheduler

Kubernetes Short Notes (1)

Cluster Architecture

Objectives