Redis

tags: redis

Redis

compare to memcached

  • support persistant volume
    • RDB
    • AOF
  • support multiple data types
  • pub/sub

commands

  • redis-cli: command line interface
  • redis-sentinel: cluster managing tool
  • redis-server: run server
  • redis-benchmark: stress testing
  • redis-check-aof: check AOF
  • redis-check-dump: check RDB

configuration

Use redis.conf. Docker official redis image not contain this file. Mount it yourself or through redis-server arguments.

types

  • String: get, set, mget, mset
  • Integer: incr, decr, setbit
  • List: lpush, lrange, lpop
  • Hash Map: hset, hget, hmset, hmget
  • Set: sadd, smember, sdiff, sinter, sunion

use docker

Before start

To connect a container, you need to know the name and the port, in the associated networks to be able to discover the service.

There is no DNS resolution in docker deault bridge network. In default network, you need to specify --link to connect the containers. The --link is a legacy feature.

Therefore, create a user-defined network is recommanded, it provide automatic DNS resolution.

Create a bridge newrok

Run a redis instance in user-defined network

Run a redis-cli connect to the redis instance

Transaction

all commands are executed as a single isolated operation, serialized and executed sequentially
atomic: all failed or all succeed

  • MULTI: open a transaction and always return OK
  • EXEC: execute commands in transaction
  • DISCARD: flush commands and exit transaction
  • WATCH: check and set, if watched key changes, not execute

Errors

  • before EXEC: e.g. syntax error
  • after EXEC: e.g. value error

The pipeline discarding the transaction automatically if there was an error during the command queueing

發表留言

Generator

Generator

  • A type of iterator
  • generator function: function that uses yield statement
  • implement the iterator protocal, call next
  • raise StopIteration exhausted

Less code

Implement an iterator

Implement a generator

More efficient

Generator Comprehensions

  • local scope
  • lazy evaluation
  • is an iterator, can be exhausted

Delegating Generator

Use the syntax yield from to yield items in a generator

發表留言

Memcached

Memcache

Store and retrieve data in memory(not persistent) base on specific hash function.

concepts

  • Slab: allocate as many pages as the ones available

  • Page: a memory area of default 1MB which contains as many chunks

  • Chunk: minimum allocated space for a single item

  • LRU: least recently used list

ref: Journey to the centre of memcached

we could say that we would run out of memory when all the available pages are allocated to slabs

memcached is designed to evict old/unused items in order to store new ones

every item operation (get, set, update or remove) requires the item in question to be locked

memcached only tries to remove the first 5 items of the LRU — after that it simply gives up and answers with OOM (out of memory)

commands with telnet

  • get

  • set

  • add: add key or return NOT_STORED if exists

  • replace: replace key or return NOT_STORED if exists

  • append, prepend

  • incr, decr

  • delete

  • flush_all

  • stats

  • version

  • quit

Run Service

Image used: memcached

Python client: pymemcache

Distributed Caching

image

Modulo Hashing

  • Pros: Balancing the distribution between instances in cluster
  • Cons: 1. Loss data if instance down 2. hard to scale

Example

Run 3 instances, expose at port 11211, 11212, 11213

Use python client to set key

Get client instance

pymemcache use Murmur3 hashing

Test with telnet

telnet the third cache server

Consistent Hashing

image

Scale up / down not affect all the servers on the ring

High Availability

  • Repcached: replica data between masters
  • KeepAlive: port forword to slave if master down
發表留言

Iterable and Iterator

Iterator & Iterable

iterator

  • get next item (__next__)
  • no indexes needed (Don’t need to be Sequence type)
  • consumable

iterable

  • collections that implement iterator

Protocal

Python need to count on certain funcionality: __next____iter__StopIteration

compare to sequence type

iteration can be more general than sequential indexing, we only need:

  • a bucket of items: collection, container
  • a way to get the next item, no need to care about ordering
  • an exception to raise if there is no next item

try to custom an iterator ourselfs:

Why re-create?

Seperate the Collection from the iterator

Iterable object

  • Maintaining the data of the collection is one object
  • Created once
  • implements __iter__, return a new iterator instance

Iterator object

  • Iterating over that data should be another object
  • throw away the iterator but don’t throw away the collection
  • Created every time
  • implements __iter__, return itself
  • implements __next__, return next item

iterable can be lazy

Caculate the next itme in an iterable until it’s actually requested

lazy evaluation

  • often used in class properties
  • properties of classes may not always populated when the object is created
  • value of property only becomes known when the property is requested/deferred

infnite iterables

  • itertools.cycle

Python Built-ins

  • range: return iterable
  • zip: return iterator
  • enumerate: return iterator
  • open: return iterator
  • reversed: return iterator

The type is important. Iterator object can be only iter over once.

iter()

when iter is called:

  • Python first looks for __iter__, if not then:
  • look for __getitem__ and create an iterator, if not then:
  • raise TypeError

Test it:

The __iter__ must return an iterator!

Iterating callable

iterator delegation

Example 1

Example 2

發表留言

IP Subnetting

tags: ccna

IP Subnetting

Something you need to know first: Binary Odometer

10.1.1.254 + 1 = 10.1.1.255
10.1.1.255 + 1 = 10.1.2.0
10.1.2.0 + 1 = 10.1.2.1

in reverse:

10.1.2.0 – 1 = 10.1.1.255

Example 1

172.16.35.123/20 or 172.16.35.123 with the mask 255.255.240.0

Binary Method

image alt

Quick Method

Figure out the subnets:

First subnet = 172.16.32.0

Next subnet = 172.16.48.0

Broadcast address = next subnet – 1

172.16.32.0 + 1 = 172.16.32.1`

Last host = Broadcast – 1
172.16.47.255 - 1 = 172.16.47.254

Subnetting

  • Class A subnetting (255.0.0.0) support 1677214 (2^24) host per network, that way too much
  • Class B subnetting (255.255.0.0) support 16382 (2^16) host per network, that way too much
  • Class C subnetting (255.255.255.0) support 254 (2^8) host, more likely we subnet down to at least 254 hosts or even further

If you subnetting a network only has 2 hosts, you can subnet with (255.255.255.254) or CIDR as /31

Network, host number

  • Networks: 2^(network bits)
    • one allocate for the subnet
    • one allocate for the broadcast
  • Hosts: 2^(host bits) – 2

Subnetting to be short

  1. “stealing” or “taking away” bits from the host portion of an address, and
  2. allocating those bits to network portion

Example 2

Origin network 10.128.192.0/18 need at least 30 subnets as many hosts as possible

image

  1. draw the line with /18 to split network and host
  2. 2^5 > 30, need 5 subnet bit, draw the line to split subnet and host
  3. network/subnet portion is 8+8+7=23 bits, host portion is 32-23=9 bits
  • First subnet: 10.128.192.0/23
  • Second subnet: 10.128.194.0/23
  • Last subnet: 10.128.254.0/23
發表留言

第一次開發選服就烙賽

Intro

自從邱威傑市民服務網站上線後,一直想要完成這篇,紀錄一下這一個月多月開發以來的心得感想,但因為專案完成,本身也剛換工作,無預警進入了懈怠期,不知不覺就拖到現在了

晚上和呱吉跟他的政治團隊吃飯,回家後提醒自己寫下紀錄,今天大概就能算是一個圓滿的結束點吧~

18.12.07 確定接下案子

在這之前,第一次和負責的團隊成員——很年輕的法律系學生 @Eddy 見面,一開始當然不知道呱吉會打算怎麼做這個系統,原本以為會有前輩主導開發,去參與討論或插花即可,但實際上是最多找2-3個人做;也以為是要做能讓多個議員註冊、使用的平台,但需求聽起來卻像呱吉個人的宣傳式網站

我認知到一件事,那就是 Eddy 沒辦法準確評估我的技術水準,只知道我有做過網站。令人訝異的是,當天握手之後就直接把任務交給我了。當然,這是風險評估出了問題呢,還是完全信任夥伴?我選擇相信後者。(團隊的風氣感覺是年輕人放手去做就對了,烙賽沒關係?)

專案需求雖然很簡單:使用者填寫表單、送出後可以在後台編輯。但我還是一樣超沒信心,因為認知到:

  • 我希望這是一個開源專案,如果要開源,會有資安上須考量的風險,加上呱吉是公眾人物,容易成為箭靶
  • 短期會有高流量,分散式部署是必要的;自己沒有實際維運過分散系統,大概知道是怎麼做,但沒把握
  • 重點是前端視覺與使用者體驗,但自己頂多做後端,前端雖然會寫,但根本在標準之下

Read more “第一次開發選服就烙賽”

4 則迴響

使用 Cloud Build 搭配 Helm 改善雲端部署

管理 Kubernetes 的服務時常有一些困擾:

  1. 須根據環境 (staging, production…) 去套用不同的設定及環境變數,整合不易
  2. secret 常是手動新增,如 cloudsql-proxy 的憑證,時間久了常忘記該 secret 是幹麻用的,及整個服務重新部署也會卡在這個手動步驟

使用 Helm 可以幫助我們管理這些部署檔案:

  1. 一鍵部署、移除
  2. 可根據不同的環境採用不同變數,有幾種可行的作法
  3. 可根據彈性的判斷式生成設定檔
  4. chart 的版本控制 (Release)

準備事項:

  1. 有個叢集
  2. Client 端安裝 Helm , Server 端安裝 Tiller
    image
  3. 叢集有 RBAC ,可以關閉 RBAC ,或給 Tiller 權限:
    • helm init預設使用的服務帳戶是default
    • 叢集的default服務帳戶綁定cluster-admin叢集角色

Helm 有提供 dependency 的功能,可以透過以下指令來部署全部的 subchart:

但這裡會有一個問題,即部署時沒辦法指定 subchart 要吃哪個 value file ,因若你的 chart 參數是分成values.prod.yamlvalues.staging.yaml,一旦 chart 打包成 package ,package 在使用時只能吃 values.yaml

下個版本可能提供相對應的作法,請參考相關 issue

透過 Cloud Build 部署

透過helm create建立並設定完 chart 後,希望能在 Cloud Build 的流程透過 helm 部署,這邊選用 cloud-builer-community 提供的helm映像檔

  1. Build 該映像檔並推至專案的 Container Registry
  2. 參考 example 來新增流程,如helm installhelm upgrade
  3. 若 RBAC 是啟用的狀態,須要給 Cloud Build 操作叢集的權限
    • Cloud Build 的服務帳戶綁定roles/container.admin角色及cluster-admin叢集角色,請參考相關指令
      image
發表留言

IP Addressing

tags: ccna

IP Addressing

  • layer 3 logical address assigned by an administrator(MAC built in NIC)
  • used to identify specific devices on a network
  • every device on the internet has a unique IP Address

Street Analogy

  • network address portion
    • identifies a specific network
    • routers route traffic via routing tables, is based on network address(Network ID), not ip address
  • host address portion
    • identifies a specific endpoint on a network
    • we can use a protocal such as ARP to find the host

Ipv4

  • connectionless protocal: no sessions formd when transmitted, no status info
  • packets treated independently
    • may take different paths: load balancing, bandwidth, hopcount
  • hierarchical addressing sturture
  • best effort delivery
  • format
    • 32 bit with 4 octets
    • like DHL or FedEx routing parcel based on an address

Classes

  • Unicast Traffic
    • A
      • start with binary 0
      • range from 0.0.0.0(00000000) to 127.255.255.255(01111111)
      • exceptions:
        • 127 is reserved for loopback: 127.0.0.1
        • 0 network is reserved for default network: 0.1.1.1
      • actual range from 1.0.0.0 to 126.255.255.255
      • portions
        • first 1 octets: Networks
        • last 3 octests: Hosts
    • B
      • start with binary 10
      • range from 128.0.0.0(10000000) to 191.255.255.255(10111111)
      • portions
        • first 2 octets: Networks
        • last 2 octests: Hosts
    • C
      • start with binary 110
      • range from 192.0.0.0(11000000) to 223.255.255.255(11011111)
      • portions
        • first 3 octets: Networks
        • last 1 octests: Hosts
  • Multicast
    • D
      • start with binary 111
  • reserved for other purposes
    • E
      • start with binary 1111

These classes replaced by Classless Inter-Domain Routing(CIDR) in 1993

Read more “IP Addressing”

發表留言

OSI Model

tags: ccna

OSI Model

  • By International Organization of Standard

Benefits

  • Standard and INTEROPERABILITY
  • Split development/role: hide developer from lower layer
  • Quicker development

Layers

You need to remember both the name and the layer number

  • Layer7: Application
  • Layer6: Presentation
  • Layer5: Session
  • Layer4: Transport
  • Layer3: Network
  • Layer2: DataLink
  • Layer1: Physical

Trick: All People Sleeping Through Networking Don’t Pass

Network Engineer: Focus on 1, 2, 3, 4 Layers
Web Developer: Focus on 5, 6, 7 Layers

Read more “OSI Model”

發表留言

序列 – Python Sequence

Sequence

必須足以下條件:

  • 物件的集合
  • countable
  • zero based indexing (__getitem__)
  • 為什麼 index 要從0開始?
    • 0 based: 0 <= n < len(s)
    • 1 based: 1 <= n < len(s) + 1 Upper bound 用小於的原因是計算長度時不須再+1
  • 有序(positional ordering)
    • 舉例來說 list 和 set 都是物件的容器,但 list 可以被排序, set 不行,因此 list 是 Sequence Type 而 set 不是

特性:

  • Homogeneous vs Heterogeneous 同質即序列的物件型態必須是相同的
  • Iterable vs non iterable 可以迭代的容器不一定是序列,如set
  • Mutable vs Immutable Mutable sequence can be modified. 要注意的是在操作新序列的時候更動到原本的序列(in-place),如 reverse()

以 list 為例,這幾個操作皆為原地算法(inplace):

  • l.clear()
  • l.append()
  • l.pop()
  • l.extend()
  • l.insert()
  • l +=
  • l *=
  • l[somesliceobj] = 若是 concat(+)、repetition(*)、slicing 都是關聯至新的物件參考

要注意的是,容器序列(儲存物件參考)的 concat 和 repetition 有可能只是儲存多個相同物件的參考

Read more “序列 – Python Sequence”

發表留言