Web Develop

深入理解 Nginx 讀書筆記 (第一部份)

為什麼選擇 Nginx

  1. 更快: 1)單次請求更快響應 2) 在高峰期比其他服務器更快響應
  2. 高擴展性: 1)由耦合度極低模塊組成 2)模塊皆嵌入到2進制文件中執行
  3. 高可靠性: 1)模塊穩定 2)進程相對獨立 3)worker出錯可快速輪替
  4. 低內存消耗: 1)10,000個非活躍 HTTP Keep-Alive 連接僅消耗 2.5 MB
  5. 高併發: 1)單機支援 100,000 以上連接
  6. 熱部署: 1)基於 master 與 worker 進程分離 2)服務不間斷下,進行升級可執行元件、配置及更換日誌
  7. BSD 許可協議

開發準備工作

必要

  1. Linux 內核版本 2.6 以上 (須靠 epoll 處理高併發)
  2. GCC 編譯器編譯 C 語言

非必要

  1. G++,用來編譯 C++ 以編寫 HTTP 模塊
  2. PCRE(Perl 兼容正則表達式),用來在配置文件中使用正則表達式,pcre-devel 是使用 PCRE 做二次開發所需
  3. zlib, 用來對 HTTP 內容做 gzip 壓縮,減少網路傳輸量
  4. OpenSSL,支持 SSL 協議,或想使用 MD5 或 SHA 雜湊

目錄結構

  1. 源代碼目錄
  2. 編譯中間文件(置於源碼目錄底下,命名為objs)
  3. 部署目錄(莫認為 /usr/local/nginx)
  4. 日誌目錄

Linux 內核參數優化

  1. 須要修改內核參數,使得 Nginx 可以擁有更高的性能
  2. 通常根據業務特性進行調整,作為內容服務器、反向代理,或是提供縮圖用的服務器,會做不同調整

通用高併發 /etc/sysctl.cnf 設置如下,執行 systcl -p 套用:

每個TCP連接都會為了維持滑動窗口而消耗內存(滑動窗口,接收方通過通告發送方自己的窗口大小,從而控制發送方的發送速度),參數 rmem_default/wmem_default/rmem_max/wmem_max 須要衡量物理內存的總大小及最大連接數量 (workerprocess * workerconnections) 來決定,滑動窗口過大可能導致 OOM,過小會影響大數據量的傳輸速度

編譯安裝 Nginx

下載源碼後,最簡單的安裝方式為執行以下3行指令:

  1. ./configure: 1) 檢測系統內核及軟件 2) 解析參數 3) 生成中間目錄 4) 根據參數生成 C 源碼文件及 Makefile 文件
  2. make: 1)使用生成的 Makefile 進行編譯 2) 生成目標文件及最終2進制文件
  3. make install: 1) 根據參數將 nginx 部署到指定的安裝目錄 2) 複製二進制文件及配置文件

補充

以下是我練習使用的 dockerfile,按照書上指式的步驟在 centos 系統上安裝 nginx:

你可以在目錄下執行以下指令來進入安裝 nginx 後的環境

編譯完後檢視編譯中間文件

安裝完後檢視部署目錄

命令行控制

默認方式啟動二進制程序,不指定 -c 路徑的情況下,會套用默認的配置文件

檢視程序

關閉服務,使用 quit 參數則等待當前請求處理完畢才停止

運行中重新加載配置,指令會先檢查配置是否有誤,若通過則以 quit 方式重啟

滾動升級 Nginx

  1. 發送訊號通知舊版服務
  2. 啟動新版本服務
  3. 以 quit 方式結束舊服務

Rabbitmq Exchange and its Types

Definition

exchange

Message routing agents.

Messages are not published directly to a queue. Instead, the publisher sends messages to an exchange.

Defined by the virtual host within RabbitMQ.

Predefined default exchanges are created at server starts.

Clients can create their own exchanges.

Exchange parameters:

  • type
  • name
  • durability
  • auto-delete (once last bound object is unbound from the exchange)

binding

Link between an exchange and a queue.

Use message attributes: routing key, header to route to the messages to queue(s).

Exchange Types

default exchange

No name. (usually referred by an empty string)

Every queue is automatically bound to the default exchange.

Binding key will be the queue name.

direct exchange

Similar to default exchange, but got a name and the bindings are not created automatically.

A message goes to the queue(s) with the binding key that exactly matches the routing key of the message.

fanout exchange

Routing key doesn’t have any effect.

Route message to all queues.

topic exchange

Similar to direct exchange, but use binding pattern not binding key.

A message goes to the queue(s) with the binding pattern that matches the routing key of the message.

image

header exchange

Similar to direct exchange, but use message header not routing key; use binding header not binding key

If x-match=all, then all header attributes should match

If x-match=any, then any header attributes match counts

image

Google Ads API Beta(v2.1) Short Notes

Prerequisites

  • Developer token
    • generate from a Google Ads Manager Account
    • you can use the same token for not linked account
    • token state: pending, approved
    • only approved token can connect to the API for production
    • Use any developer token if you using Test Manager Account view doc
  • Custermer ID
    • the account number of an Google Ads account
    • can be set or replace programmatically in the SDK
  • OAuth2 client credential
    • generate from a GCP Project/API Console
    • use for managing the API users
  • Client Library

Basic first call (installed application flow)

Step 1

  1. register an Google Ads Manager Account for production use
  2. take the developer token from UI
    > TOOL > SETTINGS > API Center

step refs:

You must use a production (non-test) manager account’s developer token to make API calls against a test account. Even if the token is pending approval, you can still use it to make calls against test accounts.

Step 2

  1. create a Google Account for testing
  2. use the Google Account to register a Google Ads Manager Account
  3. login and create one customer account (the customer created by test manager account will be test only)
  4. create a campaign under the customer account

step refs:

  • https://developers.google.com/google-ads/api/docs/first-call/overview#test_account
  • https://developers.google.com/adwords/api/docs/guides/accounts-overview#test_accounts

Step 3

If already has an OAuth client:

  1. assume the client ID is ORIGIN.apps.googleusercontent.com
  2. go to API console, find the OAuth2.0 client ID then download the client secret json file
  3. use this secret to request a generate refresh token

else:

  1. create a GCP project
  2. enable the GoogleAds API in API console page
  3. create a OAuth client, assume the client ID is NEW.apps.googleusercontent.com
  4. download the client secret json file
  5. use this secret to request a generate refresh token

step refs:

  • https://developers.google.com/adwords/api/docs/guides/accounts-overview#usingoauth2withtestaccounts

When requesting an OAuth2 refresh token, make sure you’re logged in as the test manager account user

To access a test account using OAuth2, the test manager account user must grant permission to your client application. Therefore, when requesting a refresh token, ensure you’re logged in as the test manager account rather than the production manager account.

you want to switch from the test manager account to the production manager account, simply reconfigure the client library to use the production manager account’s refresh token.

Step 4

  1. install client library
  2. create an sdk config file google-ads.yml and insert the values:
    • developer_token
    • client_id
    • client_secret
    • refresh_token
    • logincustomerid
  3. initial a client object that loads the config file
  4. make the first call to search the campaign we just created

Read More »Google Ads API Beta(v2.1) Short Notes

Python WSGI 及其應用

tags: python, wsgi, gunicorn, uwsgi

Python WSGI及其應用

WSGI是什麼?

  • 全名Python Web Server Gateway Interface
  • 基於CGI
  • Python用來描述了伺服器和請求處理程式之間傳輸資料的一種標準(協議)

PEP3333

基本原理跟目標

server、app、framework共用的界面

…proposes a simple and universal interface between web servers and web applications or frameworks: the Python Web Server Gateway Interface (WSGI).

不再重造輪子

…the goal of WSGI is to facilitate easy interconnection of existing servers and applications or frameworks, not to create a new web framework

只會支援python既有的release

…precludes WSGI from requiring anything that is not already available in deployed versions of Python

未來會增加部署的標準

…current version of WSGI does not prescribe any particular mechanism for “deploying” an application for use with a web server or server gateway. At the present time, this is necessarily implementation-defined by the server or gateway

Read More »Python WSGI 及其應用

初始Django專案並客製化設定檔 – Django Setup With Split Settings

  • Web Develop

Django Setup 專案初始化

假設我們想要建立一個django專案(1.9版本,linux為例),命名為mysite,並且不想要將特定的設定上傳,可以如何做:

clone你的倉庫(myrepo)至本地

建立虛擬環境

可以透過指令或IDE,我習慣會將環境命名為venv,然後放在專案目錄下

安裝django並開始一個專案

初始化django專案

別忘了把你安裝的套件加到mysite/requirements.txt

這時候你的tree應該長這樣

Read More »初始Django專案並客製化設定檔 – Django Setup With Split Settings

Django: Create an instance of Manager with a copy of a custom QuerySet

Sometimes we want to query by custom filter and different orders

And in this case will raise AttributeError because Queryset object can not call Manager method

Solution: as_manager()

More Django settings from document: