S Sivart 八月 23, 2019 八月 27, 2019 Python

Generator as Coroutines

cooperative multitasking (cooperative routines)
concurrent not parallel (python program execute on a single thread)

The way to create coroutines:

generators (asyncio)
native coroutines (using async /await)

Concepts

concurrency: tasks start, run and complete in overlapping time periods
parallelism: tasks run simultaneousely

cooperative: control relinquished to other task voluntarily, control by application(developer)
preemptive: control relinquished to other task involuntarily, control by the OS.
some sort of scheduler involved

Global Interpreter Lock(GIL)
Only one native thread excutes at a time.

Use Process based parallelism to avoid GIL. Not Thread based.

The Python threading module uses threads instead of processes. Threads uniquely run in the same unique memory heap. Whereas Processes run in separate memory heaps. This makes sharing information harder with processes and object instances. One problem arises because threads use the same memory heap, multiple threads can write to the same location in the memory heap which is why the global interpreter lock(GIL) in CPython was created as a mutex to prevent it from happening.

Make the right choice

CPU Bound => Multi processing
I/O Bound, Fast I/O, Limit Connections => Muilti Threading
I/O Bound, Slow I/O, Many Connections => Concurrency

Use deque

Much more efficient way to implement the stack and queue.

Operate 10,000 items take 1,000 times average:

(times in seconds)	list	deque
append(right)	0.87	0.87
pop(right)	0.002	0.0005
insert(left)	20.8	0.84
pop(left)	0.012	0.0005

Use unlimited deque with deque() or deque(iterable)
Use limited deque with deque(maxlen=n). If full, a corresponding number of items are discarded from the opposite end.

Implement producer / consumer coroutine using deque

Implement simple event loop

Read more “Generator as Coroutines”

發表留言

S Sivart 八月 20, 2019 八月 23, 2019 Python

Context Manager

what is context

the state surrounding a section of code

why we need a context manager

writing try/finally every time can get cumbersom
easy to forget closing the file

use cases

Useful for program that needs Enter / Exit handeling

create / releasing resources
database transaction
set and reset decimal context

Common patterns

open / close
lock / release
change / reset
start / stop
enter / exit

protocal

implement these two dunder methods:

__enter__
perform the setup, optionally return an object
__exit__
receives error (silence or propagate)
- need arguments exc_type, exc_value, exc_trace to handle exception
- return True to silence exception
perform clean up

examples

contextlib

nested contexts

發表留言

S Sivart 七月 17, 2019 八月 23, 2019 Python

Generator

A type of iterator
generator function: function that uses yield statement
implement the iterator protocal, call next
raise StopIteration exhausted

Less code

Implement an iterator

1

2

3

4

5

6

7

8

9

10

11

12

13

class FactIter:

def __init__(self, n):

self.n = n

self.i = 0

def __iter__(self):

return self

def __next__(self):

if self.i >= self.n:

raise StopIteration

result = math.factrial(self.i)

self.i += 1

return result

Implement a generator

1

2

3

4

def factorials(n):

for i in range(n):

yield math.factorial(i)

More efficient

Generator Comprehensions

1 2	(i**2 for i in range(5))

local scope
lazy evaluation
is an iterator, can be exhausted

Delegating Generator

Use the syntax yield from to yield items in a generator

1

2

3

4

for file in ('a.csv', 'b.csv', 'c.csv'):

with open(file) as f:

yield from f

發表留言

S Sivart 六月 16, 2019 七月 13, 2019 Python

Iterable and Iterator

Iterator & Iterable

iterator

get next item (__next__)
no indexes needed (Don’t need to be Sequence type)
consumable

iterable

collections that implement iterator

Protocal

Python need to count on certain funcionality: __next__、__iter__、StopIteration

compare to sequence type

iteration can be more general than sequential indexing, we only need:

a bucket of items: collection, container
a way to get the next item, no need to care about ordering
an exception to raise if there is no next item

try to custom an iterator ourselfs:

Why re-create?

Seperate the Collection from the iterator

Iterable object

Maintaining the data of the collection is one object
Created once
implements __iter__, return a new iterator instance

Iterator object

Iterating over that data should be another object
throw away the iterator but don’t throw away the collection
Created every time
implements __iter__, return itself
implements __next__, return next item

iterable can be lazy

Caculate the next itme in an iterable until it’s actually requested

lazy evaluation

often used in class properties
properties of classes may not always populated when the object is created
value of property only becomes known when the property is requested/deferred

infnite iterables

itertools.cycle

Python Built-ins

range: return iterable
zip: return iterator
enumerate: return iterator
open: return iterator
reversed: return iterator

The type is important. Iterator object can be only iter over once.

iter()

when iter is called:

Python first looks for __iter__, if not then:
look for __getitem__ and create an iterator, if not then:
raise TypeError

Test it:

1

2

3

4

5

try:

iter(obj)

except TypeError:

print('Not iterable ...')

The __iter__ must return an iterator!

Iterating callable

iterator delegation

Example 1

Example 2

發表留言

S Sivart 二月 22, 2019 四月 24, 2019 Python

序列 – Python Sequence

Sequence

必須足以下條件：

物件的集合
countable
zero based indexing (__getitem__)
為什麼 index 要從0開始？
- 0 based: 0 <= n < len(s)
- 1 based: 1 <= n < len(s) + 1 Upper bound 用小於的原因是計算長度時不須再+1
有序(positional ordering)
- 舉例來說 list 和 set 都是物件的容器，但 list 可以被排序， set 不行，因此 list 是 Sequence Type 而 set 不是

特性：

Homogeneous vs Heterogeneous 同質即序列的物件型態必須是相同的
Iterable vs non iterable 可以迭代的容器不一定是序列，如set
Mutable vs Immutable Mutable sequence can be modified. 要注意的是在操作新序列的時候更動到原本的序列(in-place)，如 reverse()

以 list 為例，這幾個操作皆為原地算法（inplace）：

l.clear()
l.append()
l.pop()
l.extend()
l.insert()
l +=
l *=
l[somesliceobj] = 若是 concat(+)、repetition(*)、slicing 都是關聯至新的物件參考

要注意的是，容器序列（儲存物件參考）的 concat 和 repetition 有可能只是儲存多個相同物件的參考

Read more “序列 – Python Sequence”

發表留言

S Sivart 十二月 24, 2018 八月 23, 2019 Python / Web Develop

Python WSGI 及其應用

tags: `python`, `wsgi`, `gunicorn`, `uwsgi`

Python WSGI及其應用

WSGI是什麼？

全名Python Web Server Gateway Interface
基於CGI
Python用來描述了伺服器和請求處理程式之間傳輸資料的一種標準（協議）

PEP3333

基本原理跟目標

server、app、framework共用的界面

…proposes a simple and universal interface between web servers and web applications or frameworks: the Python Web Server Gateway Interface (WSGI).

不再重造輪子

…the goal of WSGI is to facilitate easy interconnection of existing servers and applications or frameworks, not to create a new web framework

只會支援python既有的release

…precludes WSGI from requiring anything that is not already available in deployed versions of Python

未來會增加部署的標準

…current version of WSGI does not prescribe any particular mechanism for “deploying” an application for use with a web server or server gateway. At the present time, this is necessarily implementation-defined by the server or gateway

Generator as Coroutines

Concepts

Make the right choice

Use deque

Context Manager

Generator

Less code

More efficient

Generator Comprehensions

Delegating Generator

Iterator & Iterable

iterator

iterable

Protocal

compare to sequence type

Why re-create?

iterable can be lazy

Python Built-ins

iter()

Iterating callable

iterator delegation

Sequence

tags: python, wsgi, gunicorn, uwsgi

Python WSGI及其應用

WSGI是什麼？

PEP3333

基本原理跟目標

tags: `python`, `wsgi`, `gunicorn`, `uwsgi`