Python

Python Type Hierarchy

2018 年 6 月 2 日
Python

Number

Integral: Integer, Booleans
Non-Integral: Floats, Complex, Decimals, Fractions(1/3)

Collection

Sequences
- Mutable: List
- Immutable: Tuples, Strings
Sets
- Mutable: Sets
- Immutable: Frozen Sets
Mappings
- Dictionaries (relate to set)

Callables

Built-in Functions
User-Defined Fuctions
Instance Methods (e.g. len())
Built-in Method (e.g. my_list.append(x))
Generators
Classes
Class Instances(__call__())

Singletons

None
NotImplemented
Elipsis operators

Multiprocessing: Pickle Issue

2017 年 11 月 23 日
1 Comment
Python

multiprocessing use pickle module to serialize things among the process, but doesn’t support functions with closures, lambdas, or functions in __main__

In [1]: import pickle

In [2]: pickle.dumps(lambda x: x)

PicklingError: Can't pickle <function <lambda> at 0x0000000005AC8048>: attribute lookup <lambda> on __main__ failed

Here is the example I try:

In [1]: from multiprocessing import Pool

class C(object):

def __init__(self):

self.x = 2

def method(self):

def square():

return self.x**2

p = Pool(4)

p.map(square, range(10))

c = C()

c.method()

AttributeError: Can't pickle local object 'C.method.<locals>.square'

solution: dill

In [1]: import dill

dill.dumps(lambda x: x)

Out[1]: b'\x80\x03cdill.dill\n_create_function\nq\x00(cdill.dill\n_load_type\nq\x01X\x08\x00\x00\x00CodeTypeq\x02\x85q\x03Rq\x04(K\x01K\x00K\x01K\x01KCC\x04|\x00S\x00q\x05N\x85q\x06)X\x01\x00\x00\x00xq\x07\x85q\x08X\x1f\x00\x00\x00<ipython-input-14-80b851ab3476>q\tX\x08\x00\x00\x00<lambda>q\nK\x02C\x00q\x0b))tq\x0cRq\rc__builtin__\n__main__\nh\nNN}q\x0etq\x0fRq\x10.'

dill and multiprocessing: pathos

– dill: a utility to serialize all of python
– pox: utilities for filesystem exploration and automated builds
– klepto: persistent caching to memory, disk, or database
– multiprocess: better multiprocessing and multithreading in python
– ppft: distributed and parallel python
– pyina: MPI parallel map and cluster scheduling
– pathos: graph management and execution in heterogenous computing

from pathos.multiprocessing import Pool

p = Pool(4)

p.map(lambda x: x**2, range(10))

Serialize and Deserialze Sqlalchemy Nested Objects with JSON

2017 年 11 月 1 日
Python

At first we build a relationship :

from sqlalchemy import Base, Unicode

class County(Base):

__tablename__ = 'county'

name = Column(Unicode(5))

towns = relationship('Town')

class Town(Base):

__tablename__ = 'town'

name = Column(Unicode(5))

county = relationship('County', back_populates='towns')

and we want to dump a county instance to json like this:

1	{"name": "台北市", "towns": [{"name": "大安區"},{"name": "文山區"}]}

extending JSONEncoder:

class Encoder(JSONEncoder):

def default(self, obj):

d = {}

if isinstance(obj, County):

return {'name': obj.name, 'towns': obj.towns}

if isinstance(obj, Town):

return {'name': obj.name}

d.update(obj.__dict__)

return d

1	json.dumps(county_instance, cls=Encoder)

if you want to decode your json file, simply supply a function for object_hood attribute:

def hook(dict):

if dict.get('towns'):

county = County()

county.__dict__.update(dict)

return county

else:

town = Town()

town.__dict__.update(dict)

return town

1	json.load(file, object_hook=hook)

object_hook is an optional function that will be called with the result of any object literal decoded (a dict). The return value of object_hook will be used instead of the dict. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting).

when a json file loads, we can print the parameter “dict” which pass in to object_hood function, the whole process will loads from the most inner object (a dict) to the outer objects look like this:

{"name": "文山區"}

{"name": "大安區"}

{"name": "台北市", "towns": [

<Town object at 0x0000018FFE817B38>,

]}

Python Beyond the Basics

2017 年 10 月 27 日2019 年 8 月 23 日
Python

Entities

class: blueprint for an instance
instance: a constructed object of the class
type: indicates the class/instance belongs to
attribute: value of any object
method: a callable attribute defined in the class

Instance Methods

instance methods are variables define in the class
when call through the instance, the instance is automatically passed as first argument (named self) to the method, e.g. below:
because of this automatic passing of the instance, instance method are known as “bound” methods, i.e. bound to the instance upon which it is called

class Car(object):

def break(): # note not passing 'self' here

pass

car = Car()

car.break()

# TypeError: break() takes no arguments(1 given)

Encapsulation

breaking encapsulation: setting the object value without using the setter method

class Integer(object):

def set_value(self, val):

try:

val = int(val)

except ValueError:

return

self.val = val

enforce encapsulation:

class Integer(object):

def __init__(self, val):

try:

val = int(val)

except ValueError:

val = 0

self.val = val

Python Class Variable v.s. Instance Attribute

Set class variable and instance attribute the same name to show the difference and look up order:

class SomeClass(object):

val = 'this is a class variable'

instance = SomeClass()

print(instance.val) # this is a class variable

instance.val = 'this is a instance variable'

print(instance.val) # this is a instance variable

del instance.val

print(instance.val) # this is a class variable<span id="mce_marker" data-mce-type="bookmark" data-mce-fragment="1"></span>

The attribute look up order: when request an attribute to an instance, it looks for that attribute in the instance first then in the class.

Access class variable in instance:

class SomeClass(object):

val = 0

def __init__(self):

SomeClass.val += 1

Polymorphism

Different classes has same interface (i.e., method name), we can say that this group of related classes implement the same action
Being able to work with different types to call methods without checking it’s possible and handle errors using exception handling is called DOCK TYPING
Checking the type ahead of time is generally considered un-pythonic

An example from stackoverflow:

class Person(object):

def pay_bill():

raise NotImplementedError

class Millionare(Person):

def pay_bill():

print "Here you go! Keep the change!"

class GradStudent(Person):

def pay_bill():

print "Can I owe you ten bucks or do the dishes?"

or an built in len method as an interface implemented by list and set object:

l = [1, 2, 3]

s = (1, 2, 3)

l.__len__()

/*3*/

s.__len__()

/*3*/

Inheriting the Constructor

__init__ can be inherited
if a class doesn’t have an __init__ constructor, Python will check its parent class, as soon as it finds one, Python calls it and stops looking
we can use the super() function to call method in the parents class, or we may want to initialize in the parent class

class Animal(object):

def __init__(self, name):

self.name = name

class Dog(Animal):

def __init__(self, name):

super(Dog, self).__init__(name)

'''

if the name of Animal class change in the future

or rearrange the class hierarchy that Dog class

inherite from another class and we want to call

that constructor instead

'''

Depth-First Order

Multiple inheritance

class A(object):

def run(self):

print('run in A')

class B(A):

pass

class C(object):

def run(self):

print('run in C')

class D(B, C):

pass

D.mro()

'''

the order is : [<class '__main__.D'>, <class '__main__.B'>, <class '__main__.A'>, <class '__main__.C'>, <class 'object'>]

'''

After Python 2.3, add a role for diamond shape ambiguous inheritance

(Make C inherit A in this example)

class A(object):

def run(self):

print('run in A')

class B(A):

pass

class C(A):

def run(self):

print('run in C')

class D(B, C):

pass

D.mro()

'''

if the same class appears in a method resolution order, the earlier occurrences are removed

from [__main__.D, __main__.B, __main__.A, __main__.C, __main__.A, object]

to [__main__.D, __main__.B, __main__.C, __main__.A, object]

'''

Abstract Class

An abstract class is not designed to and can’t not construct instances, but subclassed by regular classes
An interface / methods that defined in abstract class must be implement by its subclass
The python abc module enables the creation of abstract class

import abc

class GetterSetter(object):

__metaclass__ = abc.ABCMeta

@m.abstractmethod

def set_val(self, input):

return

@m.abstractmethod

def get_val(self):

return

class MyGetterSetter(GetterSetter):

def set_val(self, input):

self.val = input

def get_val(self):

return self.val<span id="mce_marker" data-mce-type="bookmark" data-mce-fragment="1"></span>

Composition

Inheritance could be brittle (a change may require changes elsewhere)
Decoupled code could work independently or interactively
classes interactions will work with an interface
Not checking types: polymorphic and Pythonic

import StringIO

class FileWriter(object):

def __init__(self, file):

self.file = file

def write(self, text):

# no type checking here as long as the interface stay the same

self.file.write(text)

f = open('text.txt', 'w')

w1 = FileWriter(f)

w1.write('writing into a file object...')

w1.close

sio = String.StringIO()

w2 = FileWriter(sio)

w2.write('writing into a string io object...')

Here is an example for inheritance and composition working together:

from myappengine import webfunc, run_app

# inheritance: create class inherit from specific class

class MainPage(webfunc.RequestHandler):

def get(self):

self.response.out.write('Here is the MainPage')

class FAQPage(webfunc.RequestHandler):

def get(self):

self.response.out.write('Here is the FAQ Page')

# composition: pass class to specific app engine function as an interface

app = webfunc.WSGIApplication(

[('/', MainPage),

('/faq', FAQPage)]

)

def main():

run_app(app)

Magic Functions

In [1]: class Dog(object):

def __init__(self, color, age, name):

if not isinstance(age, int): raise ValueError

self.color = color

self.age = age

self.name = name

def __eq__(self, other_dog):

return self.name == other_dog.name and self.color == other_dog.color

def __lt__(self, other_dog):

return self.age < other_dog.age

def __gt__(self, other_dog):

return self.age > other_dog.age

mike = Dog('black', 1, 'Mike')

jack = Dog('white', 4, 'Jack')

michael = Dog('black', 10, 'Mike')

In [2]: mike < jack

Out[1]: True

In [3]: mike == michael

Out[2]: True

Attribute Encapsulation

@property should not encapsulate expensive operations, because attribute setting looks cheap
@property only controls attributes that are expected

class Me(object):

def __init__(self, var):

self._var = var

# encapsulate _var attribute to var

@property

def var(self)

return self._var

@var.setter

def var(self, var):

self._var = var

@var.deleter

def var(self)

self._var = None

In [1]: me = Me(1)

In [2]: me.var

Out[2]: 1

In [3]: me.var = 2

In [4]: del me.var

“with” in Python

class MyClass(object):

def __enter__(self):

print('We have entered "with"')

return self

# Use "with" to capture errors

def __exit__(self, type value, traceback):

print('We have leaved "with"')

print('error type: {0}'.format(type))

def do(self):

print('Do something...')

In [1]: with MyClass() as c:

c.do()

Out[1]: We have entered "with"

Do something...

We have leaved "with"

In [2]: with MyClass() as c:

c.do()

5/0

Out[2]: error type: <type 'exceptions.ZeroDivisionError'>

Custom Exception

class MyError(Exception):

def __init__(self, **kwargs):

self.msg = kwargs.get('msg')

def __str__(self):

if self.msg:

return 'Here\'s MyError Message: %s' % self.msg

else:

return 'Here\'s MyError'

In [1]: raise MyError(msg='Huston, we have a problem.')

Out[1]: ---------------------------------------------------------------------------

MyError Traceback (most recent call last)

<ipython-input-6-de465e9d3c17> in <module>()

9 return 'Here\'s MyError'

---> 11 raise MyError(msg='Huston, we have a problem.')

MyError: Here's MyError Message: Huston, we have a problem.

« 上一頁
1
...
4
5
6