Multiprocessing: Pickle Issue

multiprocessing use pickle module to serialize things among the process, but doesn’t support functions with closures, lambdas, or functions in __main__

Here is the example I try:

solution: dill

dill and multiprocessing: pathos

dill: a utility to serialize all of python
– pox: utilities for filesystem exploration and automated builds
– klepto: persistent caching to memory, disk, or database
– multiprocess: better multiprocessing and multithreading in python
– ppft: distributed and parallel python
– pyina: MPI parallel map and cluster scheduling
– pathos: graph management and execution in heterogenous computing

 

1 則迴響

Serialize and Deserialze Sqlalchemy Nested Objects with JSON

At first we build a relationship :

and we want to dump a county instance to json like this:

extending JSONEncoder:

if you want to decode your json file, simply supply a function for object_hood attribute:

object_hook is an optional function that will be called with the result of any object literal decoded (a dict). The return value of object_hook will be used instead of the dict. This feature can be used to implement custom decoders (e.g. JSON-RPC class hinting).

when a json file loads, we can print the parameter “dict” which pass in to object_hood function, the whole process will loads from the most inner object (a dict) to the outer objects look like this:

發表留言