Skip to content

Multiprocessing: Pickle Issue

Tags:

multiprocessing use pickle module to serialize things among the process, but doesn’t support functions with closures, lambdas, or functions in __main__

Here is the example I try:

solution: dill

dill and multiprocessing: pathos

dill: a utility to serialize all of python
– pox: utilities for filesystem exploration and automated builds
– klepto: persistent caching to memory, disk, or database
– multiprocess: better multiprocessing and multithreading in python
– ppft: distributed and parallel python
– pyina: MPI parallel map and cluster scheduling
– pathos: graph management and execution in heterogenous computing

 

1 thought on “Multiprocessing: Pickle Issue”

發佈留言

發佈留言必須填寫的電子郵件地址不會公開。 必填欄位標示為 *