Python 的concurrent.futures库如何接受关键词参数？

concurrent.futures这个库极大简化了创建多进程、多线程的代码工作，常用的调用方式如下：

并发执行不同的函数例子

from concurrent import futures
import time

def test1(num):
    time.sleep(2)
    return time.ctime(),num

def test2(num):
    time.sleep(2)
    return time.ctime(),num

with futures.ProcessPoolExecutor() as executor:
    future1 = executor.submit(test1,1) 
    future2 = executor.submit(test2,2) 
    print(future1.result())
    print(future2.result())

并发执行同一个函数，只是传参不一样

from concurrent import futures
import pendulum
import time 

def test(task_num):
    time.sleep(1)
    return pendulum.now().to_datetime_string(), task_num

task_num = range(400)
with futures.ProcessPoolExecutor() as executor:
    for future in executor.map(test, task_num, timeout=10):  
        pass

那假如要并发执行某个函数的参数很多，不想每次调用都写全所有的参数，只想通过关键词传某些参数呢？
这个网上已经有回答了，譬如How to use executor.map function in python on keyword arguments - Stack Overflow

def func(site, search_term, pages):
    ...

from functools import partial
from itertools import repeat
executor.map(func, zip(sites, repeat(search_term), repeat(pages)))

原理是通过 repeat 函数将本来无需传入的参构造个默认值
还有这个python - How to pass keyword argument to function called by concurrent.futures map call - Stack Overflow

res = executor.map(lambda x,y:spam(x,params=y), urls, params)

给要调用的函数用匿名函数 lambda 再封装一层

这个 2 个方法都用了executor.map()，是个函数式编程的函数，《流畅的 Python》里曾讲过 map大部分情况下都能被推导式代替，我想了个方法如下：

from concurrent import futures
import pendulum
import time


def test(task_num=0, name='test', n=0):
    time.sleep(1)
    return pendulum.now().to_datetime_string(), task_num, name, n


a = ['jack', 'Tom']
b = [9, 8]
with futures.ProcessPoolExecutor() as executor:
    future_to_stuff = [executor.submit(test, name=name, n=n)
                       for name, n in zip(a, b)]

    for future in future_to_stuff:
        print(future.result())

上面的代码只用了 concurrent.futures的executor.submit，每个不同的关键词传参就调用一次executor.submit，通过列表推导式将多个调用聚集起来，最后再遍历这个列表推导式就行了。

最后编辑于：2021.06.25 15:12:27