Multiprocess process.start() not doing anything?

[Hana-Ali]

Hello,

I have 5 separate functions I want to execute in parallel:

# Defining the BO acquisition function
acquisition_function = UtilityFunction(kind="ei",xi=1e-1)

# Defining a wrapper function that calls maximize - wilson
def wilson_wrapper():
    print('in wilson wrapper')
    wilson_optimizer.maximize(
        n_iter=n_iter_BO,
        init_points=init_points_BO,
        acquisition_function=acquisition_function
    )

# Defining a wrapper function that calls maximize - wong
def wong_wrapper():
    print('in wong wrapper')
    wong_optimizer.maximize(
        n_iter=n_iter_BO,
        init_points=init_points_BO,
        acquisition_function=acquisition_function
    )

# Defining a wrapper function that calls maximize - jansen
def jansen_wrapper():
    print('in jansen wrapper')
    jansen_optimizer.maximize(
    n_iter=n_iter_BO,
    init_points=init_points_BO,
    acquisition_function=acquisition_function
    )

# Defining a wrapper function that calls maximize - larter
def larter_wrapper():
    print('in larter wrapper')
    larter_optimizer.maximize(
    n_iter=n_iter_BO,
    init_points=init_points_BO,
    acquisition_function=acquisition_function
    )

# Defining a wrapper function that calls maximize - kuramoto
def kuramoto_wrapper():
    print('in kuramoto wrapper')
    kuramoto_optimizer.maximize(
    n_iter=n_iter_BO,
    init_points=init_points_BO,
    acquisition_function=acquisition_function
    )

in a jupyter notebook. For some reason, when I try to define each as a process in this way:

from multiprocess import Process

if __name__ == '__main__':
    wilson_process = Process(target=wilson_wrapper)
    wong_process = Process(target=wong_wrapper)
    jansen_process = Process(target=jansen_wrapper)
    larter_process = Process(target=larter_wrapper)
    kuramoto_process = Process(target=kuramoto_wrapper)

    wilson_process.start()
    wong_process.start()
    jansen_process.start()
    larter_process.start()
    kuramoto_process.start()

    wilson_process.join()
    wong_process.join()
    jansen_process.join()
    larter_process.join()
    kuramoto_process.join()

Nothing happens, with or without if __name__ == '__main__' (cell just runs in 0.3 seconds, and no print() statements print). I tried defining it in a different way, where I make one function and a list of all the optimizer objects that will be called:

def main_wrapper(optimizer, n_iter_BO, init_points_BO, acquisition_function):
    print(optimizer)
    optimizer.maximize(
        n_iter=n_iter_BO,
        init_points=init_points_BO,
        acquisition_function=acquisition_function
    )

optimizers_list = [wilson_optimizer, wong_optimizer, jansen_optimizer, larter_optimizer, kuramoto_optimizer]

then with:

from multiprocess import Process

max_pool = 5    
with Pool(max_pool) as p:
    pool_outputs = list(
        tqdm(
            p.imap(partial(main_wrapper, n_iter_BO, init_points_BO, acquisition_function),
                optimizers_list),
            total=len(optimizers_list)
        )
    )

# Getting the results of the 5 repetitions from the process outputs
multiprocessing_outputs = list(zip(*pool_outputs))
multiprocessing_outputs = list(multiprocessing_outputs[1])
print(multiprocessing_outputs)

And this also doesn't work. Is there any way to run 5 independent functions in a Jupyter Notebook? Thank you so much for your help!!

[mmckerns]

Can you reproduce the same behavior in a minimal self-contained reproducible example? https://stackoverflow.com/help/minimal-reproducible-example

Same question here:
https://stackoverflow.com/questions/76085168