issue with files alis_*phy

[sguir]

Hi there,

I am running Broccoli with an expanded dataset (around 3000 multifasta files from bacteria) as input. The expanded dataset containd a subset that I have used before as input and it worked fine. The only error reported was:

multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "path/envs/env-broccoli/lib/python3.6/multiprocessing/pool.py", line 119, in worker
result = (True, func(*args, **kwds))
File "path/envs/env-broccoli/lib/python3.6/multiprocessing/pool.py", line 44, in mapstar
return list(map(*args))
File "path/software/Broccoli-1.2/scripts/broccoli_step2.py", line 378, in process_file
a = subprocess.check_output(path_fasttree + ' -quiet -nosupport -fastest -bionj -pseudo ' + insert + ' -n ' + str(nb_alis) + ' ' + str(Path(out_dir / name_ali_file)) + ' 2>&1', shell=True)
File "path/envs/env-broccoli/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "path/envs/env-broccoli/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'path/software/FastTree -quiet -nosupport -fastest -bionj -pseudo -n 0 dir_step2/alis_2247.phy 2>&1' returned non-zero exit status 1.

it stops at step2 and it generates, in folder dir_step2, some folders (named with numbers, ex: "2247") which their respective alis_*.phy files are empty (ex: alis_2247.phy) and no log file is generated. Can you please help me with this?

[rderelle]

Hello,

Thanks for reaching out. I'm afraid that Broccoli is not designed to analyse thousands of proteomes, as it doesn’t scale well in such cases. You are probably experiencing very long runtimes and high disk usage.

My best guess is that the issue may be due to running out of disk space. Alternatively, it's possible that the specific proteome where the crash occurs contains problematic characters. If you’re able to share this proteome with me, I’d be happy to take a closer look and help debug the issue.

Romain

[sguir]

Hi Romain,

I think it is unlikely that some proteomes have problematic features since I previously ran an analysis with a subset of these data and it worked and they all came out in the same way and it is also unlikely that the problem was due to disk space since we are running the analysis in a powerful cluster. It stops always in the same point with the same empty "alias_*.phy" files. Looking into the closed issues I found some one (Run failing at step2 #9) that have a similar problem but it seems that you upgraded Broccoli after that and it worked fine. Do you thing that 3000 proteomes (even if they are relatively small because they are plasmids) is too much? in a previous issue (from V-JJ). you commented that "Overall your dataset is quite small so Broccoli should work fine (only 30k proteins)." What else can we test? We really need to have these results analyzed with Broccoli so we can compare them with the ones we've already analyzed, where broccoli worked very well for us.

[rderelle]

Without a problematic dataset it is difficult for me to understand where the issue is. If you could compress a subset of your proteomes (eg, 1500 of them) and share it with me via google drive or alternative site, I could run Broccoli on my side and debug the issue.
nb: I won't share your dataset!