Search for a command to run...
Release notes This release is almost completely devoted to fix the memory-related issues that kept appearing in the issue tracker (#164, #225, #657, #688, #766, #799, #801, #832, #833, #836). We found a snow-ball problem that may affect any workflow based on Nipype, and that particularly affected FMRIPREP for several reasons. The memory explosion typically occurred after a large number of nodes of the execution graph had been run, and was definitely triggered with a memory-hungry process finished at that point in time. Briefly, The workflow building of FMRIPREP is heavyweight routine, and the garbage collector had some trouble to clear-up its debris. With that extra memory reserved, FMRIPREP called the <code>run()</code> member of the main workflow which uses a <code>multiprocessing.Pool</code>. By default, <code>--nthread</code> workers were spun up using the <code>fork</code> system call. Subprocess generated this way allocate as much memory as the parent had allocated. Only some was free, and this meant around 350MB per worker in the case of FMRIPREP. Nipype uses <code>subprocess.Popen</code> to call the third party software (FSL, AFNI, FreeSurfer, etc.). Again, Popen calls try to allocate as much memory as their parent (the workers, or the main thread for nodes with <code>run_without_submitting=True</code>). The workers and the main thread kept growing slowly until the memory explosion condition happened. From that point on, memory grew linearly with running time. Changes in this release first address two targets: Try to keep the main thread of FMRIPREP as lightweight as possible: Isolating as much as possible the workflow creation wrapping it with a subprocess. Removing <code>run_without_submitting=True</code> from all nodes that opened images (which are memory-hungry). When <code>--nthreads 1</code> use Nipype <code>MultiProc</code> (as if parallel processing were required) for the reasons below. Try to keep the workers as thin as possible: Using multiprocessing in <em>forkserver</em> mode, which prevent subprocess from allocating as much memory as the parent. Clean-up memory as often as possible (forcing garbage collector calls, explicitly deleting objects, etc.) Restart the workers as often as possible (currently, after running just one node) Reduce FoV of images whenever possible. All these efforts also required also some changes to Nipype, that will be released with their version 0.14.0. CHANGES [ENH] Use Nipype MultiProc even for sequential execution (#856) [REF] More memory annotations and considerations (#816) [FIX] Controlling memory explosion (#854) [WRAPPER] Mount nipype repositories as niworkflows submodule (#834) [FIX] Reduce image loads in local memory (#839) [ENH] Always sync qforms, refactor error messaging (#851)