Yesterday, zmedico wrote about building multiple packages in parallel with Portage-2.2_rc2. In Gentoo Prefix, we had a sneak peak to this feature, so I have had some time to play with it on my dual-quad core box. Some timing results that you may like:
emerge -e system (excluding sys-devel/gcc)
As a baseline:
With --jobs=1 and MAKEOPTS=16, load-average=9:
Because I was skeptical of what
--jobs could really do, I decided to start with small number of parallel jobs:
With --jobs=3, MAKEOPTS=16, load-average=9:
While that was running, I noticed a very significant amount of time where my cores were idle, thanks to the handy little xfce-extra/xfce4-cpugraph widget. So, I turned
--jobs up again:
With --jobs=5, MAKEOPTS=16, load-average=9:
Meh, not much improvement there. Surprising, but I suspect that I may be reaching the limits of the parallelization (dependencies, etc).
With --jobs=10, MAKEOPTS=16, load-average=9:
(And actually, a quick visual scan showed load averages staying below 4. Only a few times did I see the average above 8 )
Relying solely on load-average to keep my system usable:
With --jobs=40, MAKEOPTS=40 load-average=15:
The highest load average I visually saw was 23. But my load average was mostly always greater than 4, so this means that my procs are obviously getting used more but I must have hit another bottleneck.
-emerge -pe system was preformed before each time trial to assume the depgraph was in cache.
-84 packages total
-no ccache/distcc running
Conclusion: 20 minutes? ~30% speedup. Wow. Good! Quite significant even. Assuming you have cores/procs to spare, go ahead and crank up those
--jobs. It is nice to be able to make the
./configure step not be the bottleneck anymore. ;-) I will keep testing and see if I can get the time down even farther (although, unlikely based on the last time trial).
Test requests? Please leave a comment.