Builds go parallel. Also, improved ergonomics overall.
[Thread Prev] | [Thread Next]
- Subject: Builds go parallel. Also, improved ergonomics overall.
- From: Ori Bernstein <ori@xxxxxxxxxxxxxx>
- Date: Tue, 25 Jul 2017 22:54:33 -0700
- To: myrddin-dev@xxxxxxxxxxxxxx
First off, the user visible changes ----------------------------------- - Builds go into an obj/ directory by default. You can select this with the '-o' option. If you're a masochist that loves getting their working directories littered with build intermediates, you can use "mbld -o ''" to get the old behavior. - Some unused options have been removed: `-l` for building libraries on the command line was junk. It's gone. `-C` and `-M` are now controlled via environment variables. - The build output has full path to files. It's a minor change, and it fell out of other things, but now your editor will know the full path to the file, and can probably directly take you to the right error message. - Implicit file additions are gone. In the old mbld, if you had 'use "foo"' in a test, then foo would be added to the test build, even if this file wasn't specified in bld.proj. This was a bad idea, and now all deps are listed. - The builds are going to be faster. Other than that, there's not much that changed. If you have a build file that used to work but doesn't now, then it's a bug. Now, how did it get faster? --------------------------- The first iteration of mbld had the arrows in the graph pointing the wrong direction, which made the whole ordeal of building things harder than it should have been. Hundreds of lines removed later, the code now does parallel builds, and outputs to an obj/ directory by default. Before this change, the arrows in the DAG pointed from the root of the tree towards the leaves, and the graph would get built in a depth first walk, short-circuiting the recursion by statting the edges. This both made it hard to figure out what could be built next, and lead to a bunch of extra statting. Afterwards, the arrows point from the leaves upwards to the root. Each node has a count of how many nodes are blocking it. The build is kicked off by iterating the leaf nodes, and decrementing the reference count of their edges. If a reference count drops to zero, it gets enqueued for a build. Then, the main build loop just launches as many processes as are enqueued, up to the parallel job limit, waits for one to exit, and unblocks its upwards edges. The core loop is here: const build = {b : build#, g : graph# for n : g.leaves if n.want stale(b, n) unblock(b, n) ;; ;; while std.htcount(b.proc) != 0 || b.queue.len != 0 while std.htcount(b.proc) < opt_maxproc && b.queue.len > 0 launch(b, std.slpop(&b.queue)) ;; wait(b) ;; } How fast is it? -------------- Using my highly professonal benchmarking method of running `mbld clean; time mbld` a couple of times, and using mforneys proposed ninja patch, here's the timing: Ninja: real 0m1.737s Mbld: real 0m1.847s Keep in mind that mbld is scraping the whole dependency graph every build, while ninja has it predigested into a single myr.ninja. -- Ori Bernstein
Re: Builds go parallel. Also, improved ergonomics overall. | "rymg19@xxxxxxxxx" <rymg19@xxxxxxxxx> |
Re: Builds go parallel. Also, improved ergonomics overall. | Ori Bernstein <ori@xxxxxxxxxxxxxx> |
- Prev by Date: Warning: Incoming Syntax Change.
- Next by Date: Re: Builds go parallel. Also, improved ergonomics overall.
- Previous by thread: Warning: Incoming Syntax Change.
- Next by thread: Re: Builds go parallel. Also, improved ergonomics overall.
- Index(es):