Eigenstate: myrddin-dev mailing list

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Builds go parallel. Also, improved ergonomics overall.


First off, the user visible changes
-----------------------------------

	- Builds go into an obj/ directory by default. You can select this
	  with the '-o' option. If you're a masochist that loves getting
	  their working directories littered with build intermediates, you
	  can use "mbld -o ''" to get the old behavior.

	- Some unused options have been removed:

		`-l` for building libraries on the command line was
		     junk. It's gone.

		`-C` and `-M` are now controlled via environment variables.

	- The build output has full path to files. It's a minor change,
	  and it fell out of other things, but now your editor will know
	  the full path to the file, and can probably directly take you
	  to the right error message.

	- Implicit file additions are gone. In the old mbld, if you
	  had 'use "foo"' in a test, then foo would be added to the
	  test build, even if this file wasn't specified in bld.proj.
	  This was a bad idea, and now all deps are listed.

	- The builds are going to be faster.

Other than that, there's not much that changed. If you have a build file
that used to work but doesn't now, then it's a bug.

Now, how did it get faster?
---------------------------

The first iteration of mbld had the arrows in the graph pointing the wrong
direction, which made the whole ordeal of building things harder than it
should have been. Hundreds of lines removed later, the code now does
parallel builds, and outputs to an obj/ directory by default.

Before this change, the arrows in the DAG pointed from the root
of the tree towards the leaves, and the graph would get built
in a depth first walk, short-circuiting the recursion by statting
the edges. This both made it hard to figure out what could be built
next, and lead to a bunch of extra statting.

Afterwards, the arrows point from the leaves upwards to the root.
Each node has a count of how many nodes are blocking it. The build
is kicked off by iterating the leaf nodes, and decrementing the
reference count of their edges.

If a reference count drops to zero, it gets enqueued for a build.

Then, the main build loop just launches as many processes as
are enqueued, up to the parallel job limit, waits for one to exit,
and unblocks its upwards edges. The core loop is here:

	const build = {b : build#, g : graph#
		for n : g.leaves
			if n.want
				stale(b, n)
				unblock(b, n)
			;;
		;;

		while std.htcount(b.proc) != 0 || b.queue.len != 0
			while std.htcount(b.proc) < opt_maxproc && b.queue.len > 0
				launch(b, std.slpop(&b.queue))
			;;
			wait(b)
		;;
	}

How fast is it?
--------------

Using my highly professonal benchmarking method of running
`mbld clean; time mbld` a couple of times, and using mforneys
proposed ninja patch, here's the timing:

	Ninja:	real    0m1.737s
	Mbld:	real    0m1.847s

Keep in mind that mbld is scraping the whole dependency graph
every build, while ninja has it predigested into a single
myr.ninja.

-- 
    Ori Bernstein

Follow-Ups:
Re: Builds go parallel. Also, improved ergonomics overall."rymg19@xxxxxxxxx" <rymg19@xxxxxxxxx>
Re: Builds go parallel. Also, improved ergonomics overall.Ori Bernstein <ori@xxxxxxxxxxxxxx>