Shell wrapper for build system profiling
mskala mskala mskala Core concept implemented. Here's a Web log posting about it: http://ansuz.sooke.bc.ca/entry/260
mskala Basically works now for the immediate need. Packaging it nicely will be a future project.
Tsukurimashou Font Family and IDSgrep
It would be really nice to be able to know, quantitatively, where all the time goes during build.
Right now I'm prepping version 0.7 for release. This has initiated an hilarious all-singing all-dancing three-ring yak-shaving party. First make distcheck failed because make check failed (and I didn't know make distcheck actually required make check to pass in the first place). At that point make check was failing because fontlint (as already well known) demands strict adherance to rules that FontForge is incapable of complying with. Then I discovered that fontlint (which I hadn't known before) is in fact a FontForge script, and thus can be replaced with a homegrown script to serve the same purpose, but better. Of course, a better fontlint would also be important for other users of fontlint, so that became a proposed contribution to FontForge instead of something specific to Tsukurimashou. But then someone else reading my bug report complained that fontlint's shebang pointed at the wrong absolute path to the interpreter, and wanted fontlint's shebang to be changed to use a path relative to the current directory (!), which is reported to work on a Mac. That's probably not what FontForge should do cross-platform, but I ended up being asked to look into it and make it right. And then I found that FontForge's script interpreter implodes when given two-dimensional arrays. Long-term that will need to be fixed too; short-term I redesigned the fontlint replacement to only use one-dimensional arrays. But even with the fontlint replacement up and running, make distcheck still fails during make check, and the options I need to pass to the fontlint replacement to let it pass so I can build a distribution are a fair bit looser than anything I want to have as the default for manually-invoked make check. Figuring out DISTCHECK_CONFIGURE_OPTIONS and related mechanisms in Autotools has required a fair bit of trial and error, and at this point the information flow still isn't quite right. I also observe that make check often seems to result in apparently-unnecessary rebuilding of Makefiles and everything that depends on them, running to at least an hour of wall time in an all-stops-out configuration before it can even start the actual tests. And for testing an actual release candidate, I do need to be building with all optional features enabled.
The important observation is that the process is more painful than it should be because I'm doing a lot of large builds and they take a long time. Furthermore, I don't know exactly *where* the builds are taking a long time. Building dependencies seems like a lot of work; building Postscript subfonts seems like a lot of work; assembling SFD and OTF fonts seems like a lot of work; kerning seems like a lot of work; and building proofs seems like a lot of work. But which of those actually are the biggest factors in the overall total? Is it 20/20/20/20/20, or 10/10/60/10/10? Which one should I concentrate on if I want to make the build faster? It would be really nice to have a clearer picture of the build time consumption than just "it seems pretty slow." We could use a profiler that will operate on the coarse level of the entire build.
I think the best way to implement profiling at this level is to wrap the shell used by Make. I'd really like to know the start and end time of every recipe Make executes; but doing that seems like it would require breaking into and instrumenting Make itself, which is probably a losing game. Replacing the shell with a wrapper (using the SHELL= setting in every Makefile) is much less intrusive and would work at nearly the same abstraction level. I envision the wrapper as a C program, which could be packaged as a Tsukurimashou parasite, that would become the target of the SHELL= setting. When executed, it would write a "start" line to a log file, execute the actual shell, and then write an "end" line to the log file, passing command line arguments and return values back and forth appropriately. It might also save its own process ID in an environment variable passed to children, and include the value of that variable received from the parent in the log lines, so that from the log we could recover the tree structure of invocations of the wrapper. Possibly also save a user-defined environment variable to allow for a larger-scale tree structure (for instance, of Make recipes) if the system hosting it is set up for that. Then external utilities can scan the log file and recover timing information, even in complicated builds that involve multiple levels of recursive parallel Makes.
A tool to do this would of course be useful for systems other than Tsukurimashou, so it should be built to be as general as possible.