The GNU Make Course: Episode 4

All right, well it is 2.30. I'm going to assume this is going out live and welcome to the fourth, I think it's the fourth edition of me showing people how GNU Make works, a tool that tons of people use but many don't understand all the complexities of it.

If you've been following along then you'll know where we've got to, which is we've got this Makefile that we're looking at right now on screen and what this Makefile does is it takes a bunch of poems and counts the number of words and lines in them and you might be, if you're new to this, might be saying why is he doing that?

Well, I decided I wanted to avoid using a particular language like, you know, a C compiler or something like that, so I invented this kind of phony thing which is it takes these files that are .poem, which you can think of those as source code, and it outputs these .count files, so for every poem there's a count file, so if you think of it as like for every .c file there's a .obj or for every Python file there's a compiled Python file etc, so you can see it's sort of a pretend compilation and it just, you know, amused me that wc sounded like it might be a compiler, but of course it's just a thing for doing word and line counts and so this is the Makefile we've got and what it does, well let's have a look at the directory we're in, so we're in a directory, we've got a source directory and if we look in the source directory we've got a bunch of poem files and if we just look at one of those poems, there you go, there's a poem, so and it's just going to count the number of words and lines in there and create a .count file and if we look in obj we've got count files and if we look at obj slash armature for example you've got, there you go, there's a number of words and number of lines, this is like that object file, so and how did that work, well it's sort of a classic Makefile in a sense, we've got an all target and the all target says make all these object files and then I use this pattern rule which says for anything in the obj directory, see obj is defined up here, with an extension .count you can make it from something in the source directory with an equivalent extension of poem and then the compilation uses a couple of internal automatic variables, this guy $at, $at is whatever this is, whatever the name of the target is, that's what the thing on the left of the colon is called a target and $less than is the thing on the right of the target, it's the first prerequisite, the first thing needs to be built from, so this is a very classic kind of Makefile thing and I also put in a .clean which deletes everything in the obj directory, deletes all the object files, so this looks fine and if you were watching this last time, I won't go through this in detail but I did this little work here which was, there's a thing called source files which finds all those .poem files using $wildcard and then there's a thing called obj files that actually for each .poem it creates a filename .count with the appropriate change of the directory from source to obj, so remembering that GnuMate just works with lists of strings basically and at the end there was this little print thing which I showed you before which allows you to print out any variable, it's kind of printf type debugging, we can just to remind ourselves we can just go here and say print the source files variable and you can see this is the list of source files and then we can print obj files, this is the equivalent list of objects, that's just the variable that's in there and if we take a look that's how they're created and last week I taught a lot about how that was done.

All right so I'm going to change tack a little bit and talk about something which often trips people up in Makefile generation which is creating directories.

If you remember I talked a lot about how GnuMate is really about files and in fact anything that isn't a file should be marked as .phony like this, if you don't do this and then you create a file called clean then GnuMate will look for it and say okay how do I make the file called clean so you mark it as .phony which means that it's not a file but what about a directory, a directory isn't a file, what would happen if we tried to make it and there are a number of different ways of making directories so first of all let's do one thing which is let's get rid of the object directory that I've got and see what happens so I'm just going to go back and I'm going to go I'm going to delete the object directory completely okay so if I just look I haven't got it and I'm going to type make and immediately I've got a problem and if you look at the error that's coming from the shell it's saying I can't do this thing and the reason it can't do it is the object directory is missing so there's nothing in here that says how will I make the object directory and there's many different ways we could do it I mean one way let's just do the simplest thing would be to when the make file is actually created but it's actually run just make the directory every single time and we could do that like this so this thing $shell will go out to the shell and it will run a command and in this case I'm going to do mkdir and then I'm going to say dash p because I might have a complex hierarchy to make and then I'm going to say $opt which is the object file directory so if I go in here and I now type make boom it works now the object file directory got made and within the object file directory there are all the files so great and if I don't make again absolutely fine so no problem at all and in fact there wasn't obviously you know the mkdir we ran again and again and again and the problem here is that literally every time you run this make file it's going to make the directory and that may be okay in our instance where we're making literally one directory and there's no problem at all but in any real make file where there's something complex going on this is expensive and it kind of goes against the spirit a little bit of make because you know if you're going to do things like that why don't you just use the shell and not bother with make files so a very obvious thing to say is well you know the object directory needs to be made before any object file is made because the object files go in the object directory so how would we express that in going to make syntax all right well we'd express it something like this it's going to add it in here we'd say object directory it's like this right sorry the other way around you see it's a bit warm in lisbon today okay we say you know in order to make the object files you have to make the object directory and then then we could tell it how to make it right and so we could just do mkdir in here and this is quite an interesting uh you know interesting question so let's go in here and just say okay in this case that you know you're what am i doing i'm doing this wrong how do we make the object directory we make the object directory we make it by doing look to dash p but we use because the other act is the thing we're making right so that's how to make a directory um that will be the files up there in fact i'm going to use this semi this this format okay so here's a simple thing how to make an object directory run this command okay let's just go back here and just let's just say make obj which is the directory right okay well there you go it's up today let's just get rid of it again oh yeah because it doesn't have to make it okay so this is the problem right this is this thing is this thing is a directory so how am i going to sort this out well the answer is we go in here and we don't actually treat directories like they're files we work with files directly and so really the best way to do this is to do something quite different which is to actually put a file in there in the directory and then make that file and as a byproduct make the directory so let's just go back to um how we might do this in the sort of simple way so let's just say this we've got this object thing here and we've got here making the directory so that seems absolutely fine um and then up here we've got to when we're making account file well we need to make sure that the object is the object the output directory is created so one of the things we could do is we could go in here and we could say obj all right we could say if we're going to make this thing then we're going to depend on needing a poem file that's fine and we're going to depend on the directory existing and then hopefully going to make we'll we'll fix all that and put it all together um so let's uh let's go back and let's just try doing that let's do a make clean make okay so you see it actually did it at the top there you see that first line there it said we'll make the directory and it did it once because the new make knows that this thing has uh has to be made once because it's one of the prerequisites here even though it's prerequisite of many many different files so that seems great right you look at it you're like yeah that's great no problem at all i can do that um that will work now there's an interesting thing going on here which is that if you change one of the dot poem files you update it then the equivalent dot count file will get will get updated so let's just try that let's just touch one of them uh bet two dot poem right and then we type make and sure enough that one got updated and the reason it got updated was that the file here the bet two dot poem was newer than the count file and that's fundamentally how going to make works it's time stamps on files if this dot c file is newer than this dot op dot o file remake it that's the fundamental idea and this brings up an interesting thing which is what happens if this directory changes here this object directory and here's the interesting thing about time stamps on directories so if you look at the timestamp on the directory let me just do al sorry right it the timestamp can get updated when the contents of the directory change and that can have a really weird effect in going to make which is that it will suddenly start updating these dot count files because it thinks that this thing has changed somehow so it's like okay so what do we do about that problem well the reality is we want to really just have a file there and not a directory so there's no weird effect happening where that gets that changes so one way to do this and this is how i really like to build directories in gnu make is not to actually mention the name of the directory but to have a file in there so we're going to take that i'm going to say dot f and i'm going to say here how to make dot f so i'm going to do this i'm going to just change this slightly so when i make the directory and then we're going to touch now actually watch and i'm going to touch the file so what's going to happen here is instead this is now making ob slash dot f and i've used a dot f dot file so it's hidden it's going to make the directory and i'll show you another way to do this in a minute and where you can actually figure out what the directory is and then it's going to touch that file so it's actually going to create that file in the directory and here each count file is going to depend on that dot f file existing and the reason to do this is that when the directory is created here that file gets created and its timestamp is set and what's good about that is that it won't change unless you actually have to remake it i.e if the directory is missing and it can't get updated in some weird way where like the timestamp on a directory changes because a file within it changed so we can go back let's just do a make clean and let's just look in the obs directory nothing is actually doing it let's do an al as well right nothing in there and let's just run make okay notice how it made here this it made that dot f file and if i do make again nothing to be done right nothing needs to be created and if i do make clean which deletes everything in the obs directory you notice this time here previously when it had to make the directory it made the directory and made the file here the directory exists and if we just do a quick you can see that dot f file is there with a timestamp on it and that way this this will work really really cleanly so i usually refer to this technique as a marker file this dot f being a marker in the directory the directory is being created and then you can safely use it here now this is nice but one of the things that's a bit tricky in gnu make is that it's assuming that if this file were to change then this somehow that's part of this command here that is important for this command to run and normally right you'd have here like dot c files dot h files all those kind of things that make sense if any of them changed but you don't actually want um necessarily this this thing to this thing changing except when it's created to make a difference to the creation of the dot count file so there is a thing in gnu make 3.80 and above called order only prerequisites and what they do is they enforce the order in which things are built built so that this this thing will have to be built before everything else can be done but if it changes it doesn't actually cause the file to change so right now if i go in here and i go touch that file everything gets rebuilt right everything gets rebuilt because that file changed because it's a prerequisite but i can actually go in here and i can put one character in here which is the bar and i can say look these things before the bar are the prerequisites and they matter they if they change you have to you have to change the output you have to rerun the commands this thing needs to exist prior to this but you don't have to uh if it changes remake everything so what i can do if i just go and do a make clean i'm going to blow the directory away as well just do that now it's going to make that thing when i make it again and then if i touch it it should be the case that it will not rebuild everything but it doesn't rebuild everything because that that file being updated doesn't mean that it's newer and therefore um everything needs to be remade so this is another thing you can do which is uh use order only prerequisites in this case it's a little bit unnecessary because there's only one way that dot f file gets created it's here it's when the directory is made but you know in terms of uh thinking about facilities i'm going to make that order only prerequisites are a really nice thing you can do now that we've got this there's another thing we can do which is if you look at um this this clean here it actually deletes all the individual files in the object directory one of the really hard problems in can you make is sort of one of the hard problems in computer science i think is making make clean work um typically it's very very common if you do make clean to discover that make clean doesn't actually work um and it's really common in make files to see like there's a clean target there's totally clean there's really clean there's totally really really clean there's like different targets and they're deleting different intermediate files and object files and different things like that and it's very very confusing and so one of the things i really like to do is instead of having something which is actually deleting files and trying to make sure it knows exactly which files need to be deleted is blow away entire directories so what what i'm going to change this to do the following so now make clean just blows away the entire object directory and this is way safer right now we know that this is uh you know it's going to be everything's going to be deleted even if i forgot in my make clean to list some file because one of the problems is as the make file grows you go in there and you keep updating your clean target with oh yeah there's that set of files too and there's this set of files you probably don't have just one variable which is object underscore files and so let's just go back making now here's an interesting thing if you wanted to verify what make clean would do um how do you do it right now it doesn't show anything and a couple of weeks ago i showed a different a couple of techniques for doing this let's just show you a different one if i do make dash n it's actually going to show you the commands that would be executed when you do that so i actually can do similarly i could do this right there's not actually running those commands you're getting exactly the output in the order in which they would be executed and you're going to say okay this is what's going to happen and this seems fantastic and if you read the manual it's like yeah make dash n you can check out your make file before you run it in reality make dash n is really hard to understand for any large make file it's just almost impossible as you as you go through it it's like the output is gigantic um and it's just hard to figure out what's going to happen so i always find that really really difficult what i like to do is something a little bit more surgical now one thing you could do is you could go here i could just go in here and i could just remove this at right then it would actually print out the command when it gets executed um and the other thing you can do is i showed you a little while ago is you could define a variable i often like to do this right so q equals at what does this do is defining a variable called let's just use colon equals finding a variable called q which is an at which means this thing dollar q will turn into an at and the at will suppress the command let's just try it this command is not there and if i want to override it i can go and go i can do that what that does is by defining q on the command line it overrides the definition in the make file and that's an interesting important fact which is if you have a variable defined inside a make file and you mention it on the command line the command line definition takes priority so you can you can override something so you can do nice things there where you can get it to print out different parts of the make file and you can do things that are much much more sophisticated than that if you want to um so that's in terms of printing things i'm just gonna just leave that like that for the moment so if you're making directories i really recommend this this um this technique of making a marker file now one other thing is that sometimes people say um okay actually well i know at the beginning of the make file i really need that directory to be created and if you remember i did that i did that technique which was to go like this right dollar shell let's do a dash p dollar i'm sure right so people sometimes do that and that's running at the parse time of the make file that ship that shell come on because it has its parse we'll figure that out um another way to do it might be to say okay you know actually um in the all target um when we're doing make all we'll make sure we build the directory first so another thing you might see people do is this so we're going to say something like this okay so they've said when you're making all you make the directory and then you make all the object files now i'm gonna go back here and just just clean this up make it boom that works right and because it's a dot f5 it doesn't get to be made so so this this does work now the reason this is a little bit weird is twofold one is you're depending on that particular all target being the one you run so you're assuming that you're always doing the all target which may not be true because there may be an intermediate step you might be building a library you might want to build a particular file in which case now you've got to sprinkle dollar obs or f all over the place in the make file the other thing is it's it's building on an assumption that these things are run left to right so actually that this thing will get built before this thing and in if you run this with dash j1 which is means you're using one thread essentially of execution then that's absolutely right so going to make will traverse from left to right in the file so it'll do the dot f file first and then it'll do the object files and that will be absolutely fantastic and it will work fine um you can just let's do another so again most absolutely fine and j1 is like this right so you just do that let me just do another clean um okay so you do that and runs it once um but there's no guarantee that if you up the number process that things will necessarily happen in exactly the right order and the reason is um and here the output gets all all messed up is that um let's try it again let's just see if we can make it break the reason is there's no dependency mentioned in this make file that says an object file can only be built if the dot f file is built and so i think one of the great challenges with make file is understanding that there is a directed graph which expresses all the dependencies between things and this doing this like this is a great example of essentially an implicit dependency you're saying because i know that the object files are going to be built after the dot f file because it's going left to right then this will work but gnu make doesn't necessarily know that and once you ramp up to using you know if especially using dash j8 and things like that where you're doing multiple things simultaneously if it doesn't know there's a dependency between the two of them then it will say well you know what i can schedule those to happen at the same time and then you can get really weird effects where it tries to build an object file before the dot f file gets built the directory doesn't exist and now you're in a world of hurt so i really don't recommend this if you're building a make file express absolutely explicitly what needs to be built in what order so another way to do this i had put the object file here another way to do this would be like this there's the object files and then to say all the object files depend on that directory okay and therefore the director get built so let's just go back here let's just do a make clean oh there you go it works and actually as i ramp up the let's just do this do you notice a difference here here's how this was done at the beginning prior to um that everything else being executed and above when i did the let me just go off then here things got interleaved and the reason that happened was this here created dependencies and now the tree in the mind of gnu make it knows the object file depend on that dot f file being created all right the last thing i would do because having the dot f everywhere and you know i find a little bit ugly is actually make myself a little function that that creates a marker file so i'm going to make myself a little function called marker and it's just going to stick someone's saying it's just going to be this now okay the syntax can be very very terse in gnu make so what i'm saying is uh whatever the parameter to this is is going to be followed by dot f and that's just you know you're not doing a string concat call it's literally going to be evaluated as a string right this is going to be evaluated dollar one's going to turn into whatever the parameter was and uh it's going to get dot f added on the end so let's just check it out use dollar info is kind of a printf type function i'm going to say call marker on the hodge okay let's just go back here notice here this thing that's the output of that info command so it's taken dollar which is obslash and it's appended dot f so if you do call dollar marker on obj you're going to get the name of that dot f file and i find that a little bit cleaner i always hate repetition in things so i'm going to go here and i'm just going to change this so i'm going to say call marker dollar obj see that's that can be the file there all right and then you know i again i can use it here right i can say down here and you know what let's just do that okay now this might look a bit odd because to the left of this colon you've got a call but remember this is just being expanded at at at the execution time basically at a past time it'll go through and expand that and it'll turn out to be dollar obj dot f which is obslash dot f let's just check that it still works okay great everything's go i can get rid of the the printout let's get rid of it where it's printing that this let's just do that one more time all right that's good now finally well i don't know about you but now i've eliminated the fact that i might want to change the name dot f to something else because i've encapsulated it in here but i don't know i find this a little bit ugly maybe i want to do something else which is really completely automate this and there is a way to do that which is that you can you can do the following you can go in and you can say give me the directory of these files and therefore figure out what it is so you can go into here and you can say you know the obj files depend on this thing and then i can say all right well um give me the directory so there's actually a call which is dir i was actually before we before we do it let's just just go up here and say um let's just use info let's go see what that prints out you see this says obslash here what it's done is it's parsed the output of this which remember is obj slash dot f and it's figured out what the directory is this is another built-in function which gives me the directory part of it and i can actually do not do it if i want to get the file name part of it so you could actually go in here and automate this and you could say okay so for these obj files i will get the directory for them and then i will get i will add the marker to them automatically and so you could go another level deeper now as i'm running out a little bit out of time what i'll do is in the next episode i'll carry on and i'll show you how to make this completely automatic so you don't have to refer to the actual directory here because if you do that then you can get the directories that are built automatically that if the directory changes then it doesn't have a problem doesn't mess up the system um and so you don't get something rebuilt automatically when it shouldn't be um you don't have to go through your mate file and make sure you've mentioned the directory everywhere and it makes it much more maintainable because it finds out what those directories are automatically so okay so that was a little bit of an introduction to building directories you're going to make i know this is a bit of a weird topic because it's tempting just to say you know what it's a directory i'll treat it like a file but i can have some pitfalls you can end up making directories over and over again it can end up being slow um if you're using dash j4 dash j8 to parallel building then you can get things in the wrong order so when you're just to summarize when you when you're making mate files everything is a file unless you say dot phony in which case it's not um be really explicit about the dependencies and make sure that they are expressed as targets and prerequisites if they're implicit like in the ordering thing i showed you um then you can have a problem when you go parallel and that's a really really common issue and i really like to encapsulate everything in functions um so that i don't have to go through and constantly you know be updating my mate files because i change one thing in one place um if you're really interested in interested in this topic there's actually an article i wrote many years ago for cm crossroads which is called making directories in gnu make if you literally google making directories in gnu make you will find it goes through all the different permutations of ways of doing this um next week i will dig in into more making directories doing it in this in the simplest way possible doing it in a way that's the most maintainable i think one of the really big challenges is with make files is making them maintainable there's a huge amount of flexibility here um and you really have to fight the urge to copy and paste stuff you know it's dead easy to copy and paste and say yeah well i'm going to make an object i'm also going to make this other directory therefore i will just copy for example this mukta thing here if you do you end up in a real mess and that's why some mate files are so hard to understand the more that you can do using the syntax of make in terms of how it understands targets and prerequisites and that director graph and then use the built -in language of all these functions the easier the mate files are going to be to maintain it does take a little bit of getting your head around the fact that you're fundamentally dealing with space separated lists and strings and those things refer to files but once you do then i think you'll find it's much more powerful so i'm going to stop there today uh next week i will go through a little bit further on this topic of making directories and hopefully we'll be able to see in a little more detail how you might do this in a really clean and simple fashion that will make your mate files really easy to maintain and that's it for me thank you so much for watching i hope this is useful if you obviously if you know going to make you this may all be old news to you but many many people don't and despite the fact that it's what 30 40 years old when you make um it's still widely used and you know hopefully you'll get a lot of life out of it and as you you know go through this course thanks very much