The GNU Make Course: Episode 5

Okay, well welcome back to my little course on using GNU Make and this is the I think the fifth one that I've done and if you've been following along then you may remember that I have a very contrived but actually something that is close to a real example of making object files from source code files and let's just recap what that looks like. So I've got a directory in here it's got a makefile it's got an object directory where the output is going to go and a source directory where there's the source code and if we take a look in source you've got these .poem files this is my source code and if we just look at one of these they are poems and I've got a makefile and the core of this makefile is this section here which makes a .count file from a .poem file by calling wc just to get the number of words and the number of lines in that poem and storing it in the count file and the idea here was that this is the .poem is sort of source code and .count is sort of object code and I built this up over time to actually find all the source codes create a variable obj files which is the name of the which is the list of all of the object files and then this all target here which actually builds those object files if you remember last time if you didn't watch it you should go back and watch it there's this technique for making the object directory if it's not there the clean just blows away the object directory and so on so this is the main file it's pretty simple I'll just just do a quick make clean and make okay there it is it makes the directory uses this thing where it touches a hidden file in there so that he knows the directory has been made and then it goes through here and runs wc on every poem that's in there so fairly classic makefile pretty simple if you want to look at how it finds their the actual source files it's this function here this variable here which is just doing a loop through a bunch of poets and doing this wildcard thing to actually get those different poem files if you just go back to here you can you can you can see there you go there they are okay I'm going to switch gears slightly and talk a little bit about recursive make and this is a very common pattern that you see and make files where not everything is in a single directory and actually projects any realistic project probably doesn't consist of just some source files that get turned into object and linked up into an executable but often contains modules and often the way this is handled is the modules are encapsulated in such a way they have their own make process and you have a top level make file which goes and runs make in all those different directories so it's a little bit like if you did a for loop and you went through and you went for and did make in all those subdirectories but it's usually done within the context of make so what I'm going to do is I'm going to rework this into a bunch of modules actually for each poet was going to pretend it's more it's a little bit more complicated and then within each of those poets will actually have a separate make file and then we'll wrap them all up and run them as one massive make process so to get us started just to recap where we are I've got object and source in here let me get rid of I just do that okay I only really need and I can get rid of the temp file there okay so I've got source code containing those poems and now what I'm actually going to do is I'm going to say it's not just one source directory it's a bunch of different modules so I'm going to make some directories I'm going to call it Armitage, Betjeman and Owen so now I've got a bunch of directories and within each of those I'm going to have separate source trees so these are like the separate modules so I'm going to do just something like Armitage, oh dear it's a good start to the day okay so within each directory I've got those so just have a quick look at that as my source directory and then I'm going to put the different source files that I had currently in this top level source into those different locations so if I go I'm going to move that into Armitage source and all the Betjemans into Betjeman source and um into Owen source okay so now my source directory here is empty so I'm going to get rid of it sorry okay so now I've got this let's just do a quick list on this so you've got Armitage it's got source code in it Betjeman it's got source code in it Owen's got source code in it okay so within each of those a similar kind of structure and you can imagine if you were building you know a C program that had a lot of libraries that had to be built you'd have the similar kind of stuff you'd have .c files turning into .obj files ultimately being linked to libs and ultimately being put all together by a linker to create the the executable but you're kind of a structure here and so each of those directories each of these Armitage, Bet and Owen would have their own um make file so what I'm going to do is I'm just going to copy the the make file that I have uh into Armitage well before I copy I'm going to repurpose it either within the context of the Armitage so let's just go back this is the top level make file this is in going to make course so I'm going to load up the one that's in here okay so this is actually fairly similar to what we had before but I think I can simplify some things so I'm going to go in here and say okay the source file as well in this case it's just everything that's in the source directory of Armitage so that's that wild card let's check about that right and the object file is very similar that can that doesn't have to change same thing as making a directory you can have all in here same thing making the marker so this pretty much should all be the same I think so let's just go back to here and let's go back to the shell and let's have a look let's go into Armitage let's have a look at the make file okay this is the one what happens if I type make so that worked perfectly it made the object directory in here so it's reminded where we are I'm in Armitage it made an object directory in the object directory is this count file for the Armitage so all all kind of the all worked really all I did was I removed what was if you looked at the this one it had to iterate through all the subdirectories so now rather than being all in one place we're going to have the separate things in each directory now the tempting thing here to do would be to say okay well that make file works great there we'll copy it into the other directories you know what let's do that just just because why not so we take Armitage make file we'll copy it into bitumen and we'll take the same make file and we'll copy it into Owen okay so I should now be able to go in here and type make and that works fine right and if I go into Owen type make it works and if I go back to Armitage and go make okay nothing to do because I did it before and if I do make clean here it just removes the object directory and I can remake it so great it all works great but unfortunately I've got a huge amount of repetition here and if you've been watching this course I hate repetition it just kills me and so what I'm going to do is I'm going to do something a lot simpler which is if you look at the fact that all of those make files are completely identical then why not have none of them have one of them and then include them everywhere and luckily make has a an include function and so what I'm going to do is I'm going to take one of those examples uh let me go where am I right now I'm here so I'm going to take that make file that's in here and copy it up and call it common.mk so that's now in the top level directory here there is this common.mk and then in each of Armitage better know and I'm going to have a really simple make file so in fact so simple I can just do it with cat include okay so I should just check that's there nicely there it is all it does is include now within Gnome there's this nice direction which is include we just literally copy and paste in if you like right there the contents of a make file and that should just work for us and let's just where am I right now let's go into Armitage let's type make nothing to be done let's do make clean make it all works even though the only make file in here is that so okay so that's nice so let's now copy that into vetrimin let's just do this file into owen okay so now we've just got the simple make files and all those let's just check that they still work let's just do make clean here in vetrimin yeah works great okay fantastic so now we've got really simple setup so I can go back here this is my main make file if I just go back to this well this guy I'm going to kill that buffer because it's gone and if I were to quickly look at that Armitage make file it's just including the common part okay so let's go back to the make file so now the top level make file doesn't need to do any of this it just needs to go into each subdirectory and run make and then it will make everything so it doesn't need to have any concept of the source it doesn't need to have a concept of the subdirector and I'm actually going to call those modules because that will make it a little bit more obvious what those things are it doesn't need to know anything about this it doesn't need to know how to do any of these other things how to make files how to make directories I always leave print dash percenting because it's super handy debugging tool although I've got it in common so it could even be included but there we go all right so there's got this list of modules which corresponds to the list of directories and we're going to tell well there's a number of different ways of doing this I was going through this and let me just comment this out for a second we'll come back to that so what are the ways of recursing okay so let's look at a number of different ways of recursing one thing you see quite a lot is people doing it in bash so they'll do something like for dir in dollar modules do slash and then they'll go dollar make and sometimes they'll even do right semicolon there okay so they'll go into those okay so when you're making all then what you do is you go into every directory in the modules directory and you execute make and that's what this dollar make is now you should always use dollar make when you're doing this kind of thing you could absolutely do this make and that would that should work I will test it out in a minute and the dash c on make changes directory prior to executing the make so you'll literally go into each of those directories and come out again there's another way to do this you sometimes see this you sometimes see this people do this like this that's another style where you actually just use a cd and then it does it it's not quite as good because what make will do if you uh if you do dollar make dash c dollars audio then what happens is make will actually tell you as it goes into it into an hour of each directory which can be super handy if you do it as a cd then you don't necessarily know and you might have a little bit more difficulty understanding the output and why is there a dollar dollar here well because you want this thing is going to be interpreted by GNU make and anything with dollars is going to be interpreted as something that GNU make needs to deal with so this is essentially an escape this will turn into a single dollar when it gets passed to the shell okay so um so let's close that up let's go back to the shell where am I okay so here's what happened right so it said you can see now it's actually been expanded out so it goes through each of those directories and it runs in this case the make i'm using which is the one on the mac it's going to make and goes into each directory and you can see that nothing to be done for all in all those locations okay great um let's just um let's just go into here and do a make clean now let's go into batch menu do a make clean now let's go into owen and do a make clean just check you know where we are we're in there and if i do make but when it goes through and did all the things you'd expect it recursed into each directories but i want to talk about why this is a bad idea so first of all i mentioned the use of cd is not a good idea because you lose information that gets printed out the other problem here is that if one of those sub makes fails um this command just terminates and it's very can be very difficult to figure out what's what's been happening but there's a more subtle problem which is that what's happening here is make when it does when it executes the all it actually goes down and runs this command this this for loop has a single execution in the shell and that means it loops sequentially through that and the problem with that is if you then try to use can you make dash j which allows you to run multiple make processes simultaneously um it'll actually go serialized here it'll literally do one after another because make has no real way of looking into this and understanding what you've done in in the loop so in general doing this is a really bad idea a better idea is to so essentially at the top level use dollar make as the command because then you make can handle it um and that makes it uh easier so um so i'm going to rework this slightly so what you really want to do is say um you know well let's do this sort of the the dumbest way in which we need to do this dollar make you could say you know dollar make dash c and then you could say you know oh in etc etc that's not such a great idea better to let you make build that tree of information and go around so you can do this you can say you have to build all the modules and then for each module i'm going to show you something might be a bit surprising here i'm going to get rid of this and i've got this and the other thing i'm going to say is oops i cannot type today all right so i've said that phone that all depends on modules and modules is going to make run dollar make in each of those directories because modules is a list here um this thing here actually expands out to three different instances so it's actually like three entirely different commands uh to entirely different recipes in the make file and lots of people get confused by that because they think that if you put two things on the left of the colon then it might mean they're made at the same time you know if you had something like food bar depends on bars that's actually equivalent to food depends on bars and bar depends on bars and if there was some command in here they got executed then it would look like that it would literally be expands out two separately different things so so this this is not atomic on the left this is literally a list and it's going to get repeated so what happens here in this uh in this make above this is going to go through and it's going to say okay for every one of those modules do a make and dollar at here will be the name of the module so we should be able to so you can see it tried going into owen it tried going into armature to try going into bet and it ran making each of those directories and the thing that's neat about this is because this these things are targets in going to make and it's built the tree saying okay these three things oh in armitage bet are prerequisites of all if you do dash j it can actually run them simultaneously and going to make has this really neat um way of using um a pipe between different make processes to determine how many makes can be running simultaneously and it's a really really cool thing but you have to do it right in the make file if you do the for loop thing it really doesn't work the other thing i did was i put this as phony because those things are directories and we don't want the directory to be made so this will force you to go in into those subdirectories all the time now if you google about recursive make you will come across article and particularly well written article called recursive make considered harmful and the reason is um this what you've got described here is okay but there's a lot of lost information so there's no information about the dependencies between those different modules so if there was some ordering and we said to be done if one depended on another it's not expressed here and there are ways to express it and i can in a later uh course i'll talk about how to solve these recursive make problems the other thing is if you have a lot of modules this has to recurse into every single one of them and so what will happen is you know you you touch one file in one module somewhere and then you find yourself going crazy actually going through all of those locations um so for example if i'm sorry if i just so if you look it went into oh in armature and if i do it again there's nothing there's no work to be done right it has to go into all of them and again and again and i'll just do j8 or something and j8 notice how the app the order changed here so and the reason the order changed is it was able to start running those things simultaneously in the in the order in the ordering above in this one here it actually ran them in the order in the make file because it's doing one make process at once so i did owen and did armitage then did betjeman um here because i did j8 allowing tab up to eight it actually ran managed to do them all in in parallel which is great but nevertheless it has to go into every single one of those subdirectories and this really can get messy in in large make file situations because it can take a really really long time and also if there are dependencies um what you've done by starting a new make process here is you have lost information because this top level make this whole make file here that i'm running it doesn't know how um the files are actually made within the subdirectories and it doesn't know anything about whether they're up to date or not so it has no choice but to recurse into them a better way of doing this is to actually have a single make file that encompasses every file of your system even if those files are actually arranged into modules and then it can figure out okay i need to recurse into this directory to build a particular thing um or not even recurse at all you actually do it as one big make file and the way you do that is you use that include directive to actually include information from all over the place and i will show uh you know the thing how to do if you go and look there are great examples of how to do non -recursive make basically how to make it into a single make file okay we've done this we've got a simple recursive make thing going on um and we have we really could do with a clean target so um what can we do there well let's just have a look at that common thing i've got here it includes the ability to do make clean so what i really want to do is do make clean in every subdirectory um and there's a few different ways to do this um we could do the for loop that we did before um we could create a a phony target for each clean command um so we'll just try it out so let's go back let's put this thing back on and basically what you need to do is you need to go into every subdirectory and you need to do make clean so you know there's a few ways we could do this we could just do um we could do the for loop so let's just do this one in um modules my bash syntax and then um that's right isn't it yeah and then make and go into that directory which is and we're going to not make all want to make clean i'll show you oops i'm going to put my there and done okay so this is going to do the classic run through every module call make in every in every um in every uh directory with a target specified on the command line so this one i didn't mention anything and what will happen is let's suppose i go into armitage make file so which includes the common and the first thing it'll come to in here is all so this is the same as doing we could actually make this super explicit we could do that right we could say make all there so we're doing make clean here so let's just check that this does what we want and type something right whoops well i'm sure you're all watching along thinking he hasn't typed done correctly i've probably missed a semicolon somewhere okay there you go it does make that did make all if i do make clean goes through in every directory and so if i do now make it should actually go in rebuild everything so that's one way of doing it another way would be to use this approach here uh but now we want to have something we can't it's not modules you want to do it's like modules clean or something so that's how we'll do this so let's just create ourselves we'll take modules and what we'll do is we will add suffix yes that's good and then we'll take okay do that and then we'll go make see and we'll come to what we need to do in a minute here um and we'll do clean and then what directory we're going to go to into well it's going to be these modules things so you need to split off oh so what have i done here um actually let's just make clean up i hate repetition so badly okay let's be a little bit clearer so this cleanup thing well you know what we have way printing that out don't do this so it's going to be oh and clean armatures clean bet clean going to be the target names which is then going to be the pre -recorded this which is going to be this and then what we have to do is we have to um we have to remove um the clean bit in order to get um in order to get uh the this the command line to look correct and um well there's actually a hideous hack i can do i'm going to show you a sort of hideous hack we can do okay so yes okay i did something really really ugly there but sort of fun thing you can do in guna so if you uh let me just look here if you go back here and you look at print cleanup right so this is these things oh and clean armatures clean bet clean which are um themselves simply um target names and i've made them phony because they're not a real thing i won't build i've made them approve the prerequisites of clean so whenever you do clean it does all those cleanup things and then i've done a similar thing to what i did with modules here i've gone in here and said okay if you're going to making oh and clean and armatures clean and bet clean for each one what you do is you recurse into a directory and then you run clean and now what i what i did here if you remember i'm just writing here because it's obvious if each of these targets will be something like owen clean right and so this thing here this dollar subs uh which is kind of um a simple tech replacement will replace a dash with a space so what will happen is this guy here would take owen clean and turn it into owen space clean all right and then that means this whole command line here dollar make dash c would end up being you know make dash c owen clean and so this will be the directory which will be used with dollar c and the clean will be the target that gets built so that is kind of an ugly hack but it does work and you can do a little bit of trickery like that the other thing you could do is you could um split this this thing on this thing and get the two components um another way it might be a little bit less ugly um if this this thing essentially took um owen clean and turned it into owen clean which in gnu make sense is a list with two elements owen and clean and so you could use built-in um functions so i could use uh first word and i could do word okay let's just go back to that and just make dash in clean is that what i expected to do yeah okay what did i do there well knowing that this thing is a list with two elements the first which is the directory name and the second of which is the word clean um i went in here and i said okay just give me the first word of that list which is owen armitage or bet and then give me the second word which is clean that's a bit redundant since i know i'm doing clean so i could just do this that should do the same thing i'm using my dash in here just to see what commands get executed sorry i just said clean and that seems to work and um that will actually work and that will work nicely with recursive making your recursive cleaning of things um there are other ways you could split this but that turning things into space separated lists in gnu make is really powerful um suppose i had done this the other way around and i had clean at the beginning and the directory name was the second thing well there are other directives for that um so suppose i had done this don't add prefix clean dash so it'll now be clean owen clean bet clean armitage um first word will be the directory will be there will be the word clean so i want the second word and you can do word two that word picks the one two three you can pick a numbered element from the list let's just make sure that that will work yeah you see oh in armatures clean and let's just actually let it fire it goes off and it runs in every directory um and then uh there's make which i should go make everything so that's like a really basic introduction to recursive make this is a good way of doing it which is to use dollar make it allows you to do parallel makes oh yeah we should do a parallel clean shouldn't we so let's just do boom off it goes and it goes into each directory simultaneously and then obviously it's only running one command but it's not that exciting but it but it will work um in a recursive way um obviously it has to recurse all those directories which is a bit of a nuisance obviously i did a little bit of hackery here turning this thing into a list but that's really powerful thing you're going to make space separated lists um and you can you can play around with that and obviously you could generalize this where this target something else so you could you could generalize even further um if you found this ugly then you could create yourself a little function called split for example you could do something like um split is a thing that substitutes whatever its first argument was with a space its second argument and then here you just go call split all right that might make you feel a bit more comfortable because now it looks like this function is a thing that splits a list on some character the character being given here let's see if i didn't then screw that up there you go that seems to work just fine sometimes that makes you feel a little bit more comfortable what you're doing it's not so obtuse if you call something helpful like split all right that was a very rapid introduction to doing recursive make we've reworked this what i will do in a future episode is talk about how to make this non-recursive but still have those separate modules because very often what you want to be able to do is go into one module type make maybe make test because you want to run the test in the module but you'd like your overall structure to understand the complete structure where all the files are in the system so thank you very much for watching i hope this was useful um you know please study the glue mate manual it's the best reference on all this stuff and i will see you next time on this little glue mate course thanks very much