The GNU Make Course: Episode 6

All right, welcome to GNU Make Course, I think it's episode six. And today I want to talk about something a little bit different. So if you remember the last few episodes I've been talking about a particular makefile which I've been working on. This time I want to show you one weird trick that will help with makefiles. I already showed you one before actually which was the one with the print thing. If you remember this one print like that and then at echo we can just go dollar star equals dollar bracket dollar star this thing remember this weird trick which allows you to print out any variable. So let's define a variable ubar baz and then if I go back to here and I go make print foo it gives me the value of that variable. So that's super helpful, lets you print out a variable. There's another trick I want to show you which is to get some more information about actual command execution and it can be very helpful if you're trying to debug something. So let's start again we'll leave uh we'll leave that print thing in there you know when it comes in handy and let's just start out by making ourselves a really simple makefile. So let's go with something all and all is going to make something called a.foo and a.foo I'm going to do the simplest thing of all I'm just going to do a touch um but when I actually not use dollar I just make it really super clear. So this is very very simple and this command gets executed here and so if I go over to here and I type make it makes a.foo and a.foo is there on the file system and you saw the command get printed and obviously sometimes if you have a lot of commands being printed it can be kind of ugly and so you often find people doing at in front of the command. I did it here almost instinctively which suppression suppresses the command so let's just do that okay doesn't print the command now but sure enough a.foo got built so that's great and that suppresses commands but what if you want to turn the commands back on again well there's a couple of options so let's just go here there's make-in which will show you what the makefile would do if it was executed doesn't actually execute it so if we have a look isn't there didn't actually get made and that's great but what if you actually wanted to execute the makefile and see what happened even though you've got ads in there well two ways to do that I showed this way of doing it before which was to do this so so what I've done is I've defined a variable q which has got which is at and then it expands out to be at here so this actually what happens is before going to make looks at the command whether it should print it it expands out everything in the whole line so that will turn into an app that will suppress the touch and that will work normally so if I just go back I don't have that so I should go make doesn't nothing appears but sure enough there's the there's that file now what if I wanted to print the command well I can actually do an override on the command line and do this and there you can see the command got printed and what that did was it set the variable q to nothing and because it was nothing then this thing here would get expanded out to nothing and so the command would get printed and things that are defined on the command line of make override definitions that are in the makefile so you got q is there and then okay so it's going to get overridden to nothing we could override to wherever you wanted and that will work and that's great that's another thing but what if you wanted to go a little bit deeper and get a lot more information in particular where commands are coming from well I'm going to show you a little trick you can do um the the shell that gets executed for a new make is in a variable called shell and we can actually just print out what it is so let's just do that use info as a print you can do it like that a bin shirt or you can do print shell tell us okay shell is bin sure okay in this case now that's great but we can also just change the shell value so if I wanted to use something else shell bin bash right and then this thing should run fine if I just do and make print shell then bin bash okay and let's just move that so that we've actually made the file using bash okay great so you can change the shell um which means you can do other things with the shell um now there's one disadvantage of changing the shell if um you change the shell to something that you make doesn't understand you change that variable then it will always execute the shell uh even if it could have directly exact the um the the program being run so actually in this example we need to touch here because greenweight doesn't recognize anything shell specific on this command line it doesn't actually execute the shell it can actually go ahead and run touch directly with the environment of make which is an optimization which causes you to um you know not keep loading shells just to run one command and so typically maybe I was actually don't run the shell very often unless there's some shell specific characters and actually I couldn't make as a as a way of detecting that you're trying to use the shell but if you need to use it for debugging it's okay to change shell so what we can do is we can actually modify the shell the shell the definition of the shell to print out some information so for example I'm going to go here and if I say I'm going to save the value of shell what it currently is uh in a variable called underscore shell and notice I'm using colon equals here so I grab whatever its value is right now um and um that will allow it to um that will allow it to grab the actual variable that's in the shell now um after that we can then modify shell having got its value from before so we go in here we do shell equals and we'll say dollar underscore shell so this will do this will be a null up at this point it will just it'll just run the shell we have existing but we can actually modify this slightly so if we stick in something else at the beginning here so if we do warning this is a very interesting thing here so what we've got here is um you've got shell we'll run the shell this is actually the shell command this thing will get expanded every single command and we'll do this warning thing and the interesting thing about warning is that warning gives you the line number where things will run and this thing dollar at will give you the target that's being run so what'll happen is when we get to this touch here because we've redefined shell this will get expanded it'll figure out bin sure but it'll also while it's at it print out the context which will be the line number and so what we can do if i just go here this will work right i get a thing saying on line eight of the make file we were making a dot foo right and so here it was line eight this is where we were doing it see that line eight down the bottom there we were making a dot foo and you could expand this if you wanted to know if there were prerequisites anything like that you can you can make this thing up here much much more complicated um another nice thing you could do is you could add in dash x uh i think it's dash x right my memory memory serves me okay and notice dash x causes the shell to actually print out the command is executing with a plus in front of it and so now you've got a lot more information you've got the main file was building a dot through at line eight and it ran this command so this can be super useful if you wanted to um you know if you wanted to do something uh you know more complicated from a debugging perspective and you could go even further right you could use um you know dollar if in here and you could do it only for a particular target right so you could use if you could decide if you're on a specific target then only print out the information the rest of the time use a different shell so you can make this arbitrarily complicated if you want to target a particular part of the make file so that's kind of a neat trick for getting information about what's being run in the in the make file now what i'm going to do i want to expand this a little bit to show you um something which might be a little bit surprising so i'm going to make this make two files i'm going to say this okay so we're going to say make a.foo and b.foo and here's how to make them and if i go back here actually let me just make myself a little clean target because i'm getting a bit fed up with typing rm let's just do that okay so we should have no foo files anymore and here's what happened so it did a.foo it did b.foo and that seems to work fine and you might think oh this make file worked great it looked at exactly what i expected but actually this make file didn't do quite what you expected which is that this thing here looks like it says how to make a.foo and b .foo actually this is the same as saying how to make a.foo or b.foo so this thing is identical to doing that let me just write this out just so it's really quick let's just do that and that looks really weird right because this a.foo thing touches b.foo and this b.foo thing touches a.foo and that is in general a really bad thing to do in in make files it seems to work nicely in this example but if you notice it's only when it made a .foo that it did the to the touch of these of b.foo because it actually by the time it got around to making b.foo something else had done it and this is a great example of a side effect happening in a make file make doesn't know that a.foo updated b.foo finds out finds out afterwards doesn't run this and this could have had different commands in it right so if it had something else some other way of touching so you have to be really explicit in make about what a particular target it builds and not have side effects like this or at least not have side effects of files that make might actually look at later on so don't ever think that something for a colon with a space in it is a sort of atomic single thing of multiple files it's not it's actually equivalent to this all right i'm going to show you another weird thing so here you've got make a.foo and a .foo will make b.foo as well and i'm going to add some on this all i'm actually going to add some commands i'm going to add two things i'm going to say okay so all is going to make a.foo which will make amb and then it's going to use dollar wildcard which remember is that thing for getting a list of files and print out that and it's going to do an ls to also do what should be the same thing and if i go back here and i type make clean um can you guess what's going to happen well if you're used to going to make a lot you might but if you haven't you might be really surprised so what happened okay line 10 the rule for a.foo got run it touched a.foo touched b.foo let's just go back and look at that this is line 10 so that's where it got executed okay great so it did those two things and then we got to all and it did an echo and it echoed nothing and so that means that this wildcard didn't find any files called.foo but immediately afterwards line six again let's go back line six up here so we were executing it did an ls and it found files on the file system so this looks really weird why was wildcard empty there okay well i'm going to make this slightly more complicated i'm going to add an output directory i'm going to say we'll put the files in let's make a directory called out and we'll just change this so that it it uses out everywhere normally you would use dollar out here but i want it to be very explicit so you can see what was happening with all the different file names and this just let's do that okay so we've got a directory so what's it going to do is going to make something in some directory same thing but in a sub directory okay let's go back here um and let's just do the following um let me just make a directory um okay make out okay and let's run make okay so what happened this time uh we got a um the following happened it touched it touched a.foo it touched b .foo both in the out directory and then it didn't echo and this time it listed something look at the difference the echo above here there was nothing and now suddenly it finds those files in that directory so that's really weird okay let's try and just try and see why that is i'm going to do one last experiment which is i'm going to override make the out out variable and i'm going to say in the current directory so let's just let's just do so okay and that echoed and showed those files let's just let's just make sure that uh here we go if those files were not existing it didn't find them so what's happening here why if it's in a sub directory does it work and if it's in the top level directory it doesn't work well here's the here's the issue canoemate keeps inside itself a um a cache of directories that it has already done a glob on and so what happens here is when you go to this directory called out canoemake has never actually gone in there and had to get a list of files in there and so when it does wildcard here it actually goes and gets that list of files and discovers that it is made a.foo and b .foo so you get the same output from wildcard as you do from this ls because they because the cache is loaded but on the one where out was empty so if we just do that you don't get it and the reason is canoemake at the beginning because you're dealing with all which could be in the current directory and because it's in the current directory it has already actually filled the cache for that directory and so this is a real problem which is that if you use wildcard in this way assuming that it'll give you a list of what's in in the in the current in the directory you're looking at it may not at runtime because canoemake is building a directory cache and it will actually intelligently fill up that cache if it knows the files get built at certain times but it can be really dangerous so if you're going to use wildcard I always recommend that you use it at the top level of the of the directory so you only use it here up here so you say you know foo files so like that because that is guaranteed to work it's here but if you use it someplace else then it's not necessarily guaranteed to work so that's a real gotcha with wildcard that can do really weird things and so it's you know this is a really good way good way to do things so bear this bear this in mind if you're if you're doing anything wildcard can be kind of dangerous here the shell trick here is really helpful you can do much much more interesting things with it but don't have side effects like this in directories now there is a time when you can have a side effect and that's in a pattern rule so it's you know you can do something like this if you have a pattern then it will treat it as atomic which is a really weird thing but and the reason for that is there are certain programs certain compilers that produce two output files and so they can you know then then it can be really useful okay so that's some tricks with wildcard and with the shell variable and you know bear those in mind if you're going to if you're going to use things this trick with the dash x can be helpful when debugging make dash n can be helpful there's another make thing which is sorry let's do this which is make dash p and make dash p dumps you an enormous amount of information about make so what it will tell you is it'll give you information about all of the uh all of the commands all the targets that are in the make file and all the information about it now it can be useful but for any realistic make file it's really really hard to actually use that information so i highly recommend that you actually use something like this trick the dollar the dollar the dollar q trick i showed you or um you know the print thing here is also very very useful there's another little enhancement you can do with print which is there are some uh there are some built-in functions that can tell you where a variable came from let's add this into here because it's kind of fun so let's just do this see it says file here the reason it says file there is because um the out variable was defined in a file in this case it's in the make file but let's suppose i overrode it on the command line out equals temp you know it says command line here and if it was defined in the environment um let's do something else out there you'd see it say environment that sometimes can be useful if you want to know where a particular variable came from whether it was overridden and you can use that with some of the preprocessor stuff which i'm going to talk about next time and there's also a thing which is value value is an interesting thing so let's have a look at value so let's say print um let's say print out first of all well out i get the same thing this is the value of it and this is what dollar value replied and dollar value what it's giving you is the stored value not the expanded value now in this case because out is this we just define with colon equals they're the same thing but this one which i defined uh recursively here equals will not be the same thing so let's just have a look at that one so um why is that expanded out oh this is a rather confusing example because of the way in which shell is handled let's just make ourselves another other variable um let's call it out copy equals dot out all the thing for why that's not working on this version but it should be able to give you the um the unexpanded value of a variable let's just uh let's just look that up so oh you know what i bet i have the wrong version of make on here yes sorry i have the wrong question but value could give you the unexpanded value which can be can be very helpful i'm going to show you a couple of other little things you can do that are useful sometimes when working with paths so um i in this food files thing here i got all of the files in the output directory so let me just do print i print food files rise those list of directories if we wanted to extract from those just the directories we could do this do this okay let's just do that let's just print that out food is you see it says out out here that's the directory extracted from each of these um each of these file paths and you know how you've got duplicates there and you don't want them well it turns out the the sort function in guinea make actually does sort and unique see so now you get that so if you if you wanted to get just a list of directories from a list of files you can do this you can do sort there's also an equivalent thing which is called not i just run that see that gives you the path that's not a directory so it's useful for doing splitting of things if you need to do something complicated sometimes like when you have a command here you might need to extract say this part the file name part from it because it's um you need it for some command we're going to put it in a log message you don't need the full the full path but not there will give you that and uh will give you the the directory part so that that those little things can be handy too for doing stuff and especially sort for removing for removing duplicates so you see that look there is not a separate um unique function if you wanted to do not sorted but unique then you'd have to write it yourselves which would be a fun a fun little task for somebody to do it's not not terribly hard to do as a recursive function maybe we'll do that next time okay so it gives you like shell directory kind of stuff while carding and how it can go horribly wrong um and you know how to hide how to hide uh the commands that you're executing let's clean this up a bit because i don't think i really need this i think i really need that let's put an act in here for cleanliness okay and this thing and this thing really shouldn't be touching that if you want to clean this up completely you would do something like that um if on the other hand let's suppose you really wanted to make you know multiple commands with exactly the same thing it's valid to use the fact that before a colon if there's a space it's a list and do the same thing so suppose that we actually had you know we wanted to do something like this um let me see there's two of these files i was going to take this out so we don't need any more and then you just said in here look for all of those things run touch and you know what to my liking this may pose a bit messy i can do this if you hadn't seen this before i've used this before but if you have a single line command in a makefile you can use semicolon like this rather than on a separate line just tends to make your make your make files a bit neater so now i'm using the fact that this is going to repeat the same command for each of those things and i'm using dollar at here so it picks it up and what we should be able to do is go in here we're going to make clean and then make um yeah and the out directory and there you go the two files have been have been created so this is okay to do um if you if you want to do a as long as this command only touches one single thing and when if it's time to have a side effects you're going to have a really big problem so that's great um the little the little directory trick i showed you we could actually make sure the out directory is built so if we wanted to figure out what it was obviously we know what it is here but if we didn't if we if this had a lot of different directories in it we could do you know make these directories uh sort directory and this will give you a list of directories that need to be made and then you can use one of the techniques i showed last time for making directories uh my favorite which is the one where you have a a dot f file so let me just show you how you do that you can remember it you do something like this you say um send or f and um you tell it to um make the directory portion blur out and then touch dot in fact it'll be clearer if you do that you do a thing like that and then what will happen there is it'll make a directory i'm really typing badly today and then touch a file in there and then you can say you know it make all those directories and put and touch that file so you can do something like markers um let's just make sure this works okay so let's print out some of these variables um um food files is what okay and then um make there is okay it's only that thing that needs to be made and then print out the markers all right so it's going to tell it to make a directory and put a file in it and the reason using a file is the it's timestamp will only change if the file changes um so let's just go do this rm minus rf out okay see what it did it went in there and it made the directory and it made a file let's just go back okay let's put in some hats there'd be better so this will make sure those those directory exists and then make sure the file exists now because these marker things are actually um you know they're they are files we can treat them as files so we can actually say here that um sorry this this guy each food file will depend on the um the directory being made so you can actually go in here and you can say something like the directory of um well let's just do it explicitly because otherwise it'll be a little bit confusing so if we say out um out a.foo right depends on out slash dot f and then touch all right so i'll make sure the directory gets built beforehand and let's just do that so what am i doing here oh yeah straight slash in there so we should actually have gotten our directory yeah with that in it so it got built so this was this can just be treated like a file and that will make sure that the directory is built at the right time uh using this and so you can then go one further which is to use you can actually use dir get the directory of the target and you have to use double expansion to do that but these this way you can really make this so that you're really minimizing in the make file the number of times you have to repeat yourself and that because i hate repetition to make files because they're so complicated to maintain if you copy and paste stuff around so really look into using all that functionality okay so next week i'll get back to our original make files and i want to go on to the next step which was the linking step because i didn't have anything like a linker in there and that will happen and we'll show that and then after that i want to get back to recursive make which i've already treated one time and how to eliminate it completely because recursive make has lots of its own problems so what do we get this week we got the shell trick which you can look up online if you want to the print the print trick making directories with dot f files how wildcard can go horribly wrong because of gnu makes cash and caching functionality for wildcard and eliminate eliminate eliminate redundancy because it drives me crazy all right well that's it for me this week thank you very much for watching i hope you enjoyed watching me type and delete things and i'll be back next week with more gnu make course bye