The GNU Make Course: Episode 1

Alright, well, it's 2.30 here in Lisbon, so I'm going to go live and talk about GNU Make. And what I'm going to do in this course is start right from the very basics and explain how GNU Make works and what you can do with it using some examples that I've invented. And hopefully this will give people an idea of the power of the program. In this first episode, I am going to be doing something very, very basic. So I'm going to start with really basic GNU Make file setup. I'm going to go through an example. And if you've used GNU Make before, you may think, oh, this is really boring. I know this stuff. That's fine. Tune out. Come into another episode later on when I get into some of the more advanced stuff. So first of all, what is Make for? Well, the origins of Make are really around compiling programs. And once you get a large number of, say, C files, and that was really the origin was around C, creating all the object files, linking them together into a program becomes a bit messy. And what people would do is they would do that in Bash or whatever shell they were using. And that was a bit messy in itself. And so Make was invented. And the idea is that Make looks at the relationship between files and will make files, hence the name Make, if another file changes. And so what I'm going to do in this example is run through something slightly made up, but it's designed to illustrate the sorts of things that happen in Make files. Now, I'm not going to do compilation because I don't want to get focused on a particular language. I've invented a scenario. It's a little bit silly, but hopefully it will give you the concepts and then we can build up from there. So I've got an editor open. I happen to like Emacs. I know that many of you like VI. Please don't write into me about the editor you should be using. But I'm going to use this, and I've got a bunch of files in here, so I'm going to have a quick look in here. And so in here, I've got this set of poem files. Armitage 1 poem, Bed 1 poem, Bed 2 poem, and OM1 poem. And these are poems. So if I look at one of them, let's have a look at Armitage 1. So this is a poem by Simon Armitage, and that poem is there just as a text file. So these are just text files. And I'm going to do a kind of crazy thing, which is I'm going to, for every poem file, I'm going to make another file which contains the number of lines in this poem and the number of words, just using WC. So if you look at that particular poem, you could do something like this, the Armitage poem, and you get some output like this, right? So 101 lines, 455 words in that poem. And you could do the same for every poem I've got in here. Now, this I'm using as this is my compilation step. And what I'm going to do is I'm going to take this data and store it in a file. So for this, I'm going to end up storing it in a file called Armitage .count. So this would be a little bit like if you were compiling something, you would say, you know, you'd use GCC or something, and you take your source file, in this case, a poem, and you turn it into an object file, in this case, a count. So if we look at that last thing in there, there it is. OK, so that's the same thing. So and then ultimately, I'm actually going to merge these things together to make one file. That would be a little bit my linker step. Now, obviously, I could do exactly what I'm talking about using bash, using a for loop. I could just go through every file in here and create an equivalent egg .count file. Really quite simple to do. But I'm going to use make to do this. So first of all, I've got make installed on here. So I type make. OK, so make says this. I don't have a make file here. And there's an important word in here, which is it says no target specified and no make file found. So let's just talk about some of the language of make. A target is a file that make is going to create. So let's have a look. First of all, I'm just going to get rid of any count files I've got. So that's like getting rid of my object files. And I'm going to switch to a make file. So here's a blank make file. There's nothing in it. If I wanted to put something in, I could put a comment made by John and so on. Now I've got a make file. So let's go back to the shell and type make. It says no targets. It doesn't know how to build anything. OK, let's just try. So I would actually like it to make armitage1.count. OK, it doesn't know how to make it. There's no rule to make it. So let's go in the make language and actually create that rule. So we're going to say armitage1.count. So we're going to make a file called that. And we're going to make it from the equivalent poem. So there it is. And the thing on the left of the colon here is called a target. The thing to the right is a prerequisite. And fundamentally, these are files. It's always very important to realize that make uses files. And if the poem file changes, then the count file will be updated using what's called the recipe. And if you've used make, you will know that make loves to use tabs for the recipe. So what's the recipe? Well, the recipe is that command that we did before. OK, armitage1.poem and output it into armitage1.count. Just save that. OK, so we've got a very simple make file. Actually, I'm going to get rid of this comment. It's a bit useless. All right. So this take says if the poem file changes, write out this file called count. This is the compilation step like that. So let's go back to the here and let's run it. OK, so it ran that command. If we have a look, there's now a count file there. And if I just look at that, there's that equivalent thing. OK, that's great. So I've got that one. Now, I've actually got a whole bunch of poems in here. So what I can do is I can go and make my make file do a bit more work. So let's go back in. I'm going to do this. I'm going to do the really naive thing. I'm just going to go here and go. I'm just going to change armitage to Betjeman. There you go. So that's going to do the Betjeman one. OK, and then I'm going to do this one. I'm just going to do this line. And no one would actually make a make file like this because there's a huge amount of repetition in here. And let's do this. There's also one from Owen as well, isn't there? So we're just going to change Bet to Owen. That's actually dash one. Let's just change that. OK. So now we've got a make file that knows how to make dot count files for each of the poems that are in there. Let's just check that I got all of them. Armitage, two John Betjeman poems, one by Wilfred Owen. Great. OK, so there it is. All right, so let's now run this make file. OK, it doesn't do anything. Now, why does it not do anything? Well, what happens in a make file is it parses it. It goes through and it's learned about all these recipes. And the first target that's mentioned is the one that will be made by make. So actually what's happened in here, it's come against Armitage one dot count. We've actually already made that. I just made it a minute ago. It's newer. The dot count file is newer than the equivalent dot poem. So it has nothing to do. And in fact, it said it's up to date. So there's nothing we can do. Now we could force it. Let's just go through and just say I'll just touch that file, the poem file, and just so actually run a command there. And two things going on here. One is make fundamentally deals with timestamps on files. It doesn't look at, say, a hash of the file to see if it's changed. It looks exclusively at the timestamp. And so touch is enough to cause it to be updated. And the count file got updated. In fact, we could just take a look at that. And you can see the timestamp on here. The poem changed to 1437. And a little bit after that, I changed the count file. Okay. But it only did the armitage one. And in my make file, I had a whole lot of other stuff. That's because the first target it found is this one. So if we want to tell it to do more than that, we need to be very, very explicit. And what you'll typically see in a make file is something like this, all. Now what's interesting here is all is not actually a file, even though from make's perspective, everything is a file. And in fact, if there was a file called all, this would have really weird effects. So we're going to pretend there isn't a file called all, and I'll show you how to deal with that in a minute. So we're going to say we're going to make the count. I want to make the John Betjeman count, number two count, and the Wilfred Owen count. Oops. Get to watch me type live. Okay. So now, because all is the first thing, this should work. So let's just go through. Let's just clean up those count files just so that we can do it. Boom. Okay. So now I've made everything, and now if I have a look, I've got a whole load of files, equivalent count files for every poem. Great. So that has now worked. Now, one of the interesting things is this all at the top here. What you'll often see in a make file is this, phony all. You'll often see these dot directives like this with a dot here. This looks a bit like it's a recipe for doing something, but actually it's an instruction to make about something. This is saying that this thing called all is not a file, and you should ignore it if there is a file called all on the file system. And, in fact, you're not going to make a file. And so this is just saying this all thing is essentially sort of virtual, in a way, and it's saying if in order to make all, you have to make all these count files. And then you can see all the commands for making the count files. Now, this is okay for what we've done here in this simple make file, but if I had a lot more poems, this would be a real mess, and you wouldn't want to do things this way, because you've got this repetition. In fact, if you look at these commands here, these commands are all the same, and there's a huge amount of error you could have in here. I could accidentally do something like that, and now I'm making the wrong count file. So that would be great if I did that. That would mess everything up. So you really want to use a recipe that can be shared between the different things, and there's a few ways to do this, but I'm going to use what's called a pattern rule. So I'm going to say this. Okay, so percent count colon percent poem. What that means is that anything ending in count can be made from the equivalent thing named poem. So foo.count can be made from foo.poem using a particular recipe rule. So this is going to be wc-wl. I need to now refer to the things that I'm actually doing wc on. Now, originally, I could do that with the specific names, because I had specific rules, but here I can't. So I can do this. So I can say dollar less than goes to dollar at. And now I don't need any of this stuff. So we've said all makes those things from this rule. So let's just go back to here. Let's just kill all those count files. Let's try making it. Okay, so now it worked. Let's go back and see how that worked. Came in here and said all is phony. These are the things you have to make. We're trying to make these things called dot count. And then here's a rule to make a dot count from a dot poem. And because all those dot poems existed, then it worked well. If I were to add, let's just add on here. I suppose there was another Wilfred Owen poem. I'll count like that, and let's go back and just do this. Okay, it doesn't know how to do it, because the equivalent Owen-2.poem doesn't exist. So this is not going to make that file for me. It's just going to say, okay, I've got to go look for it, and that's the equivalent thing. So let's just take that off for a minute. Now, the other thing that would get difficult with this is if I added a lot more poems, that all line would become exceedingly long. And so we can use some variables to make that simpler. So I'm going to do that. But I'm going to go back and look at a couple of variables here. Within this recipe here, I used two things. I used $lessThan and $at. And within Gume, there are a load of what are called automatic variables. And so $at is whatever this file is. So it's whatever this thing matched. So when we were making, for example, armitage-1.count, and this rule was used to make it, this $at will be armitage-1.count. And the $lessThan will actually match the things on the right-hand side here, the prerequisites. So this will actually expand out to be what you want using $at and $lessThan. There's a whole load more automatic variables we can talk about, but I'll talk about those later. And in fact, if we just switch back, you can see there in here how it expanded those things out, the $lessThan and the $at. So now in this rule here, we've got one rule which does everything, and that's very, very convenient. Now, if you really hate tabs in Makefiles, which can be weird because, for example, if I were to do this and then I go back and I type make, it's going to complain. It's going to say missing separator. You have to use tabs, and that's kind of an easy thing to fall into. But there is a shortcut. In the case that you have one line in your recipe, obviously many recipes have multiple lines, you can actually do this. You can put a semicolon here, and that is a one -line recipe. It doesn't have a tab in it. It's actually quite nice for quite compact things. So I'm just going to go back here and get rid of those count files again. There you go. It still works. Great. So that's fantastic. So that still works. That's a one-line short thing. Another thing you see really quite common in Makefiles is that people don't want the commands printed, and they do this, put an at at the beginning. The at suppresses the command from being printed when it comes out. So we can go back here. Let me just do that. And now it doesn't do anything, but at least it did. It actually made those files, but it didn't print anything. And if we really wanted it to print them, we can do make-n. Make-n will tell us what it would have done had it actually executed. So it'll actually show the commands. That's useful sometimes for debugging. Although, to be honest with you, once your Makefile gets really big, the output from make-n can be super hard to parse. Anyway, back to this Makefile where I'm trying to simplify things. I've got this one line here. I'm going to go put this back to the tab style just because most people are familiar with it. And back to this problem of this huge list of variables here, this huge list of files to be made. Now, what we can do is we can put those in a variable, and then it might be a little bit easier to deal with. So what I'm going to do is I'm going to say I'm going to call this count files. And I'm going to go up here. So first of all, let's just go make sure this actually works. Get those around. And let's have a look. Yeah, it made all those files. Great. Okay, so this did work. Now, what happened here? Here, I'm defining a variable called count underscore files. It's quite common to use uppercase for variable names in Make. And this is the assignment operator, and here it is. And this is a string, but it is also a list. So one of the peculiarities of Make or one of the things you need to get really used to is that anything with spaces in it is a list. So this thing here is a list of files, in this case file names. And here, when the Make file is parsed and before any execution starts happening, this reference to count files here in parens, it can also be in braces if you prefer, like this. Now, I tend to use parens like that. That will get expanded into literally that string, which is a list. And therefore, all will depend on all those dot count files, and therefore, we'll go looking for a rule to make them. We have a rule to make them. Do we have the equivalent poem files? Yes, we do, and we can do everything. And so this is sort of the basics of a variable within a Make file. Now, if you've looked at other Make files, you might have seen an assignment like this. Let's just see. Does that work? Let's try it out. Yeah, there we go. Those things are there. You know what? I'm going to remove that at, just so it becomes a little bit clearer. So these two things were both the same. The colon equals and the equals both worked in this instance. What's the difference? The difference is how the right-hand side of this assignment is treated. So what happens with the equals is this is just setting literally to whatever the string is on the right-hand side. And later on, when the variable is used, i.e. when it's used here, Make will actually, what's called, expand that variable and look for other variable references. Now, in this case, I don't have any other variable references, but let's just go make some more. So let's suppose that we had, let's suppose we had war. So we have the war poets all together. So that's Owen. And then we'll have, let's just say modern files. That will be Simon Armitage. And what should we call Betjeman? We should call sir files, since he had a knighthood. And I don't think Simon Armitage has been knighted yet. He probably should be. Okay. So now I'm going to do this. So we're going to say that the count files we want to build are, well, we'll have all the sir files. We'll have the modern poets. And we'll have the war files. Okay, so this here, this count definition will turn out to be all of these strings put together. The extra spaces here will end up getting turned into single spaces. I'll show you how that, what that looks like in a minute. But to illustrate the difference between equals and colon equals, here, when count files is used, wherever it's used, it is necessary to recurse through all of the variables. So it'll have to go, oh, count files consists of sir files, modern files. Okay, I'll go get those values. And in large make files, that can turn out to be actually a problem. Because what can happen is that the recursion can be huge. And if you've ever hit make, and there's been a really big pause before the make file starts doing things, that is often caused by the make file just having incredible depth of these recursive variables. And now, recursive variables are great, because it doesn't really matter what order I define them in. So, for example, well, first of all, let's just make sure this still works before I do anything, because you don't want me to have messed it up. Okay, whoops. Let me see, I can't type. Okay, all right, great, it still works. It does the same thing, slightly different order, but that doesn't really matter. So it works just fine. Now, what's interesting is, because the variable is only actually expanded at the time in which it's referenced, I can reorganize this make file and do that. Let's just do the... Oh, yes. Silly me, you can't quite do what I wanted to do, because it's actually got used. Let me just go here, like this. I'll show you what I did in a minute and why I messed it up. Okay, there you go, it works. So what I messed up was I put it too far down. But if you look at what happens is, count files references here at the top, a bunch of variables that don't actually exist yet, and then they get defined. And it's only when it's used that it actually gets expanded and actually turns into that list of count files. So let's have a look at just one way to look at this. Make has a little thing called info. Let's just go here and type this. Notice that blank line there? That blank line is actually this info command. This is a bit like a print command, equivalent of printf debugging. And at this point, count files was an empty string, because these other things were not defined. If I go in and just insert my printfs everywhere. Oops. What's going on here? What have I messed up? That should work. Why is that not working? Okay, that's interesting. Oh, am I using too old a version of Make? Okay. My explanation isn't going to work out great here. This is why we do these things live. But what happens is, this variable gets expanded right here, and it then turns into the long set of these strings. And they can get defined later. And, in fact, a lot of Make files will use this style, because it's more flexible. You can say, well, I'm going to do this, and then I'm going to go get the list of files from someplace else later on. It has the performance impact if these things are very large. What I did before was I used colon equals. Now, colon equals gets set when that variable is set. So when you use the colon equals, the right-hand side gets expanded right then. So surf files, modern files, war files better have the right values in it. Otherwise, it won't work. So now it should work. Okay. It works quite nicely. But if I had tried to do that up here, it doesn't have anything to do. Why doesn't it have anything to do? Well, because right here, it got set. These things got expanded. They hadn't been set yet. They got set later. So count files was empty. So if you're using Make a lot, I always get into the habit of using colon equals. It has a slight cost that you have to use your brain a little bit to remember the order in which you do things. Now, the other thing here that's perhaps a little bit different to how you might do things in a Make file is this is talking about essentially what are object files, the dot count files here. And normally, you probably would talk about the source files, which are actually the poems. So what we can do is I'm going to modify this. So I'm going to save this. Okay. So I'm going to save. So these are now actually going to be the source files. Now, this is not going to work because it doesn't know how to make that. So what I need to do here is I need to tell it to make dot count from dot poem. It always needs to know what target it's making. And so we can use a couple of different things here. We can change the suffix. So we should be able to do something like. Let's try and see if that will do what I want. Oh, yeah, better. Just my own advice and put that in the right place. Okay, it made things. Now, let's have a look at what's happened here. This thing, count files colon dot poem equals dot count. What this does is it actually changes the suffix of, changes suffixes. So it goes through this list, ser files, modern files, war files, and changes them all to dot count. And you'll often see this in programs where you're changing all the dot C to dot O files. Okay, we've done all that. Now, this is great, but it means manually keeping a list in here of all of the poems we have. So one other thing we'd like to do is possibly just go and get a list from the disk, essentially do the equivalent of LS. And so I'm going to change this completely. And I'm going to go here and I'm going to say wildcard star dot poem. Okay, so what wildcard is going to do is it's going to go onto the disk. It's going to do essentially an LS, get a list of all of the files that are dot poem. And then the transformation will transform dot poems dot count. Or we'll say these are all the dot count files that need to be made. And then the last thing is this percent dot count, comma percent dot poem, which tells you how to make a count file from a poem file. So hopefully this will work. Now, one of the things that's really a bit tricky here in Make is you notice that this is stars. So this is like a glob pattern, like what you'd be familiar with in the shell. And you notice here it uses percent. So one of the ways you get tripped up in Make is you might think that that would actually be a pattern match and that would be how you define a rule. It's not. You have to use percent here. And there's a few other places where percent becomes important. And I'll talk about those as we go through this course. Okay, let's go back to here. Same thing. But in this case, the Make file didn't have to know how, what the files were called ahead of time. So if we added another one, and I'm actually just going to do that. I don't actually know. Well, I guess I could pretend I know another John Betjeman poem. So how about we start with Miss Joan Hunter Dunn. Miss Joan Hunter Dunn. It goes like this, doesn't it? Furnished and burnished by all the shot sun. Okay, that's about all I remember of that particular poem. Make file is not going to change. Let's go back to the shell. It's just for fun. Let's just do that. And you can see right there it's made that particular extra one, the third John Betjeman poem in there. So this is really super simple Make file. It takes, you know, it gets a list of poem files, turns them all into .count files, and uses this rule to actually make them. You might see another style sometimes, which is something like this, as a rule for making things, which is .poem.count. This is slightly older format. You quite often see it in old Make files, things like this, .c.o, how to make an O file from a C file. I tend to be a bit pedantic, so I tend to like to type stuff like this, and I think it's a bit clearer for if you're learning how to use Make, but you can choose any particular style. Now, before I go, I'm going to show you one little trick with a variable, which is I'm going to define a variable called queue, which contains the at, and I'm actually going to use it right here. And what I'm going to do with this, well, actually, let's just run it. Okay. So nothing came out. Why? Well, in here, when it came time to run this command, there was expansion that happened. So remember, the $less than got turned into the name of the poem, the $at into the .count file, and this $q, well, it turned into at, and at is the thing that suppresses a command, so this was equivalent to that. And so, okay, that's great, but why put it in a variable? Well, because we can do something else. So what I did there was on the Make command line, I redefined the variable queue to be nothing, and that means it expanded in the Make file to be nothing right here, which means I got to see the commands. So that's actually kind of a cool way. Instead of using Make-N, which actually doesn't do work, if you just want to get it to temporarily print out the commands, instead of using at, put a variable like q for quiet, and I override it on the command line. And as a general rule, if you define something on the command line like this, it overrides the thing that's inside the Make file, so you could override anything that's in there. So we could actually go in, and you see this thing where it says count files, we can decide I'm going to override that. I could say make count files equals batchman one dot count. And in fact, let's just do the equals bet one dot count. Okay, it's going to be set. Queue as well. What it would do is it would just make one file, just as you would just do with LS as well. So you see, there's just the one bet one dot count there. So you can actually override anything that's in the Make file from the command line. All right, I have got only a few seconds left. This has been really basic introduction to what you can do in Make, to give you some of the syntax. I'm going to come back next week, expand this a little bit further, hopefully figure out why I couldn't use dollar info, what was going on there. It was all live coding. And hopefully this is useful. If you've used GNU Make, wait a few weeks, we'll get into some deeper stuff. And thanks very much. If you have any questions, send them to the live studio.