The GNU Make Course: Episode 2

Okay, so welcome back to The GNU Make Course, where I am working through from the very beginning how to use GNU Make to make things, and this is the second episode.

If we just review what I did in the last one, it was something very simple using a few files that I generated.

So let's have a look. I've got a Makefile, which was the thing I created last time, and I have a bunch of poems, and they literally are poems.

So let's just look at one of these. So that's a poem by Wilfred Owen, and the Makefile takes each poem and creates a file with the same name, but with the extension .count, containing the word count and the line count in that poem, and this part of the Makefile, the all here, causes it to make a count file for every poem file.

So let's just run that, and we just go make here. Okay, what do we get?

So it runs wc for each of those things and creates count files, and if we take a look, we've got a bunch of those, and if we just take a look at this count file, there you go.

That's the output from it, and the reason I was doing that was I was using the idea of wc as being kind of like a compiler.

So you can think of this as source code files, which are poems, and object files, which are counts of words and lines.

A little bit artificial, but the reason I did it was I didn't want to concentrate on an actual compilation, because people will get distracted by things like, wait, why is he using GCC and not this compiler, or why is this all about C programs, and I'm really interested in Rust or whatever.

The idea was to demonstrate some of the functionality of GNU Make.

So here we've got star.poem, things that are poems, and the equivalent file is called .count, which is just like you might have if you're doing like .c to .obj, or .py to .pyc, or something like that.

So just think about that. This is a compilation step, and if you remember last time, what I did was a couple of things.

I created a variable here called count files, which contains all of the files you want to create, and then it's made up of these three other variables, war files, that's war poets, modern files for modern poets, and ser files for poets who were knighted.

So it's just completely artificial, but you can imagine for a program having different parts of the program in different variables, and we have this variable called count files, and then what happens in here is we say, okay, in order to make all, remember the first thing in a make file is the thing that will be made.

It's the first target that make comes across will be made, and what we do is we transform all those .poem file names into .count.

So this is saying make, owen1 .count, bet1.count, etc.

It's going to be a long list of just all those file names, and then this rule, which contains a pattern, the percent pattern, which is a wildcard, shows you how to make a .count from an equivalent .poem.

So the percent here will match the percent here, and if you look back, this is exactly what happened.

It said, you know, bet1.poem was transformed into bet1.count, and so on. Now, one thing to think about with guineamake is that the way in which this part of the make file works is primarily around matching strings against patterns or rules that are in here with a fixed value.

So what happened when this make file was parsed, it was discovered that all had to be made, and all was what was called a phony target.

It's not a real file that's going to be made, because fundamentally, guineamake makes files, and in fact, whenever you have something which is a target, be it a pattern one here or a specific file name, and these are the prerequisites, the thing it's made from, those things should be real files.

Always think about this as real files, and if you're not, then you use this phony thing to say, no, no, this is something special.

In this case, it's very common to have a thing called all, and you know, it says all depends on you have to make these things before you can make all, and so what are you making?

There's .count files, and so on.

So it's building this graph of things to be made, and this thing here is just a string.

In fact, we can kind of print it out, so we just use info. We just do the same thing.

Well, let's start with count files and just show what's in there. So we do that.

You can see this print, this line here is printed out. That was all the .poem files in the directory, and why, how do we get them?

Well, it's this definition up here, and let's just do the transformation.

So we're going to say .poem equals .count, and let's just go back and run that again, and you can see how all the file names got changed by that transformation.

You can see that here, everything with .poem turned into .count.

So this rule here, all colon this variable expansion, actually is equivalent to writing all colon, and then naming every file.

So GNU Make is working just with strings here. It's not doing anything clever to search for things, and it's just string matching.

So beware of that. There's, in a way, less going on here than meets the eye, because sometimes people expect things to automatically find the files.

For example, a quite common pitfall, I'm going to put a comment in here.

I'm going to comment that piece of code, that out, and then let me just get rid of the count files.

It says nothing to be done for all, and the reason it says that is all, it doesn't know what to do.

This thing here, it has no idea what files exist, so it won't start pattern matching.

It's not wild-carding or globbing to find those files.

So you have to be super specific about what it's going to do.

So I'm going to transform this a little bit in a few ways that I like to do, and then we'll take it from here and make it a little bit more complicated.

But first of all, this .info technique here is good, but it's a bit like printf style debugging.

You have to go in and keep modifying the makefile.

So one thing I really like to add to all of my makefiles is this, and this is a super little hack.

So let's see what this does.

So I'm going to say dollar star equals dollar star. Okay, let's just go back.

Before I explain what that does, let's go try using it. So it's going to make print count files.

Okay, now what's interesting is I did one interesting thing, and I'll show you how it works.

Normally when you type make, it's going to run the first thing that's found in the makefile, in this case it's the all, and then all of the rules associated with it.

But above, where I did make print count files, what I did there was I said make this specific thing.

So make went looking for a rule to make it, and it came across this thing, print dash.

So you can imagine this is a wildcard, and I used the single line version of a recipe here.

I could have done that with a tab in there, as is typically make. That would work just as well.

Let's try it out. There you go, works just as well. But I prefer the single line because this is kind of a one-liner in make, and that semicolon version is very good.

And what it does is it's going to say, in the case where I did make print dash count files, this percent is going to match count files, and it will be stored in the automatic variable dollar star.

You'll notice above I used dollar less than, that's the prerequisites, dollar at, the target, that's how the wc command got put together.

Here I'm using another automatic variable, dollar star, which will print out whatever the percent matches.

And then I'm doing something a little bit more complicated, and sometimes this causes people to go, huh, what are you doing?

I'm doing dollar in parens of dollar star. And what will happen is when GnuMake expands this out to figure out what's happening, the dollar star will be count underscore files, the literal string count underscore files, as it will be here.

And so it will then turn into, it's a bit like it was this. And then it will say, oh, I've got dollar parens, so this is a variable reference, and I will now go look up the variable that is being referenced.

So by doing this, you're actually a dynamically referred to a specific variable, and that can be set, in our case, on the command line, but it could be set elsewhere within the makefile.

And if you look in some complex makefiles, you will see this kind of thing where there's dollars nested, like there's variable expansions where the name of the variable is actually being computed.

So let's just go back to this example. So count files was there.

Let's say all the serve files are there, all those ones, which was all of the John Betjeman ones.

And so this is a super, super one-line thing you should add to every single makefile you ever come across, because it is very, very helpful to try and debug what's happening.

Now, you can't actually go in here and type in dollar count files, colon, and all that, and get this particular expansion.

Or if you try to, it can get very messy because of the way in which escaping works and things like that in GNU Make.

But you could always just store that in a variable, and then you could take it from there.

So print dash percent is super, super useful.

Add it to every file you have. Now, I want to work on this a little bit, because I did this last week, and there's a few things here I don't kind of like, and I want to clean it up.

So first of all, we need to be very clear about what's being built and what is being built from.

So at the top here, I have this thing called count files.

And actually, if you look at it, it's coming from these things which are .poem.

So these are really the poems. They're not actually the .counts.

And since we typically be dealing with source and object files, I'm actually going to change the name of this and call it source files.

So this is going to be the list of all the poems.

And then the output would typically be object files.

So I'm going to do that myself here. I'm just going to take this, and do that, and then use object files here.

And this makes it a little bit clearer that what all it's doing is it's depending on object files, the .count files, to be built.

And then this should work just as before. One other thing I'm going to do is I'm going to use colon equals.

I talked about this a little bit last week.

But every time you expand a variable, if it's like this, which means it is a variable that can be recursively expanded, every time war files is used, it will actually have to do a dollar wildcard.

It'll have to go to the file system, glob to find out the particular pattern, particular files that are on the file system.

And here I've used that actually twice. You can kind of see this. I've used it here, war files, and set it again as another recursive variable.

And then in here, this is now source files, right?

So when source files gets used, then again, it has to go to war files, it has to go to wildcard.

And if I was to refer to object underscore files more than once in this main file, dollar wildcard would actually get called multiple times, which can be very, very inefficient.

If you want to find out how often something is called, one thing you can do is you can stick in a little thing like this, and then you should be able to go here.

If I make, oh, if I could type, then it probably would be good, wouldn't it?

Let's go back here.

What have I done wrong? Yes. Okay, if I could type. Okay, so you see the X there printed out?

That means that war files was expanded. Its value was got once.

But if you refer to object files more than once, then you might end up expanding it multiple times.

Wildcard could be expensive, your main file could be slow.

So in general, if you can, you use colon equals. And what that does is it sets in stone the value of those variables at that point.

So war files will be globbed once, will be a string, modern files will etc.

The only restriction, of course, is you have to do this in the right order.

You can't do them in inverse order, otherwise the variables will not get set correctly.

All right, so this is looking a bit cleaner.

I've got source files, I've got object files. And I'm actually just going to remove the count files and just see that works.

Great, that does what we did before, similar kind of order.

All right, now, depending on how you like to do things, you might say, well, I kind of like this to be done.

I like cleanliness, I'm going to sort this. So actually, I'm going to go in here.

And I'm going to use another function. If you notice, the first function I really used in GNU Make here was $wildcard.

And this is a has a parameter and some functions will have multiple parameters after that with commas along them.

This is another one, $sort. And there's a whole load of functions you can use within a Makefile.

So let's just, let's just do that again. And now you can see the order came out slightly different.

You notice they did armitage first, then bet one, then bet two, then bet three.

The reason for that is that the object files, let's just use my little print thing.

The object files are now in that order. And Make will traverse here where object files is from left to right.

So it'll try and make them in that order.

So that's how we got a nice ordering. The ordering doesn't really matter.

I just find it a little bit, a little bit cleaner. Okay, great. So this has got a nice ordering, it's all nicely sorted out.

But if I go and have a look at my directory here, it's a bit of a mess.

I've got my object files in here, I've got my source files in here, and it'd be much cleaner if I didn't do it like this.

So I'm going to clean things up. So what I'm going to do is I'm going to make a source code directory, I'm going to make an object code directory, move things around, and hopefully we'll have a bit of a cleaner setup.

So first of all, let's get rid of those count files.

And then let's make ourselves a source code directory.

Let's make an object directory. Let's move all the poems into source. Okay, let's have a look.

Okay, there we go. Got my object. And if I just look in there, there's my poems.

Great. So now what do you think is going to happen if I type make?

We haven't got any dot count files. So the make file will try to make them.

But will it? So let's just have a go. Okay, it says nothing to be done. So why is that?

Well, let's go back and look at the make file. These variables here are looking for poems within the current directory where make is running.

And, well, it's not going to find them because they've been moved into source.

So we need to specify here the source directory.

So let's just go in and hack this in the worst possible way.

Okay, so it's going to do that.

That's going to say, we'll just prefix those with the directory.

And let's try and make it. Okay, now what did it do? Now what's interesting here is if you look carefully, it's gone and said, take this particular poem and make a count file in the same directory.

And the reason it did that is this percent thing here matched the whole of this string, which is source slash the thing.

So we can, again, we can use a little print thing. If we go in here and we say print the object files, you'll see they've all got source in front of them.

And we don't want to have source in front of it.

We really wanted to have, you know, we really wanted to have obj in front of it.

So we need to alter that. So there's a few different things we can do here to solve this problem.

One is we can change source to obj in the object files, and therefore it will be, it will, the count files will go into the obj directory.

The other one is we can change the rule so that the rule has object files go here and source files goes there.

And these are, these are different ways of solving it.

So I'm going to do both and then we'll, we'll see how that works out.

So remember, obj files has got source at the beginning of it.

So I don't want that. So I'm going to do, I'm going to use another, which is I'm going to say subst, and I'm going to say source slash obj slash.

Okay, let's just make sure that, yeah, all right, let's try and make this now.

Okay, now what happened?

Let's go and have a look again at obj files. Oops. Now what's happened here is it's, it's gone through, but it has not done what we wanted to do, which is it substituted source with object, but only at the beginning.

So there are different things we can use in GNU Make for doing substitutions in a, in a situation like this.

So let's just have a quick look and just go, um, let's get this right.

Let's substitute, get my things right around.

Let's just print. Okay, that's a bit better.

So I've got slightly, slight problems. Let me just fix it. I'll make, so this subst has gone, said every instance of source slash turned into obj slash.

Let's go back and just do that again. Oops. Okay. So now the object files are going in the right place.

And if I were to look at, um, source files, you would see the source files are coming from the source directory.

So that's great. So object files go here, source files go there.

And if we go in here and we look at it and say, this looks really great.

I've got those looks like it's nicely set up. Um, is there anything in my object directory?

No. So if I hit make it ought to make everything and I'm going to hit make and okay.

It doesn't work now. Why doesn't it work?

If you look at the, the, um, the error message is very, very clear. It says there's no rule to make obj slash armitage dash one dot count.

Okay. Why is that?

Well, let's go and have a look. If you remember when I had everything with source slash this percent and this percent were the same thing.

They were able to match and put everything in the source directory.

But now you've got ob slash something is made from source slash something.

So it's not working. So, okay. So let's go in here and let's go, you know, object files go here.

Source files go there.

We'll change that rule. Let's go back. Yeah. Let's go make. All right. Now this is much better.

It's now taken the source file. You can see here, for example, yeah.

And it's put the object file here. And actually, if we don't have a look in obj, there are those files with exactly what we wanted and source has not been contaminated.

So this is a fairly simple way of using directory names to do things.

And, but it's very, very explicit here how it works, but just think back. We had to be absolutely explicit in obj files and source files, which strings that matched exactly files in the file system.

There's no globbing going on here. There's no intelligence about what's happening.

All right. Now, one thing that bothers me about this make file is I've got a ton of repetition.

I've got source and obj all over the place.

So I'm going to go back to the beginning and I'm going to define some variables.

Notice I use colon equals here. It's actually unnecessary because this string doesn't refer to any other strings, any other variables, but I just have a habit, always using colon equals that way.

I don't get myself into trouble. So go like that.

I'm just going to go through and just replace that. So there you go. And let's replace ob slash with dollar obj.

Not that one, that one, that one. Okay.

So that now should work fine. Let's just remove everything in. All right. Now we do make.

There you go. Great. It still works. So wonderful. We have this working nicely.

Now it's starting to look like the sort of make files you might have had to deal with in the past.

And they can be kind of dense and difficult to understand. And using the print trick, you can actually get some more information about what the variables are and really figure out what they are.

And some of these variables get enormous.

If you have thousands of source files, things like source files or obj files could be gigantic strings, but it can be helpful to do it this way.

But still, the core of the original make file is here that's doing this stuff.

Okay. So this looks nice. I've now got some things. Now, there are other things you might want to do.

You might want to discover, you know, here's some hard coding, the names of the poets.

You might want to put those in variables, iterate through them.

We can certainly do that. We can really dig in here in different ways.

One of the things I'm going to do is I'm going to go in and I'm going to say, you know, I'm going to give myself a list of poets.

I'm going to say my poets are Owen, Armitage, and Bet for Betjeman.

And rather than having all this specific thing like this, I'm going to simplify things.

I'm going to say the source files are, right, and then you can do this for each.

And we're going to say p, because it's a poet, in poets, wildcard, in the source directory, no slash, because it's already got a slash, dollar p, star dot poem.

And just to be clear. Okay. I'm going to go and see if that has worked.

Aha.

So now that worked. Now let's go back and examine what it did. Source files is now exactly the same thing as it was before.

It's all of the poems that are in there.

So what did I do? All right. I used a for loop. Now you may not think of GNU Make as having a rich language in it, but in fact, it really does.

And actually, it's kind of a functional programming language in a way.

So we saw some simple functions like sort to sort a list.

Remember, a list is separated by spaces. And by the way, this is why GNU Make has a really hard time with file names that have spaces in them, because it doesn't know that they are any different from a list separator.

And so you get into all sorts of problems. So in here, again, there's another list here, space separated list.

And what foreach did was it set p to be one of these strings, Owen, then Armitage, then Betjeman.

And then it did this. It expanded that.

So it did wildcard in the source directory of the star.poem. And then the output of $Wildcard for each of those values of p just got concatenated with spaces in between it.

So you get another list. And then we did this thing to turn source into obj.

And then the entire program ran. And so it's worth looking into these functions within GNU Make because you can write really quite rich programs.

In fact, you can write things that are quite large and sometimes a little bit difficult to understand.

But you can dig in here and see what they are. This is the variable.

I tend to use lowercase here just to distinguish it. It's sort of a local variable.

It's only valid within this foreach. It can be easier to do it that way.

You can use a capital if you want to. You can give it a more complicated name.

You could get rid of the parenthesis. It's a single character variable.

You're allowed to do that. I tend to be a bit more pedantic. I just like it to be really, really clear and lay things out like that.

Okay. So this is fantastic.

This is now looking like a real Makefile. Now, what I'm going to do in the next episode of this, in episode three, is I'm going to add a linking step where I merge together the various files, which will make this a little bit more complicated.

But I think it's worth looking at some of the other kind of functions that you can use in a file.

We've seen a few, right? So we've seen foreach.

We've seen wildcard. We've seen substitute for substitution, sorting. But there's also things like multiple complex pattern substitution, things for actually manipulating a list directly.

So for example, if I go in here and I say, first, poet.

Actually, let's do this. Let's do count poets. And then we can do words.

What poets?

Okay, now just go back here and I'll say, what is that variable print count poets?

Three. There are three poets. So this function words actually pulled here.

They counted the number of items in the list. And that sometimes can be handy. There's also a way to, I think if I get it right, it's like this.

Let's call it something else.

Let's say second poet. Let's just go and print that guy out.

Oops, I got it the wrong way around then.

That's what happens when you don't have the manual in front of you and look up the order of.

I don't know. Is it because I typed in wrong? Classic.

It was right the first time. It's just I typed the thing wrong. Let's just do this.

And back again. Make print dash second poet. Okay. There you go. So that's actually, you know, you can get a specific part of a list from there.

So there's lots of stuff you can do with list manipulation.

And I took this to a real extreme with the thing of, you know, the GNU Make standard library, which was to add a load of functionality to GNU Make, like list manipulation, sets, associative arrays, arithmetic.

But you can do it all within the languages in there. You don't need to go out to the shell.

What you'll sometimes see in Makefiles is people going out to the shell to do things.

They'll suddenly go into the shell and do something that I, you know, they could actually have done within the Makefile itself.

If you do need to go out to the shell, then there is actually a function called shell, which allows you to do something.

So for example, if I just call this variable LS, just to be clear, and if I just go and print that, so there you go.

It gave you, that was the output of the LS command, which is what you get in this directory.

Turn into a list, and you could then go through that. You can actually do arbitrary things in the shell.

This can be quite expensive. You're spawning a shell, you're running commands there, and should be used, you know, sparingly if you have to use it at all.

But it is there, it is there as an option. There are also options for printing stuff out.

I showed you $info, which will print out some information.

There's also $warning, which if you want to say there's a problem, there's also $error.

So we could actually go in here and say, you know, something went horribly wrong.

I'm going to make sure I'm just going to stop at line eight and say there's a problem, and sometimes that's useful.

For example, in makefiles where you might have multiple architectures, like if you're targeting different processor architectures, sometimes there'll be a whole lot of if -defs, and I'll get into that, into the preprocessor, and you might say, you know, oh, I'm not on an architecture that I understand.

Stop the makefile with $error right here.

So those, that's useful, and there's the equivalent, the other one was $warning, and I'll show you how that works.

Something went slightly wrong. It's not quite so severe.

So you get a thing like that, but if you notice the make, nothing to be done for all here, means that it was actually going to carry on trying to do things, make things.

The last thing we haven't added, and this is so common in makefiles, is a target called clean, and I'm going to do it now.

Let's do it here, clean, and let's do a one -liner.

So sometimes you want to clean up the object files that are being made, and, you know, a classic way to do this is go in there and remove those files.

The other thing you could do here is you could probably do something like, you know, you remove everything that's in the object directory, since we've got everything nicely in the directory, you know, that could be our kind of choice how we do that, and that can be super handy, and then try to do make, and it makes everything again.

Let's just try touching one thing, just so that we see the makefile doesn't make everything every time.

There you go, I've already made that one thing.

If I touch that particular poem file, what you'd kind of expect, that poem file was newer than the count file, so it made it.

The other way is let's make an object file disappear, so let's just make that one disappear, and it should remake it.

Yeah, seems to work fine. Okay, so one of the things that's really bad about make clean, and this is almost a rule, I jokingly gave it a name, actually called it Usman's Law in my book about going to make, is that all make clean almost never does.

There's almost always something that's not being cleaned up, so if you're going to do a clean function, then I would put all your object files in the directory and blast the directory to make sure you get rid of everything, and even then, it can be tricky.

All right, that was 30 minutes whirlwind on cleaning up this makefile, me forgetting the order of parameters in going to make functions, but it gives you something that's, I think, now getting a little bit closer to a classic makefile for building some object files from source, and then next time, I'll move on to linking them together into an executable from these object files, and that will give us something that's closer to a real working makefile.

Thanks very much for watching.