1 ITEC400 Make Utility and UNIX Files (Part 2) George Vaughan Franklin University

2 Topics What is Make? What is GNU Make? The ‘makefile’ Make Variables More Makefile features Implicit makefile Rules View Pathing

3 What is Make? ‘Make’ is a utility that is used to assemble things. Such items include programs, documents or any other product that may require multiple steps. Make is especially useful in constructing a component that requires a multi-step construction process. A type of Inference Engine

4 What Is Make? Make can be made aware of dependencies. –If A is dependent on B and B is dependent on C, then: Make will not attempt to build A until B is built Make will not attempt to build B until C is built Make is aware of modification dates: –If B is somehow modified, then: only A and B will be built, C will not be built (A is dependent on B, but C is not dependent on either A or B) Make will only build what is necessary and nothing more.

5 What is Make? Make, by itself, does not know how to build anything. Make is a rules based system (similar to an expert system). Make knows how and what to build based on: –the dependencies we specify –the rules we specify –modification times of build components.

6 What is Make? The rules needed to build ‘something’ are specified in a file called a ‘makefile’. In the case of building a program, rules are established in the ‘makefile’ that specify such things as: –how to construct object files from source and header files. –how to construct an executable from object files.

7 What is GNU Make? There are several make utilities available including those that are part of standard Unix distributions and from 3 rd party sources (see d_Management/Make_Tools/) d_Management/Make_Tools/ This lecture is focused on GNU Make GNU Make comes from the Free Software Foundation and is part of Red Hat. It is also the default make utility on einstein. Type ‘make -v’ to determine what version of make you have. GNU Make was implemented by Richard Stallman and Roland McGrath. This lecture will use a simple C++ program as an example of using Make.

8 Example ex0510.dir In our first example, we will look at a simple C++ program composed of the following files: –ClassA.h - contains class declaration –ClassA.cpp - contains class implementation –main.cpp - contains implementation of main(). The C++ program really doesn’t do very much except print the name of the C++ class (ClassA), but it is useful for this makefile illustration We will look at: –The source code. –Building this program without the use of a make file. –Building this program using a make file.

9 Example ex0510.dir Contents of classA.h: 0001 #include class ClassA { 0004 public: 0005 void myClass(); 0006 }; Contents of ClassA.cpp: 0001 #include "ClassA.h" void ClassA::myClass() { 0004 printf("%s\n", "ClassA"); 0005 }

10 Example ex0510.dir Contents of main.cpp 0001 #include "ClassA.h" int main() { 0004 ClassA objectA; objectA.myClass(); 0007 return 0; 0008 }

11 Example ex0510.dir Notice that the this example is not a single file – there are 3 source files that make up this program. It is common for large projects to consist of many source files (in the thousands) Some source files depend on other source files – for example ClassA.cpp includes ClassA.h. This means that the object file, ClassA.o not only depdends on ClassA.cpp, but also ClassA.h Object files (files whose names end in ‘.o’) are not source files - they are the products of compilation. Therefore we need to build based on dependencies. The next slide shows a dependency graph for this example. Note that graph includes both source files and object files.

12 Example ex0510.dir myProgram (executable) ClassA.o (object file)main.o (object file) classA.h (source)main.cpp (source)classA.cpp (source) Note: Italicized file names are build products (i.e. created by running make)

13 Example ex0510.dir The next slide shows the commands we need to build the application manually (without using a makefile). In this example, we are using “g++” which is the “C and C++” complier from GNU. This compiler is available on einstein and on Redhat and Knoppix (your itec400 CD). We first use the “-c” option on g++ to compile the source files (create the object files) We then use the “-o” option to create the executable program from the object files. Please Note: The contents on the next slide is output captured from the build process. The next slide is not the contents of some source file

14 Example ex0510.dir 0001: $ g++ -c ClassA.cpp 0002: 0003: $ g++ -c main.cpp 0004: 0005: $ ls 0006: ClassA.cpp ClassA.h ClassA.o main.cpp main.o 0007: 0008: $ g++ -o myProgram ClassA.o main.o 0009: 0010: ls 0011: ClassA.cpp ClassA.h ClassA.o main.cpp main.o myProgram 0012: 0013:./myProgram 0014: ClassA Line 1: First, compile ClassA.cpp Line 3: Second, compile main.cpp Line 5: directory now contains object files (files that end in “.o”)… Line 8: link object files into an executable whose name is “myProgram” line 10: directory now contains executable which is named “myProgram” Line 13: execute “myProgram”. Line 14: output of program.

15 makefile We will now create a simple makefile to build “myProgram” for us. The difference between using a makefile versus a shell script is that the makefile will only rebuild what is necessary. The makefile contains a list of rules which describe: –targets –dependencies –actions

16 A Simple makefile The makefile rule has the following format: target : prerequisites (dependencies)… command_1 command_2 –where: target is typically the name of the file to be generated. A target may also be an action to be performed prerequisite(s) are file(s) that that this target depends on. command_n are the SHELL command(s) necessary to produce the target (one command per line). Each command line MUST be prefixed with a TAB. –If you use spaces to indent instead of TABs, your make file will not work. The commands MUST, in some way, alter the time-stamp of the target file. –This because make uses the timestamp to determine if the file is up to date. –If the timestamp were not updated, the file always be built (even if the file is up to date – which is a waste of time)

17 A Simple makefile Note that the rules in the makefile are not executed in sequential order. They are executed based on dependencies. The next slides illustrate how to use a makefile to build the program that was seen in example 0510.dir

18 Example 0520.dir 0001 myProgram : main.o ClassA.o 0002 g++ -o myProgram main.o \ 0003 ClassA.o main.o : main.cpp ClassA.h 0006 g++ -c main.cpp ClassA.o : ClassA.cpp ClassA.h 0009 g++ -c ClassA.cpp clean : 0012 rm myProgram *.o NOTE: It is easier to understand this make file by referring to the dependency graph on the next page. The graph is the same graph we saw earlier This make file is stored in a file called “makefile” Line 1: executable ‘myProgram’ is dependent on object files. Line 2-3: executable is constructed by linking object files. Line 3 is a continuation of line 2 (notice the continuation character “\”) Line 5: main.o is dependent on main.cpp and ClassA.h Line 8: ClassA.o is dependent on ClassA.cpp and ClassA.h Line 11: ‘clean’ is an action. ‘clean’ is used to remove generated files so we can build from scratch.

19 Dependency Graph (repeated) myProgram (executable) ClassA.o (object file)main.o (object file) classA.h (source)main.cpp (source)classA.cpp (source) Note: Italicized file names are build products (i.e. created by running make)

20 Make Exercise 0001 myProgram : main.o ClassA.o 0002 g++ -o myProgram main.o \ 0003 ClassA.o main.o : main.cpp ClassA.h 0006 g++ -c main.cpp ClassA.o : ClassA.cpp ClassA.h 0009 g++ -c ClassA.cpp clean : 0012 rm myProgram *.o Try answering the questions below using the makefile from Example ex0520.dir. It might also help to refer to the dependency graph. What needs to be built if: 1.A second ‘make’ is run immediately after a first ‘make’? 2.ClassA.cpp is changed? 3.ClassA.h is changed? 4.main.cpp is changed? 5.if clean is specified? * The answers appear on the next few slides.

21 Answers To Make Exercise 1.What needs to be built if a second ‘make’ is run immediately after a first ‘make’? Nothing happens – Make does not make anything. >make make: `myProgram' is up to date. 2.What needs to be built if ClassA.cpp is changed? >make g++ -c ClassA.cpp g++ -o myProgram main.o \ ClassA.o

22 Answers To Make Exercise 3.What needs to be built if ClassA.h is changed? >make g++ -c main.cpp g++ -c ClassA.cpp g++ -o myProgram main.o \ ClassA.o

23 Answers To Make Exercise 5.What needs to be built if main.cpp is changed? >make g++ -c main.cpp g++ -o myProgram main.o \ ClassA.o 6.What needs to be built if if clean is specified? Before ‘clean’ >ls ClassA.cpp ClassA.o main.cpp makefile ClassA.h main.o myProgram After ‘clean’ >make clean rm myProgram *.o >ls ClassA.cpp ClassA.h main.cpp makefile

24 More on makefiles By default, make attempts to build the first target defined in a makefile. The first target is known as the default goal. The default goal is the target that is built if no explicit targets are given to make on the command line. A target is rebuilt if any of its prerequisites have a newer timestamp than the target itself. Thus, make performs a recursive decent through the target tree building the lowest level targets first.

25 Example ex0530.dir Assume that our project gets more complicated - assume we introduce classes ClassB and ClassC. The source code for this example (like all examples) is located in the examples directory and will not be reproduced in these lecture notes.

26 Example ex0530.dir 0001 myProgram : main.o ClassA.o \ 0002 ClassB.o ClassC.o 0003 g++ -o myProgram main.o \ 0004 ClassA.o ClassB.o \ 0005 ClassC.o 0006 main.o : main.cpp ClassA.h ClassB.h \ 0007 ClassC.h 0008 g++ -c main.cpp 0009 ClassA.o : ClassA.cpp ClassA.h 0010 g++ -c ClassA.cpp 0011 ClassB.o : ClassB.cpp ClassB.h 0012 g++ -c ClassB.cpp 0013 ClassC.o : ClassC.cpp ClassC.h 0014 g++ -c ClassC.cpp 0015 clean : 0016 rm myProgram *.o The name of the file is makefile Lines 1-5: Things are getting complicated… if we forget an object file, especially in the prerequisites, we can have a hard time debugging. There has to be a better way…

27 makefile Variables There are some things we can do to simplify the makefile on the previous slide. One thing we can do is to take advantage of makefile variables. In this case, we will use a makefile variable to store the list of object files. We will call the variable OBJECTS By default, make uses the Bourne shell for the command list for each rule. We can specify our own shell preference with the special make variable, “SHELL” For example, if wish to use Korn Shell we would include the following line in our makefile: SHELL=/bin/ksh

28 Example ex0540.dir 0001 SHELL=/bin/ksh 0002 OBJECTS = main.o ClassA.o \ 0003 ClassB.o ClassC.o 0004 myProgram : $(OBJECTS) 0005 g++ -o myProgram $(OBJECTS) 0006 main.o : main.cpp ClassA.h \ 0007 ClassB.h ClassC.h 0008 g++ -c main.cpp 0009 ClassA.o : ClassA.cpp ClassA.h 0010 g++ -c ClassA.cpp 0011 ClassB.o : ClassB.cpp ClassB.h 0012 g++ -c ClassB.cpp 0013 ClassC.o : ClassC.cpp ClassC.h 0014 g++ -c ClassC.cpp 0015 clean : 0016 rm myProgram $(OBJECTS) Line 1: The special make variable ‘SHELL’ is used to tell make to use Korn Shell Line 2: The variable ‘OBJECTS’ is defined. Also “\” is a continuation character. Line 4,5 and 16: The variable ‘OBJECTS’ is used. Notice that when a makefile variable is used, it is enclosed by parentheses and prefixed by $.

29 Implicit makefile Rules As mentioned previously, make may be used to build anything, as long as actions can be defined to produce the target. However, since make is used so often to build software products, it can use ‘implicit’ rules for building common target types. For example, we do not need to explicitly state that ‘file.o’ comes from or depends on ‘file.cpp’ In other words, in this case, we can drop file.cpp from the prerequisites list and further, we do not need to explicitly define the action - make understands what we want.

30 Example ex0550.dir 0001 OBJECTS = main.o ClassA.o \ 0002 ClassB.o ClassC.o myProgram : $(OBJECTS) 0005 g++ -o myProgram $(OBJECTS) main.o : ClassA.h ClassB.h \ 0008 ClassC.h ClassA.o : ClassA.h ClassB.o : ClassB.h ClassC.o : ClassC.h clean : 0017 rm myProgram $(OBJECTS) Lines 7, 10, 12, 14: we have: –removed reference for *.cpp. –removed action

31 More makefile Features Makefiles contain one or more of the following elements: –explicit rules –implicit rules –variable definitions –directives –comments Explicit Rules describe how to make a target based on prerequisites and commands.

32 More makefile Features Implicit Rules describe how to make a class of files based on the filename (meta-rule). –Implicit Rules may also contain prerequisites and commands. Directives describe special actions to be taken while the makefile is being processed. –For example, the ‘include’ directive may be used to include the contents of one make file into another. Makefiles can have comments: –The syntax is the same as Bourne shell –A comment starts with a ‘#’ and continues to the end of line.

33 Example ex0560.dir The next example illustrates the use of implicit rules and directives. In our previous example, we saw that we add a new rule for each new class. But in this example, all class rules have the same format: ClassA.o : ClassA.h If in our project, we can assume that all targets for future classes will have the same format, we can write an implicit rule to support all classes.

34 Example ex0560.dir We can create an implicit rule for our needs. It might look like this: Class%.o : Class%.cpp Class%.h g++ -c $< where: % is used for pattern matching (e.g. A, B, and C in this case) $< is an automatic variable used to represent the name of the first perquisite (in this case, the source file).

35 Example ex0560.dir Furthermore, we decide that multiple makefiles need to make use of this implicit rule (and perhaps other implicit rules). We can address this need by putting this implicit rule in its own makefile, which we will call ‘rules.mk’ (it could be called anything).

36 Example ex0560.dir The rules.mk file looks like this: 0001: Class%.o : Class%.cpp Class%.h 0002: g++ -c $< The makefile looks like this: 0001 include rules.mk OBJECTS = main.o ClassA.o \ 0004 ClassB.o ClassC.o 0005 myProgram : $(OBJECTS) 0006 g++ -o myProgram $(OBJECTS) 0007 main.o : ClassA.h ClassB.h \ 0008 ClassC.h 0009 clean : 0010 rm myProgram $(OBJECTS) Line 1: include ‘rules.mk’ file. Notice that we no longer have explicit rules describing how to build the object file for each class.

37 View Pathing GNU make supports the concept of view pathing. It is based on the make variable, VPATH. VPATH contains a list of (parallel) directories for make to search for prerequisites. Each developer only needs to have the actual files he/she is working on in the local directory structure. All other files can exist in a parallel directory structure.

38 View Pathing Make will effectively merge the directories in the view path presenting a single view to the build tools. When the developer is satisfied with the new changes, he/she ‘submits’ these changes so that they become part of the official view. Much better than creating a snapshot of all files (with the risk that the snapshot becomes stale). View path always the latest and greatest official version. View pathing can also be used to separate source form product.

39 Ant Ant is a relatively new build tool and is an alternative to make. Ant was developed by the Apache Jakarta project. Ant is Java based, making it platform independent.

40 Ant Make is extended using shell based commands. Ant is extended using Java classes. Build (make) files in Ant are written in XML.

41 More on Unix Files The next portion of this lecture continues the investigation into Unix Files

42 Links Links can be viewed as entries in a directory that reference other files. In Unix we can create 2 types of links: –physical (hard) links –symbolic (soft) links

43 Physical (Hard) Links A physical link references the exact same file (inode) as the file entry itself. An inode is a unique number assigned to a file by the file system (we’ll see more about inodes, later in these slides). In fact, a file entry in the directory can be viewed as a physical link and is no different than any other physical link.

44 Symbolic (Soft) Links A Symbolic Link references a “pointer file” which has its own inode. The pointer file points back to the directory entry that references the target file (inode) Therefore the inode for the symbolic link and the target file are different. If the target file is deleted, then the symbolic link points to nothing…

45 Example ln 0001 /export/home/vaughang/trash>ln file1 phys_link 0002 /export/home/vaughang/trash>ls -li 0003 total rw-r--r Feb 4 20:18 file rw-r--r Feb 4 20:18 phys_link Line 1: create physical link to file 1 Line 2: ls –li (long listing with inodes), notice that both file1 and phys_link reference the same inode.

46 File Systems From The OS Perspective Filesystems and disks Directories inodes

47 File Systems Technically, a file system is a formatted disk partition containing a fixed number of blocks. The UNIX Directory Structure starts with root and may be composed one or more file systems. In a later class, we will learn: –how to create a file system –how to mount and unmount a file system from the Unix Directory Structure

48 System V File System Layout A file system is composed of many, many disk blocks This is ‘old’ System V format Many systems today use more sophisticated layout (but more complex - we will see it later) Block 0 is the boot block Block 1 is the super block. It contains information about the file system such as file system size, block size, etc. Blocks 2 – lib: blocks containing inodes Blocks fdb - ldb: data blocks BlockContents 0 Boot Block 1 Super Block 2 Start of inode blocks lib last inode block fdb first data block ldb last data block

49 Directories Directories are structures that map the user defined file name to an inode. A file may be associated with one or more directory entries. A file is associated with exactly one inode. This is why a directory entry referencing a file can be viewed as yet another physical link to that file. To put it another way, a directory is a list of physical links.

50 inodes Every file is associated with 1 inode. The inode keeps of the following file specific information: –file mode –count of hard links –owner id –group id –time of last file access –time of last file modification –file size –file addresses You can see the inode of a file by typing: ls -i

51 inodes (simple view) Simple view of inodes Directory maps file names to inodes. Each file has 1 inode Each file may have more than 1 directory entry. An inode contains a list of disk block addresses. Dir Entry 1 myfile Dir Entry 2 yourfile inode 203 disk addresses owner, etc inode 379 disk addresses owner, etc Data Block 34 Data Block 379 Data Block 1023

52 inodes (hard link) With hard link, 2 or more directory entries point to same inode (i.e. same file). inode keeps track of number of hardlinks pointing to it. Deleting a file deletes entry from directory. If link count is 1 at tome of delete, the inode is also deleted. Dir Entry 1 myfile Dir Entry 2 yourfile inode 379 disk addresses owner, etc Data Block 34 Data Block 379 Data Block 1023

53 inodes (soft or symbolic link) inode 379 does not know or care about symbolic links pointing to it. The symbolic link is not updated if “yourfile” is deleted. Dir Entry 1 myfile Dir Entry 2 yourfile inode 203 pointer file pointing to yourfile inode 379 disk addresses owner, etc Data Block 34 Data Block 379 Data Block 1023

54 inode structure An inode can only hold a fixed number of direct data block addresses (10 for Linux). For really big files, indirect block address may be used For really, really big files secondary indirect block addresses may be used For really, really, really big files tertiary indirect block addresses may be used.

55 inode structure Fields in an inode: –File/Dir = 1/0 (File = 1, Dir = 0). –Owner = login of user when file was created. –Access = owner/group/world (read/write/execute). –Size = file/directory size in bytes. –Links = Number of hard links to inode. –p0 – p9 = direct block pointer. –I1 = block pointer with one level of indirection. –I2 = block pointer with two levels of indirection. –I3 = block pointer with three levels of indirection. A block can contain 256 pointers. max file block size = * *256*256 (Note: information on this slide comes from:

56 File Systems From The Sys Admin Perspective Creating, Mounting and Unmounting file systems will covered in a future class dedicated to disks and devices. chown command –allows a user (usually root) to change the owner of a file. –root can change the owner of all files. –On some systems, the file owner an change the owner of a file (but can’t change it back).

57 Disk Space and inodes A system Administrator needs to be concerned with disk space and inode usage. It is possible to have a file system with plenty of space but no more inodes. Often this requires recreating the file system

58 Disk Space and inodes You can use the df command to check both disk space and inode utilization. Example of checking disk space utilization on Linux: df -h Filesystem Size Used Avail Use% Mounted on /dev/hda2 12G 3.4G 8.0G 30% / /dev/hdb2 4.8G 244M 4.3G 6% /home /dev/hdb1 19G 15G 4.3G 77% /mnt/msdos/d

59 Disk Space and inodes Example of checking inode utilization on Linux: df -i Filesystem Inodes IUsed IFree IUse% Mounted on /dev/hda % / /dev/hdb % /home /dev/hdb /mnt/msdos/d

60 References “GNU Make, A Program for Directed Compilation”, by Richard Stallman and Roland McGrath, ro_dev/make.htmlhttp:// ro_dev/make.html pt de.htm