Advanced Bash-Scripting Guide

[Pages:916]Advanced Bash-Scripting Guide

An in-depth exploration of the art of shell scripting

Mendel Cooper

10

10 Mar 2014

Revision History Revision 6.5 'TUNGSTENBERRY' release Revision 6.6 'YTTERBIUMBERRY' release Revision 10 'PUBLICDOMAIN' release

05 Apr 2012 27 Nov 2012 10 Mar 2014

Revised by: mc Revised by: mc Revised by: mc

This tutorial assumes no previous knowledge of scripting or programming, yet progresses rapidly toward an intermediate/advanced level of instruction . . . all the while sneaking in little nuggets of UNIX? wisdom and lore. It serves as a textbook, a manual for self-study, and as a reference and source of knowledge on shell scripting techniques. The exercises and heavily-commented examples invite active reader participation, under the premise that the only way to really learn scripting is to write scripts.

This book is suitable for classroom use as a general introduction to programming concepts.

This document is herewith granted to the Public Domain. No copyright!

Dedication

For Anita, the source of all the magic

Advanced Bash-Scripting Guide

Table of Contents

Chapter 1. Shell Programming!.........................................................................................................................1

Chapter 2. Starting Off With a Sha-Bang........................................................................................................3 2.1. Invoking the script............................................................................................................................6 2.2. Preliminary Exercises.......................................................................................................................6

Part 2. Basics.......................................................................................................................................................7

Chapter 3. Special Characters...........................................................................................................................8

Chapter 4. Introduction to Variables and Parameters..................................................................................30 4.1. Variable Substitution......................................................................................................................30 4.2. Variable Assignment.......................................................................................................................33 4.3. Bash Variables Are Untyped..........................................................................................................34 4.4. Special Variable Types...................................................................................................................35

Chapter 5. Quoting...........................................................................................................................................41 5.1. Quoting Variables...........................................................................................................................41 5.2. Escaping..........................................................................................................................................43

Chapter 6. Exit and Exit Status.......................................................................................................................51

Chapter 7. Tests................................................................................................................................................54 7.1. Test Constructs...............................................................................................................................54 7.2. File test operators............................................................................................................................62 7.3. Other Comparison Operators..........................................................................................................65 7.4. Nested if/then Condition Tests.......................................................................................................70 7.5. Testing Your Knowledge of Tests..................................................................................................71

Chapter 8. Operations and Related Topics....................................................................................................72 8.1. Operators.........................................................................................................................................72 8.2. Numerical Constants.......................................................................................................................78 8.3. The Double-Parentheses Construct.................................................................................................80 8.4. Operator Precedence.......................................................................................................................81

Part 3. Beyond the Basics.................................................................................................................................84

Chapter 9. Another Look at Variables...........................................................................................................85 9.1. Internal Variables............................................................................................................................85 9.2. Typing variables: declare or typeset.............................................................................................104 9.2.1. Another use for declare.......................................................................................................107 9.3. $RANDOM: generate random integer..........................................................................................107

Chapter 10. Manipulating Variables.............................................................................................................119 10.1. Manipulating Strings...................................................................................................................119 10.1.1. Manipulating strings using awk........................................................................................127 10.1.2. Further Reference..............................................................................................................127 10.2. Parameter Substitution................................................................................................................128

i

Advanced Bash-Scripting Guide

Table of Contents

Chapter 11. Loops and Branches..................................................................................................................138 11.1. Loops..........................................................................................................................................138 11.2. Nested Loops..............................................................................................................................152 11.3. Loop Control...............................................................................................................................153 11.4. Testing and Branching................................................................................................................156

Chapter 12. Command Substitution.............................................................................................................165

Chapter 13. Arithmetic Expansion................................................................................................................171

Chapter 14. Recess Time................................................................................................................................172

Part 4. Commands..........................................................................................................................................173

Chapter 15. Internal Commands and Builtins.............................................................................................181 15.1. Job Control Commands..............................................................................................................210

Chapter 16. External Filters, Programs and Commands...........................................................................215 16.1. Basic Commands........................................................................................................................215 16.2. Complex Commands...................................................................................................................221 16.3. Time / Date Commands..............................................................................................................231 16.4. Text Processing Commands........................................................................................................235 16.5. File and Archiving Commands...................................................................................................258 16.6. Communications Commands......................................................................................................276 16.7. Terminal Control Commands.....................................................................................................291 16.8. Math Commands.........................................................................................................................292 16.9. Miscellaneous Commands..........................................................................................................303

Chapter 17. System and Administrative Commands..................................................................................318 17.1. Analyzing a System Script..........................................................................................................349

Part 5. Advanced Topics.................................................................................................................................351

Chapter 18. Regular Expressions..................................................................................................................353 18.1. A Brief Introduction to Regular Expressions..............................................................................353 18.2. Globbing.....................................................................................................................................357

Chapter 19. Here Documents.........................................................................................................................359 19.1. Here Strings................................................................................................................................369

Chapter 20. I/O Redirection...........................................................................................................................373 20.1. Using exec...................................................................................................................................376 20.2. Redirecting Code Blocks............................................................................................................379 20.3. Applications................................................................................................................................384

Chapter 21. Subshells.....................................................................................................................................386

ii

Advanced Bash-Scripting Guide

Table of Contents

Chapter 22. Restricted Shells.........................................................................................................................391

Chapter 23. Process Substitution...................................................................................................................393

Chapter 24. Functions....................................................................................................................................398 24.1. Complex Functions and Function Complexities.........................................................................402 24.2. Local Variables...........................................................................................................................413 24.2.1. Local variables and recursion...........................................................................................414 24.3. Recursion Without Local Variables............................................................................................417

Chapter 25. Aliases.........................................................................................................................................420

Chapter 26. List Constructs...........................................................................................................................423

Chapter 27. Arrays.........................................................................................................................................427

Chapter 28. Indirect References....................................................................................................................456

Chapter 29. /dev and /proc.............................................................................................................................460 29.1. /dev..............................................................................................................................................460 29.2. /proc............................................................................................................................................463

Chapter 30. Network Programming.............................................................................................................469

Chapter 31. Of Zeros and Nulls.....................................................................................................................472

Chapter 32. Debugging...................................................................................................................................476

Chapter 33. Options........................................................................................................................................487

Chapter 34. Gotchas.......................................................................................................................................490

Chapter 35. Scripting With Style..................................................................................................................499 35.1. Unofficial Shell Scripting Stylesheet..........................................................................................499

Chapter 36. Miscellany...................................................................................................................................502 36.1. Interactive and non-interactive shells and scripts.......................................................................502 36.2. Shell Wrappers............................................................................................................................503 36.3. Tests and Comparisons: Alternatives..........................................................................................509 36.4. Recursion: a script calling itself..................................................................................................509 36.5. "Colorizing" Scripts....................................................................................................................512 36.6. Optimizations..............................................................................................................................525 36.7. Assorted Tips..............................................................................................................................528 36.7.1. Ideas for more powerful scripts.........................................................................................528 36.7.2. Widgets..............................................................................................................................539 36.8. Security Issues............................................................................................................................541 36.8.1. Infected Shell Scripts.........................................................................................................541 36.8.2. Hiding Shell Script Source................................................................................................541

iii

Advanced Bash-Scripting Guide

Table of Contents

Chapter 36. Miscellany 36.8.3. Writing Secure Shell Scripts.............................................................................................541

36.9. Portability Issues.........................................................................................................................541 36.9.1. A Test Suite.......................................................................................................................542

36.10. Shell Scripting Under Windows...............................................................................................543

Chapter 37. Bash, versions 2, 3, and 4..........................................................................................................544 37.1. Bash, version 2............................................................................................................................544 37.2. Bash, version 3............................................................................................................................548 37.2.1. Bash, version 3.1...............................................................................................................551 37.2.2. Bash, version 3.2...............................................................................................................552 37.3. Bash, version 4............................................................................................................................552 37.3.1. Bash, version 4.1...............................................................................................................559 37.3.2. Bash, version 4.2...............................................................................................................560

Chapter 38. Endnotes.....................................................................................................................................564 38.1. Author's Note..............................................................................................................................564 38.2. About the Author........................................................................................................................564 38.3. Where to Go For Help.................................................................................................................565 38.4. Tools Used to Produce This Book..............................................................................................565 38.4.1. Hardware...........................................................................................................................565 38.4.2. Software and Printware.....................................................................................................565 38.5. Credits.........................................................................................................................................566 38.6. Disclaimer...................................................................................................................................567

Bibliography....................................................................................................................................................569

Appendix A. Contributed Scripts..................................................................................................................577

Appendix B. Reference Cards........................................................................................................................787

Appendix C. A Sed and Awk Micro-Primer................................................................................................792 C.1. Sed................................................................................................................................................792 C.2. Awk..............................................................................................................................................795

Appendix D. Parsing and Managing Pathnames.........................................................................................798

Appendix E. Exit Codes With Special Meanings.........................................................................................802

Appendix F. A Detailed Introduction to I/O and I/O Redirection.............................................................803

Appendix G. Command-Line Options..........................................................................................................805 G.1. Standard Command-Line Options...............................................................................................805 G.2. Bash Command-Line Options......................................................................................................806

Appendix H. Important Files.........................................................................................................................808

iv

Advanced Bash-Scripting Guide

Table of Contents

Appendix I. Important System Directories..................................................................................................809 Appendix J. An Introduction to Programmable Completion.....................................................................811 Appendix K. Localization...............................................................................................................................814 Appendix L. History Commands...................................................................................................................818 Appendix M. Sample .bashrc and .bash_profile Files.................................................................................820 Appendix N. Converting DOS Batch Files to Shell Scripts.........................................................................837 Appendix O. Exercises....................................................................................................................................841

O.1. Analyzing Scripts.........................................................................................................................841 O.2. Writing Scripts.............................................................................................................................843 Appendix P. Revision History........................................................................................................................853 Appendix Q. Download and Mirror Sites.....................................................................................................856 Appendix R. To Do List..................................................................................................................................857 Appendix S. Copyright...................................................................................................................................858 Appendix T. ASCII Table..............................................................................................................................860 Index....................................................................................................................................................862

Notes..............................................................................................................................................899

v

Chapter 1. Shell Programming!

No programming language is perfect. There is not even a single best language; there are only languages well suited or perhaps poorly suited for particular purposes.

--Herbert Mayer A working knowledge of shell scripting is essential to anyone wishing to become reasonably proficient at system administration, even if they do not anticipate ever having to actually write a script. Consider that as a Linux machine boots up, it executes the shell scripts in /etc/rc.d to restore the system configuration and set up services. A detailed understanding of these startup scripts is important for analyzing the behavior of a system, and possibly modifying it.

The craft of scripting is not hard to master, since scripts can be built in bite-sized sections and there is only a fairly small set of shell-specific operators and options [1] to learn. The syntax is simple -- even austere -similar to that of invoking and chaining together utilities at the command line, and there are only a few "rules" governing their use. Most short scripts work right the first time, and debugging even the longer ones is straightforward.

In the early days of personal computing, the BASIC language enabled anyone reasonably computer proficient to write programs on an early generation of microcomputers. Decades later, the Bash scripting language enables anyone with a rudimentary knowledge of Linux or UNIX to do the same on modern machines.

We now have miniaturized single-board computers with amazing capabilities, such as the Raspberry Pi. Bash scripting provides a way to explore the capabilities of these fascinating devices.

A shell script is a quick-and-dirty method of prototyping a complex application. Getting even a limited subset of the functionality to work in a script is often a useful first stage in project development. In this way, the structure of the application can be tested and tinkered with, and the major pitfalls found before proceeding to the final coding in C, C++, Java, Perl, or Python.

Shell scripting hearkens back to the classic UNIX philosophy of breaking complex projects into simpler subtasks, of chaining together components and utilities. Many consider this a better, or at least more esthetically pleasing approach to problem solving than using one of the new generation of high-powered all-in-one languages, such as Perl, which attempt to be all things to all people, but at the cost of forcing you to alter your thinking processes to fit the tool.

According to Herbert Mayer, "a useful language needs arrays, pointers, and a generic mechanism for building data structures." By these criteria, shell scripting falls somewhat short of being "useful." Or, perhaps not. . . .

When not to use shell scripts

Chapter 1. Shell Programming!

1

Advanced Bash-Scripting Guide

? Resource-intensive tasks, especially where speed is a factor (sorting, hashing, recursion [2] ...) ? Procedures involving heavy-duty math operations, especially floating point arithmetic, arbitrary

precision calculations, or complex numbers (use C++ or FORTRAN instead) ? Cross-platform portability required (use C or Java instead) ? Complex applications, where structured programming is a necessity (type-checking of variables,

function prototypes, etc.) ? Mission-critical applications upon which you are betting the future of the company ? Situations where security is important, where you need to guarantee the integrity of your system and

protect against intrusion, cracking, and vandalism ? Project consists of subcomponents with interlocking dependencies ? Extensive file operations required (Bash is limited to serial file access, and that only in a

particularly clumsy and inefficient line-by-line fashion.) ? Need native support for multi-dimensional arrays ? Need data structures, such as linked lists or trees ? Need to generate / manipulate graphics or GUIs ? Need direct access to system hardware or external peripherals ? Need port or socket I/O ? Need to use libraries or interface with legacy code ? Proprietary, closed-source applications (Shell scripts put the source code right out in the open for all

the world to see.)

If any of the above applies, consider a more powerful scripting language -- perhaps Perl, Tcl, Python, Ruby -- or possibly a compiled language such as C, C++, or Java. Even then, prototyping the application as a shell script might still be a useful development step.

We will be using Bash, an acronym [3] for "Bourne-Again shell" and a pun on Stephen Bourne's now classic Bourne shell. Bash has become a de facto standard for shell scripting on most flavors of UNIX. Most of the principles this book covers apply equally well to scripting with other shells, such as the Korn Shell, from which Bash derives some of its features, [4] and the C Shell and its variants. (Note that C Shell programming is not recommended due to certain inherent problems, as pointed out in an October, 1993 Usenet post by Tom Christiansen.)

What follows is a tutorial on shell scripting. It relies heavily on examples to illustrate various features of the shell. The example scripts work -- they've been tested, insofar as possible -- and some of them are even useful in real life. The reader can play with the actual working code of the examples in the source archive (scriptname.sh or scriptname.bash), [5] give them execute permission (chmod u+rx scriptname), then run them to see what happens. Should the source archive not be available, then cut-and-paste from the HTML or pdf rendered versions. Be aware that some of the scripts presented here introduce features before they are explained, and this may require the reader to temporarily skip ahead for enlightenment.

Unless otherwise noted, the author of this book wrote the example scripts that follow.

His countenance was bold and bashed not.

--Edmund Spenser

Chapter 1. Shell Programming!

2

 ................
................

In order to avoid copyright disputes, this page is only a partial summary.

Google Online Preview   Download