This file documents GNU Sed.

Sed is a traditional and standard Unix Stream editor.  The function of sed is to read one line at a time from one or more input streams and to write an edited version of these lines to an output stream.  One argument to sed, either on the command line or in a file, is an editing script that defines the rules for how each line should be edited.

Many programs fit the description of sed or are generalizations of it (such as Awk or even Emacs).  The distinguishing characteristics of sed are that it is small and simple, standard and ubiquitous, yet fast and powerful enough to be used in many situations.  One notable use of sed is in the configuration scripts and Makefiles for GNU software.

Some other distinguishing characteristics of sed are its arcane scripting language, its irregular and somewhat arbitrary virtual machine, and its quirky feature set.  A subset of sed is clean and simple, almost natural and regular -- but I would exclude from this subset features such as the hold buffer and the t flag (see The Sed Machine).  When hacking sed, one gets the feeling that it started out as just the subset and then a few convenience features were added here and there to provide ad hoc solutions to increasing demands on sed-scripts.

This manual somewhat reflects that (made up) history of sed.  Sed programming is introduced in two stages -- first, the basic commands you are likely to actually use, and second, the full command set in all of its arcane glory.

This documentation was written for Sed 3.1 but most of it applies to any version of sed.

GNU Sed was first authored by Jay Fenlason (hack@gnu.ai.mit.edu) and subsequently maintained and tweaked by Tom Lord (lord@gnu.ai.mit.edu).

Copying
*******

Copyright (C) 1996
Tom Lord

Permission to use, copy, modify, distribute, and sell this documentation for any purpose is hereby granted without fee, provided that the above copyright notice appear in all copies and that both that copyright notice and this permission are preserved.

NO WARRANTY

BECAUSE THE PROGRAM IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR OR CORRECTION.

IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Overview
********

This chapter introduces sed by way of an example of its use.

Consider a typical unix password file:

     root:xXlv/n/peesXx:0:0:root:/root:/bin/bash
     us:sDSFLKJdfj:501:100:Uncle Sam:/home/us:/bin/bash
     bb:0gbezeIH7eFcA:503:100:Big Brother:/home/bb:/bin/bash
     att:NbBsVnasfK/oU:505:100:Ma Bell:/home/att:/bin/bash

Suppose we want a list of user ids.  We can make such a list by editing the password file and replacing the first colon and everything following on each line with "", the empty string.

     root
     us
     bb
     att

Editing the file by hand isn't always convenient or even an option. For example, if a shell script must convert the password file into a list of user ids, it probably won't run Emacs or VI to do so.  That's where Sed comes in.

Sed is used by providing an editing script that tells how each line should be edited.   By default, sed reads from standard input and writes to standard output.

Sed scripts are written in a language similar to the command language of ed, the standard and traditional Unix editor.  In this case, a very simple script will do the job:

     s/:.*//

The script is a substitute command that replaces everything matching the regexp :.* with the empty string.  Sed will apply this to every line of the password file if we run:

     % sed -e "s/:.*//" < /etc/passwd
     root
     us
     bb
     att

(Note that the substitute command is quoted so that the shell doesn't interpret the "*" character as a wild-card in a file-name.)

Using various command line options, sed can be directed to read input from one or more files instead of its standard input.  Sed scripts can be considerably more complicated than the example shown here and sed can be directed to read its script from a file.

Most of the remainder of this manual, then, is concerned with explaining the various command line options to sed and documenting the sed scripting language.

Invoking Sed
************

This chapter describes the arguments that sed accepts.

Sed has built-in help:

     % sed --help
     Usage: ./sed [-V] [--version]
               [-h] [--help]
               [-n] [--silent] [--quiet]
               [-e script]  [--expression=script]
               [-f script-file] [--file=script-file]
               [file...]

Here is what the options do:

  * [-V] [--version]

     Print the current version number and exit immediately.

  * [-h] [--help]

     Print a usage message for sed.

  * [-e script]  [--expression=script]

     Append the string SCRIPT to the editing script.

  * [-f script-file] [--file=script-file]

     Append the contents of script-file to the editing script.

  * [-n] [--silent] [--quiet]

     Suppress output except in response to explicit output commands in the editing script.  This cancels the usual behavior which is to print each line after it has been edited.

  * [file...]

     Read lines to be edited from the list of files given. If no files are specified, input is taken from standard input.

The -e and -f options can be mixed and repeated any number of times to build a complete editing script.  At least one or the other of these options must be provided.

The Sed Machine
***************

This chapter introduces the concepts that are necessary to understand the sed scripting language.

It is natural to think of sed as a very specialized virtual machine.  This machine loads a program from the -e and -f arguments provided to sed.  Other arguments set-up the input sources for the machine.

The virtual machine operates in a cycle.  At the beginning of each cycle, a new line of input is read into the line buffer. The editing script is then interpreted from the beginning.   When the script has completed (and sometimes in response to commands within the script), the line buffer is written to standard output.  Then, a new cycle begins.

If the -n or --quiet option is provided, then the last step of the cycle, writing line buffer to standard output, is omitted.

The core of the machine is therefore an input stream, a line buffer, an output stream, and an editing script -- but the sed machine has a few additional parts as well.

The hold buffer is, like the line buffer, storage for characters. The sed programming language includes commands to move characters between the line and hold buffers.  Scripts can use the hold buffer as temporary storage for a string.

The append buffer is yet a third text buffer.  During the execution of a script, text may be added to the append buffer.  At the end of the cycle, after the line buffer is printed, the append buffer is printed and then emptied.  The append buffer is printed even if the -n or --quiet flag is specified.

One of the scripting language commands, the s command, performs substitutions on the line buffer.   That is, it looks for matches of a particular regexp within the line buffer, and replaces any it finds with a particular string (the regexp and the string are arguments to the s command).

When the s command finds a match to replace, it sets the t flag.  The t flag is a boolean variable that is checked by the t command, a conditional branch.

To summarize, the sed virtual machine has these parts:

  * line buffer

     into which lines of input are read and out of which lines of output are usually written.

  * hold buffer

     which is used as temporary storage for text.

  * append buffer

     which is used as temporary storage for pending output.

  * t flag

     which is set by the s command and tested by the t command.

  * editing script

     which is applied to each line of input.

Sed operates as a cyclic process that reads a line into the line buffer, evaluates the editing script, and unless the -n or --quiet option was supplied, writes output from the line buffer.

The cycle continues until no more input is available.

The Sed Programming Language
****************************

A sed program consists of a sequence of commands.  Each command is terminated by either a newline or a semi-colon.  A string provided to the -e command is appended to the script and a newline is appended after that.  Thus, the end of a string passed to -e is always the end of a command, regardless of whether or not the string explicitly ends with a newline or semicolon.  Similarly, the end of a script file is also treated like an implicit newline -- it always terminates the preceding command.

A script can include comments which begin with # and continue to the next newline.

The general form of a scripting command is this:

     <address> <command> <command arguments>

The <address> part specifies which input lines this command applies to.

The <command> part specifies what kind of operation the sed machine should perform at the addressed lines.

The <command arguments> supply any additional parameters needed to complete the command.

The scripting language is best understood in two parts.  First, the syntax and meaning of addresses will be explained.  Then, the commands and the arguments they expect will be described.

Simple Addresses
================

When execution of a script reaches a particular command, the address of that command determines whether or not the command is evaluated.

the null address
----------------

The simplest form of address is the null address, written as the
empty string (i.e., no address is provided for the command).   The null
address addresses every line.   A command with the null address always
applies, whenever execution reaches that command.

For example, this script applies a dramatic change of year to the first occurrence of "1996" on every line:

     s/1996/3998/

simple numeric addresses
------------------------

Another form of address is a numeric address.  Every line of input read into the line buffer is numbered, beginning at 1.  A simple numeric address matches that numbered line.  For example, the command:

     1s/1996/1996, 1997/

substitutes the string "1996, 1997" for "1996", but only on the first line of input.

Note that even if multiple input files are specified, there is only one line numbered 1.  The first line of the second input file is numbered n + 1 where n is the number of lines in the first input file.

simple regexp addresses
-----------------------

A simple regexp address specifies that a command should only apply to lines which match a particular regexp.   Normally, this is written as a regexp between slashes:

     /^#[ \t]*endif/s/ .*//		# this command only applies to #endif lines

Any character can be used to delimit a regexp address by preceding the first occurrence of that character with a backslash:

     \,^#[ \t]*endif,s/ .*//		# this command only applies to #endif lines

the $ address
-------------

The end-of-input address, written $, is the same as the numeric address N where there are exactly N lines of input available.

For example, this script adds a signature at the end of a file:

     % cat example.sed	# This also illustrates how newlines can be escaped
     $s/$/\
     --- \
     That's all folks./

     % sed -f example.sed < /etc/passwd
     root:xXlv/n/peesXx:0:0:root:/root:/bin/bash
     us:sDSFLKJdfj:501:100:Uncle Sam:/home/us:/bin/bash
     bb:0gbezeIH7eFcA:503:100:Big Brother:/home/bb:/bin/bash
     att:NbBsVnasfK/oU:505:100:Ma Bell:/home/att:/bin/bash
     ---
     That's all folks.

Two Part Addresses
==================

A two part address consists of two simple addresses.  The first address specifies the starting condition, the second the ending condition.  These addresses are interpreted as follows.

The starting condition is applied as an ordinary simple address. When it matches a line, the command becomes operational which means that it will be applied to the current input line, and to all subsequent input lines, until the ending condition is met.

When a command first becomes operational, the ending condition is examined.  If the ending condition is a numeric or end-of-input address, then it is tested immediately.  If the ending condition is one of those types, and is met on the same line as the starting condition, then the command executes just for that input line and then ceases to be operational.  Otherwise, if the ending condition is not met on the same line as the starting condition or if the ending condition is a regexp address, the command executes on the current input line and then remains operational.

On subsequent cycles, when an operational command is reached during evaluation of the editing script, the ending condition is checked.  If it matches, then the command will execute on the current line, but will then stop being operational.  After a command ceases being operational, it may become operational again if the starting condition is met by a later input line.

Here is an example that tries to modify a copyright date, but only near the beginning of input (within the first hundred lines):

     1,100s/\(Copyright.*1996$\)/\1, 1997/

Here is an example that only applies to the middle section of a yacc grammar (recall that yacc grammars are written in three parts, separated by lines containing just "%%"):

     /^%%/,/^%%/s/statement/stmt/

Note that the starting and ending conditions of a command are tested only if the command is actually reached during evaluation.  If a branch causes evaluation of a command to be skipped, it may not become operational or non-operational when you might expect.  For example, suppose that the command:

     1,100s/\(Copyright.*1996$\)/\1, 1997/

happens to be executed for line 1, at which time it becomes operational. But suppose that branching causes the command to be skipped at line 100. Then the command will not cease to be operational and will continue to be operational for lines 101 and greater.

Similarly, if this command happens to be skipped for the first line containing "%%", then it will be operational for the third section of a yacc grammar:

     /%%/,/%%/s/statement/stmt/

Negated Addresses
=================

An address can be followed by the character !, indicating that the address is negated.  A negated address functions like its non-negated counterpart, except that when its counterpart is operational, the negated command is non-operational, and vice versa.

For example, by negating the address for the middle section of a yacc grammar, we get an address for the first and third sections:

     /^%%/,/^%%/!s/BLD_TREE/PARSER_BUIlD_TREE/

Essential Commands
==================

This section presents the basic sed commands in an expository order. Some of these descriptions recur in the next chapter, where they are are entered alphabetically in a list of all sed commands.

Basic Substitution
------------------

A very commonly used command is the s command.  Many useful sed scripts are one-liners that use this command.

  * s/regexp/replacement/flags

     This command performs regular expression substitution on the line buffer.  It replaces one or more (according to the flags) occurrences of regexp with replacement.

     The regular expression syntax, known as the Posix Basic Regular Expression syntax, is documented in an appendix (see Posix Basic Regular Expressions).

     A replacement string is used literally, except for backslashed escape codes.  The string \& in the replacement expands to all of the text matched by the regexp.  A string \<n>, where <n> is a single digit number, is replaced by the string matched by the <n>th parenthesized subexpression.  If <n> is 0, this means the same thing as \&.

     Remember that the s command only applies to lines matched by its address, regardless of what lines may be matched by the REGEXP argument.

     Any character can be used as a delimiter for the s command.  The summary above illustrates the use of / as a delimiter.  No whitespace may occur between the s and the first delimiter.

     The flexibility to use any character as a delimiter is convenient if the regexp or replacement string contains the delimiter you normally choose. For example, the following replaces a path which contains a slash and uses comma as a delimiter:

          s,/etc/inittab,/etc/init.rc,

     By default, the s command looks for the first match of regexp on a particular line and then replaces at most one match per line.  These details and others can be modified by flags appended to the end of the s command.  Some of the supported flags are listed below.  See Sed Command Reference for a complete list.

       - g

          The g flag causes the s command to apply to all occurrences of the regexp on a particular line.  The g flag can occur at most once.  It overrides a numeric flag that may be provided.

       - <n> (any number)

          A numeric flag, <n>, causes the s command to apply to the <n>th match of the regexp on a particular line.  At most one numeric flag can be provided.  A numeric flag is overridden by a g flag.

  * y/string1/string2/

     This command performs a character-oriented substitution cypher on the line buffer.  The two argument strings determine what substitutions are made.

     string1 and string2 must have an equal number of characters.  Each occurrence of a character in string1 is replaced by the corresponding character of string2.

     The delimiter used to surround and separate the two strings is arbitrary.  It can be a slash, as illustrated, or any other character. There may not be any whitespace between the y and the first delimiter.

Compound Commands
-----------------

Sometimes it is convenient to treat multiple editing commands as a unit.  For example, a single address might apply to the entire group -- although the individual commands may themselves have addresses, the individual addresses will only be tried if the address for the entire group matches.

  * { ... commands ... }

     This is a compound command.  The embedded commands are executed as a unit.  This is especially useful with a non-null address for the compound as a whole.

The Hold Buffer
---------------

These commands manage the hold buffer:

  * G

     This command appends the contents of the hold buffer to the line buffer.

  * g

     This command replaces the contents of the line buffer with the contents of the hold buffer.

  * H

     This command appends the contents of the line buffer to the hold buffer.

  * h

     This command replaces the contents of the line buffer with the contents of the hold buffer.

  * x

     This command simply swaps the hold buffer and the line buffer.

Flow of Control
---------------

In longer scripts, these commands can be used to modify flow of control through the editing script:

  * : <label>

     This command has no effect, but can be used as a branch target by conditional and non-conditional jumps.

  * b ?label?

     b is the branch command.   With no argument, this command brings the current cycle to an immediate end.   With an argument, the branch command transfers execution to a different part of the script. Execution is transfered to label which must be an argument to a : command elsewhere in the script.

  * t ?label?

     The t command is a conditional branch.  It checks the t flag which was set (or reset) by the most recently executed s command.

     If the t flag is not set, the t command is a noop.

     If the t flag is set, the t command is branch.  If no argument was provided, then the branch brings an immediate end to the current cycle.  If an argument was provided, it must be a label defined using the : command and execution is transferred to that point of the editing script.

     Note that the t flag is modified only by an s command which is actually evaluated.  If an s command is skipped by branching or is not executed because its address doesn't match, then that s command has no effect on the t flag.

  * q

     This command causes an immediate end to the current cycle and causes sed to exit.

Input and Output Commands
-------------------------

The default behavior of sed is to read input lines and write output lines to and from the line buffer.  Some commands modify this behavior by causing sed to read input lines "ahead of schedule", to write extra output or normal output ahead of schedule, or to read or write from ancillary files.

  * i text...

     This command immediately writes text... on the output.  Contrast this with the a command which adds text to the append buffer for output at the end of the cycle.

     The text can be continued over multiple lines by preceding newlines with backslashes.  Whitespace, including escaped newlines, leading up to the first non-whitespace character of text is discarded in the output of the i command.

  * a text...

     This command adds text... to the append buffer.  This means that at the end of this cycle, after the line buffer is printed, text... will be printed along with the rest of the append buffer.

     The text can be continued over multiple lines by preceding newlines with backslashes.  Whitespace, including escaped newlines, leading up to the first non-whitespace character of text is discarded in the output of the a command.

  * N

     This command tries to read a new line of input and appends to the current line buffer.   If no input is available, this command causes sed to exit.

  * n

     This command first prints the line buffer to the output, and then tries to read the next line of input.

     If the -n or --quiet flag was provided, this command doesn't produce output, but does try to read the next line of input.

     If no input is available, this command causes sed to exit.

     If input is available, execution of the editing script continues normally without beginning a new cycle.

  * P

     This command prints on the output everything up to and including the first newline of the line buffer.   If the line buffer contains no newline, the entire line buffer is printed.

  * p

     This command prints the entire line buffer on the output.  It doesn't change the line buffer and script executing continues as usual.

     This command produces output even if the -n or --quiet option was given.

  * r filename

     This command reads the entire contents of the named file and adds it to the append buffer.  At the end of the current cycle, the entire append buffer will be printed on the output.

     When a script containing a r command is first loaded, the corresponding file is opened for reading.  This is done even if execution of the script never reaches the r command itself.

  * w filename

     This command writes the entire contents of the line buffer to the named file.

     When a script containing a w command is first loaded, the corresponding file is opened for writing and truncated to 0 bytes. Each w command executed appends to the truncated file.

Sed Command Reference
*********************

This chapter contains a summary of sed syntax and addresses, and an alphabetical list of all sed commands and what they do. For a fuller explanation, see The Sed Programming Language.

Syntax Overview
===============

The general form of a scripting command is this:

     <address> <command> <command arguments>

The <address> part specifies which input lines this command applies to.

The <command> part specifies what kind of operation the sed machine should perform at the addressed lines.

The <command arguments> supply any additional parameters needed to complete the command.

Address Overview
================

An address is either null, numeric, end-of-input, a regular expression or a 2-part address.

An address may or may not be negated.

The null address matches every line.

A numeric address, <n>, matches the <n>th line.

An end-of-input address, written $, matches the last input line.

A regular expression address, written /regexp/ or, with an arbitrary delimiter such as comma \,regexp,, matches lines that match that regexp.

A two part address, written addr1,addr2 specifies a starting and ending condition for matching.  Both addr1 and addr2

The address starts matching lines when the addr1 matches and continues matching lines until ADDR2 matches.  There is a tricky irregularity here: addr2 is given the opportunity to match on the same line as addr1 if addr2 is a numeric or end-of-input address, but if addr2 is a regexp address, it doesn't get a chance to match until the next input line for which this command is tried.

Commands
========

  * : <label>

     This command has no effect, but can be used as a branch target by conditional and non-conditional jumps.

  * =

     This causes the current line number, followed by a newline, to be written to the output.

  * { ... commands ... }

     This is a compound command.  The embedded commands are executed as a unit.  This is especially useful with a non-null address for the compound as a whole.

     This command can span multiple lines.

  * a text...

     This command adds text... to the append buffer.  This means that at the end of this cycle, after the line buffer is printed, text... will be printed along with the rest of the append buffer.

     The text can be continued over multiple lines by preceding newlines with backslashes.  Whitespace, including escaped newlines, leading up to the first non-whitespace character of text is discarded in the output of the a command.

  * b ?label?

     b is the branch command.   With no argument, this command brings the current cycle to an immediate end.   With an argument, the branch command transfers execution to a different part of the script. Execution is transfered to label which must be an argument to a : command elsewhere in the script.

  * c

     Here is an odd little command.  The BSD man page says it best:

          c text
            Delete the pattern space.	With 0 or 1 address or at the end of a
            2-address range, text is written to the standard output.

     The text can be continued over multiple lines by preceding newlines with backslashes.  Whitespace, including escaped newlines, leading up to the first non-whitespace character of text is discarded in the output of the c command.

  * D

     This command deletes all characters in the line buffer, up to and including the first newline.  If the buffer contains no newline, it is simply emptied.

     If any characters are left in the line buffer, this command restarts the sed machine cycle, but without reading additional input.

     If no characters are left in the line buffer, the current cycle continues execution at the next command in the script.

  * d

     This command deletes all characters in the line buffer and immediately terminates the current cycle.   Execution will continue by printing the append buffer and then trying to read the next line of input.

  * G

     This command appends the contents of the hold buffer to the line buffer.

  * g

     This command replaces the contents of the line buffer with the contents of the hold buffer.

  * H

     This command appends the contents of the line buffer to the hold buffer.

  * h

     This command replaces the contents of the line buffer with the contents of the hold buffer.

  * i text...

     This command immediately writes text... on the output.  Contrast this with the a command which adds text to the append buffer for output at the end of the cycle.

     The text can be continued over multiple lines by preceding newlines with backslashes.  Whitespace, including escaped newlines, leading up to the first non-whitespace character of text is discarded in the output of the i command.

  * l

     The l command immediately pretty prints the input buffer on the output.  Pretty printing in this context means that only printable ASCII characters are used in the output and that a final newline is automatically provided if the line-buffer doesn't already contain one.

     If the line buffer contains a non-printable character an escape sequence is printed.  For most characters, this sequence is a backslash followed by a three-digit octal representation of the character.  Some characters, such as newline, are printed using a special escape sequence, such as \n.

  * N

     This command tries to read a new line of input and appends to the current line buffer.   If no input is available, this command causes sed to exit.

  * n

     This command first prints the line buffer to the output, and then tries to read the next line of input.

     If the -n or --quiet flag was provided, this command doesn't produce output, but does try to read the next line of input.

     If no input is available, this command causes sed to exit.

     If input is available, execution of the editing script continues normally without beginning a new cycle.

  * P

     This command prints on the output everything up to and including the first newline of the line buffer.   If the line buffer contains no newline, the entire line buffer is printed.

  * p

     This command prints the entire line buffer on the output.  It doesn't change the line buffer and script executing continues as usual.

     This command produces output even if the -n or --quiet option was given.

  * q

     This command causes an immediate end to the current cycle and causes sed to exit.

  * r filename

     This command reads the entire contents of the named file and adds it to the append buffer.  At the end of the current cycle, the entire append buffer will be printed on the output.

     When a script containing a r command is first loaded, the corresponding file is opened for reading.  This is done even if execution of the script never reaches the r command itself.

  * s/regexp/replacement/flags

     This command performs regular expression substitution on the line buffer.  It replaces one or more (according to the flags) occurrences of regexp with replacement.

     The regular expression syntax, knows as the Posix Basic Regular Expression syntax, is documented elsewhere (for example, in the Rx manual).

     A replacement string is used literally, except for backslashed escape codes.  The string \& in the replacement expands to all of the text matched by the regexp.  A string \<n>, where <n> is a single digit number, is replaced by the string matched by the <n>th parenthesized subexpression.  If <n> is 0, this means the same thing as \&.

     Remember that the s command only applies to lines matched by its address, regardless of what lines may be matched by the REGEXP argument.

     Any character can be used as a delimiter for the s command.  The summary above illustrates the use of / as a delimiter.  No whitespace may occur between the s and the first delimiter.

     The flexibility to use any character as a delimiter is convenient if the regexp or replacement string contains the delimiter you normally choose. For example, the following replaces a path which contains a slash and uses comma as a delimiter:

          s,/etc/inittab,/etc/init.rc,

     By default, the s command looks for the first match of regexp on a particular line and then replaces at most one match per line.  These details and others can be modified by flags appended to the end of the s command.  The supported flags are:

       - p

          The p flag causes the line buffer to be printed on the output after the s command completes the rest of its work.  The line buffer is printed even if a -n or --quiet option was specified.

       - g

          The g flag causes the s command to apply to all occurrences of the regexp on a particular line.  The g flag can occur at most once.  It overrides a numeric flag that may be provided.

       - w filename

          The w flag causes the line buffer to be appended to the named file after the s command completes the rest of its work.  The line buffer is printed even if a -n or --quiet option was specified.  (If both a w flag and a p flag are provided, it doesn't matter what order the outputs occur in.)

          When a script containing a w flag on an s command is first loaded, the corresponding file is opened for writing and truncated to 0 bytes.  Each w command executed appends to the truncated file.

       - <n> (any number)

          A numeric flag, <n>, causes the s command to apply to the <n>th match of the regexp on a particular line.  At most one numeric flag can be provided.  A numeric flag is overridden by a g flag.

  * t ?label?

     The t command is a conditional branch.  It checks the t flag which was set (or reset) by the most recently executed s command.

     If the t flag is not set, the t command is a noop.

     If the t flag is set, the t command is branch.  If no argument was provided, then the branch brings an immediate end to the current cycle.  If an argument was provided, it must be a label defined using the : command and execution is transferred to that point of the editing script.

     Note that the t flag is modified only by an s command which is actually evaluated.  If an s command is skipped by branching or is not executed because its address doesn't match, then that s command has no effect on the t flag.

  * w filename

     This command writes the entire contents of the line buffer to the named file.

     When a script containing a w command is first loaded, the corresponding file is opened for writing and truncated to 0 bytes. Each w command executed appends to the truncated file.

  * x

     This command simply swaps the hold buffer and the line buffer.

  * y/string1/string2/

     This command performs a character-oriented substitution cypher on the line buffer.  The two argument strings determine what substitutions are made.

     string1 and string2 must have an equal number of characters.  Each occurrence of a character in string1 is replaced by the corresponding character of string2.

     The delimiter used to surround and separate the two strings is arbitrary.  It can be a slash, as illustrated, or any other character. There may not be any whitespace between the y and the first delimiter.

Posix Basic Regular Expressions
*******************************

The Posix Basic Regular Expression language is a notation for describing textual patterns.  Regexps are typically used by comparing them to a string to see if that string matches the pattern, or by searching within a string for a substring that matches.

This chapter introduces the Posix regexp notation.  This is not a formal or precise definition of Posix regexps -- it is an intuitive and hopefully expository description of them.

An Introduction to Regexps
==========================

In the simplest cases, a regexp is just a literal string that must match exactly.   For example, the pattern:

     regexp

matches the string "regexp" and no others.

Some characters have a special meaning when they occur in a regexp. They aren't matched literally as in the previous example, but instead denote a more general pattern.   For example, the character * is used to indicate that the preceding element of a regexp may be repeated 0, 1, or more times.  In the pattern:

     smooo*th

the * indicates that the preceding o can be repeated 0 or more times.   So the pattern matches:

     smooth
     smoooth
     smooooth
     smoooooth
     ...

Suppose you want to write a pattern that literally matches a special character like * -- in other words, you don't want to * to indicate a permissible repetition, but to match * literally.  This is accomplished by quoting the special character with a backslash. The pattern:

     smoo\*th

matches the string:

     smoo*th

and no other strings.

In five cases, the pattern is reversed -- a backslash makes the character special instead of making a special character normal.  The characters +, ?, |, (, and ) are normal but the sequences \+, \?, \|, \(, and \) are special (their meaning is described later).

The remaining sections of this chapter introduce and explain the various special characters that can occur in regexps.

Literal Regexps
===============

A literal regexp is a string which contains no special characters. A literal regexp matches an identical string, but no other characters. For example:

     literally

matches

     literally

and nothing else.

Generally, whitespace characters, numbers, and letters are not special. Some punctuation characters are special and some are not (the syntax summary at the end of this chapter makes a convenient reference for which characters are special and which aren't).

Character Sets
==============

This section introduces the special characters . and [.

. matches any character except the NULL character.  For example:

     p.ck

matches

     pick
     pack
     puck
     pbck
     pcck
     p.ck

     ...

[ begins a character set.  A character set is similar to . in that it matches not a single, literal character, but any of a set of characters.   [ is different from . in that with [, you define the set of characters explicitly.

There are three basic forms a character set can take.

In the first form, the character set is spelled out:

     [<cset-spec>]	-- every character in <cset-spec> is in the set.

In the second form, the character set indicated is the negation of a character set is explicitly spelled out:

     [^<cset-spec>]	-- every character *not* in <cset-spec> is in the set.

A <cset-spec> is more or less an explicit enumeration of a set of characters.  It can be written as a string of individual characters:

     [aeiou]

or as a range of characters:

     [0-9]

These two forms can be mixed:

     [A-za-z0-9_$]

Note that special regexp characters (such as *) are not special within a character set.  -, as illustrated above, is special, except, as illustrated below, when it is the first character mentioned.

This is a four-character set:

     [-+*/]

The third form of a character set makes use of a pre-defined "character class":

     [[:class-name:]] -- every character described by class-name is in the set.

The supported character classes are:

     alnum	- the set of alpha-numeric characters
     alpha	- the set of alphabetic characters
     blank	- tab and space
     cntrl	- the control characters
     digit	- decimal digits
     graph	- all printable characters except space
     lower	- lower case letters
     print	- the "printable" characters
     punct	- punctuation
     space	- whitespace characters
     upper	- upper case letters
     xdigit	- hexidecimal digits

Finally, character class sets can also be inverted:

     [^[:space:]] - all non-whitespace characters

Character sets can be used in a regular expression anywhere a literal character can.

Subexpressions
==============

A subexpression is a regular expression enclosed in \( and \).  A subexpression can be used anywhere a single character or character set can be used.

Subexpressions are useful for grouping regexp constructs.  For example, the repeat operator, *, usually applies to just the preceding character.   Recall that:

     smooo*th

matches

     smooth
     smoooth
     ...

Using a subexpression, we can apply * to a longer string:

     banan\(an\)*a

matches

     banana
     bananana
     banananana
     ...

Subexpressions also have a special meaning with regard to backreferences and substitutions (see Backreferences).

Repeated Subexpressions
=======================

* is the repeat operator.  It applies to the preceding character, character set, subexpression or backreference.  It indicates that the preceding element can be matched 0 or more times:

     bana\(na\)*

matches

     bana
     banana
     bananana
     banananana
     ...

\+ is similar to * except that \+ requires the preceding element to be matched at least once.  So while:

     bana\(na\)*

matches

     bana

     bana(na\)\+

does not.   Both match

     banana
     bananana
     banananana
     ...

Thus, bana\(na\)+ is short-hand for banana\(na\)*.

Optional Subexpressions
=======================

\? indicates that the preceding character, character set, or subexpression is optional.  It is permitted to match, or to be skipped:

     CSNY\?

matches both

     CSN

and

     CSNY

Counted Subexpressions
======================

An interval expression, {m,n} where m and n are non-negative integers with n >= m, applies to the preceding character, character set, subexpression or backreference.  It indicates that the preceding element must match at least m times and may match as many as n times.

For example:

     c\([ad]\){1,4}

matches

     car
     cdr
     caar
     cdar
     ...
     caaar
     cdaar
     ...
     cadddr
     cddddr

Alternative Subexpressions
==========================

An alternative is written:

     regexp-1\|regexp-2\|regexp-3\|...

It matches anything matched by some regexp-n.  For example:

     Crosby, Stills, \(and Nash\|Nash, and Young\)

matches

     Crosby, Stills, and Nash

and

     Crosby, Stills, Nash, and Young

Backreferences, Extractions and Substitutions
=============================================

A backreference is written \n where n is some single digit other than 0.  To be a valid backreference, there must be at least n parenthesized subexpressions in the pattern prior to the backreference.

A backreference matches a literal copy of whatever was matched by the corresponding subexpression.  For example,

     \(.*\)-\1

matches:

     go-go
     ha-ha
     wakka-wakka
     ...

In some applications, subexpressions are used to extract substrings. For example, Emacs has the functions match-beginnning and match-end which report the positions of strings matched by subexpressions.  These functions use the same numbering scheme for subexpressions as backreferences, with the additional rule that subexpression 0 is defined to be the whole regexp.

In some applications, subexpressions are used in string substitution. This again uses the backreference numbering scheme. For example, this sed command:

     s/From:.*<\(.*\)>/To: \1/

first matches the line:

     From: Joe Schmoe <schmoe@uspringfield.edu>

when it does, subexpression 1 matches "schmoe@uspringfield.edu". The command replaces the matched line with "To: \1" after doing subexpression substitution on it to get:

     To: schmoe@uspringfield.edu

A Summary of Regexp Syntax
==========================

In summary, regexps can be:

abcd -- matching a string literally

. -- matching everything except NULL

[a-z_?], ^[a-z_?], [[:alpha:]] and [^[:alpha:]] -- matching character sets

\(subexp\) -- grouping an expression into a subexpression.

\n -- match a copy of whatever was matched by the nth subexpression.

The following special characters and sequences can be applied to a character, character set, subexpression, or backreference:

* -- repeat the preceding element 0 or more times.

\+ -- repeat the preceding element 1 or more times.

\? -- match the preceding element 0 or 1 time.

{m,n} -- match the preceding element at least m, and as many as n times.

regexp-1\|regexp-2\|.. -- match any regexp-n.

A special character, like . or * can be made into a literal character by prefixing it with \.

A special sequence, like \+ or \? can be made into a literal character by dropping the \.

Ambiguous Patterns
==================

Sometimes a regular expression appears to be ambiguous.  For example, suppose we compare the pattern:

     begin\|beginning

to the string

     beginning

either just the first 5 characters will match, or the whole string will match.

In every case like this, the longer match is preferred.  The whole string will match.

Sometimes there is ambiguity not about how many characters to match, but where the subexpressions occur within the match.  This can effect extraction functions like Emacs' match-beginning or rewrite functions like sed's s command.  For example, consider matching the pattern:

     b\(\[^q]*\)\(ing\)?

against the string

     beginning

One possibility is that the first subexpression matches "eginning" and the second is skipped.  Another possibility is that the first subexpression matches "eginn" and the second matches "ing".

The rule is that consistent with matching as many characters as possible, the length of lower numbered subexpressions is maximized in preference to maximizing the length of later subexpressions.

In the case of the above example, the two possible matches are equal in overall length.  Therefore, it comes down to maximizing the lower-numbered subexpression, \1.  The correct answer is that \1 matches "eginning" and \2 is skipped.

Hacking Sed
***********

GNU sed is a short and simple program.  Most changes to sed are easy to plan just by reading the source.

One word of advice if you plan to port GNU sed to an environment where memory is at a premium: modify the caching parameter to the regular expression matcher used by sed.  These parameters are set near the top of main in sed.c.