AWK Command for Linux and Unix: Syntax, Examples, and Text Processing

The AWK command is a text-processing tool included with Linux and Unix systems. It examines input one line at a time, separates each line into individual fields, and applies pattern-and-action rules that you define to filter, modify, analyze, or report on the data. AWK takes its name from its creators, Alfred Aho, Peter Weinberger, and Brian Kernighan. It has been part of Unix since 1977 and continues to be a dependable utility for Linux administration.

AWK is particularly useful when you want to extract columns from structured text, examine log files, process CSV data, or generate quick reports directly within a shell pipeline without creating a complete script.

This tutorial explains AWK syntax and provides practical examples of pattern matching, field separators, and built-in variables. It also shows how AWK differs from other text-processing utilities such as grep and sed.

Key Takeaways

• AWK reads text one line at a time and separates each line into fields such as $1, $2, and $NF according to the configured field separator.
• The standard syntax is awk 'pattern { action }' filename. If the pattern is left out, AWK performs the action on every input line.
• The -F option defines the field separator, such as -F, for CSV files or -F: for colon-separated data.
• Important built-in variables include NR for the current line number, NF for the number of fields, FS for the input field separator, OFS for the output field separator, RS for the input record separator, and ORS for the output record separator.
• A BEGIN block executes before input lines are processed, making it useful for initializing variables, configuring settings, or displaying headings.
• An END block executes after all input has been processed and is useful for summaries, totals, and final calculations.
• AWK works especially well for extracting columns, calculating values from fields, transforming structured text, and summarizing data such as logs and CSV files.
• Use grep for simple searches and sed for text substitutions or edits. When a task involves fields, calculations between columns, or custom-formatted output, AWK is often the more suitable tool.

Run Linux workloads on scalable and reliable cloud computing infrastructure designed for developers.

Prerequisites

To follow this tutorial, you will need:

• A Linux or Unix system with AWK installed. Most distributions include GNU AWK (gawk) by default. You can check the installed version with awk --version.
• Basic familiarity with using the Linux command line.

What Is the AWK Command in Linux and Unix?

AWK is both a pattern-scanning language and a text-processing utility. You provide rules written as pattern { action }, and AWK evaluates those rules for each line of input. If a line satisfies the pattern, its associated action is executed. When no pattern is supplied, the action is performed for every line.

GNU AWK, commonly called gawk, is one of the most widely used AWK implementations and serves as the default awk interpreter on many Linux distributions. It expands on the original POSIX AWK specification with capabilities such as improved Unicode handling and additional string-processing features. You can identify the installed version by running:

awk --version

A typical Linux system might display output similar to:

GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)

How AWK Is Different from grep and sed

AWK, grep, and sed all process text, but each tool is designed around a different type of task. grep searches for lines matching a specified pattern and returns those lines. sed performs substitutions and other editing operations on streams of text. AWK can handle similar tasks while also understanding fields, supporting calculations, and creating formatted reports.

A simple comparison is shown below:

For additional information about the individual tools, consult tutorials covering the grep command and the sed command.

AWK Syntax and Basic Structure

Handling structured text files on Linux or Unix can involve repeatedly performing similar parsing operations. AWK simplifies this process by allowing you to define the patterns and actions you need while the utility takes care of processing the input.

The AWK Command Format

The basic AWK syntax is:

awk options 'pattern { action }' input-file

You can also redirect the generated output into another file:

awk options 'pattern { action }' input-file > output-file

Frequently used options include:

• -F defines the field separator, such as -F: for files whose fields are separated by colons.
• -f instructs AWK to load its rules from a script file instead of directly from the command line.
• -v assigns a variable before AWK starts processing the input.

Understanding Pattern and Action Blocks

An AWK rule consists of a pattern and an action enclosed in curly braces. Either part can be omitted:

• When the pattern is omitted, AWK executes the action for every input line.
• When the action block is omitted, AWK automatically prints every line that matches the pattern.

# Print every line (no pattern, print is the default action)
awk '{ print }' file.txt

# Print only lines that match a pattern (no explicit action needed)
awk '/error/' file.txt

How to Run the AWK Command in Linux

AWK is useful for structured text whenever you need to quickly select, change, or summarize individual columns. Before moving to more advanced operations, it helps to understand the basic methods for running AWK commands on Linux.

Creating a Sample File

The examples throughout this tutorial use a file named file.txt. Create the file so that you can reproduce the commands:

cat > file.txt << 'EOF'
Item    Model   Country Cost
Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Watch   Galaxy  Korea   299
Tablet  iPad    USA     499
Camera  Sony    Japan   799
EOF

This example file contains four columns: Item, Model, Country, and Cost.

Running AWK Against a File

Specify the input filename as the final argument:

awk '{ print $0 }' file.txt

The $0 variable represents the complete current line. The command produces:

Item    Model   Country Cost
Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Watch   Galaxy  Korea   299
Tablet  iPad    USA     499
Camera  Sony    Japan   799

Running AWK with Command-Line Input

Instead of reading input from a file, you can pipe data from another command directly into AWK:

echo "Alice Engineering 95000" | awk '{ print $1, "works in", $2 }'

The resulting output is:

Alice works in Engineering

Running AWK from a Script File

When an AWK program becomes longer or needs to be reused, you can store its rules in a separate file and load that file with the -f option:

# Save rules to a script file
echo '{ print $1, $4 }' > print_item_cost.awk

# Run it
awk -f print_item_cost.awk file.txt

The output is:

Item Cost
Phone 999
Laptop 1299
Watch 299
Tablet 499
Camera 799

AWK Field Separators and Column Operations

AWK divides every input line into separate fields according to a field separator. By default, AWK treats whitespace, including spaces and tabs, as the separator.

Using the Default Field Separator

AWK numbers fields beginning with $1, while $0 always represents the complete line.

To display the second and third columns, use:

awk '{ print $2 "\t" $3 }' file.txt

The output is:

Model   Country
iPhone  USA
MacBook USA
Galaxy  Korea
iPad    USA
Sony    Japan

Setting a Custom Field Separator with -F

Use the -F option when the input uses something other than whitespace to separate fields. This is common in CSV data and in system files such as /etc/passwd, where individual fields are separated by colons.

# Print the first field (username) from /etc/passwd
awk -F: '{ print $1 }' /etc/passwd | head -5

The output is:

root
daemon
bin
sys
sync

Printing Columns with $1, $2, and $NF

The special $NF reference represents the final field in the current line, regardless of the number of fields contained in that line.

# Print the first and last fields
awk '{ print $1, $NF }' file.txt

The result is:

Item Cost
Phone 999
Laptop 1299
Watch 299
Tablet 499
Camera 799

AWK Built-In Variables Reference

AWK includes several predefined variables that provide information about the current input and allow you to control the way input and output are formatted.

NR, NF, FS, RS, OFS, and ORS Explained

Practical Examples with Built-In Variables

Use NR to print each line together with its corresponding line number:

awk '{ print NR, $0 }' file.txt

The output is:

1 Item    Model   Country Cost
2 Phone   iPhone  USA     999
3 Laptop  MacBook USA     1299
4 Watch   Galaxy  Korea   299
5 Tablet  iPad    USA     499
6 Camera  Sony    Japan   799

You can use OFS to change the separator used between output fields. The following command creates comma-separated output:

awk 'BEGIN { OFS="," } { print $1, $2, $3 }' file.txt

The output becomes:

Item,Model,Country
Phone,iPhone,USA
Laptop,MacBook,USA
Watch,Galaxy,Korea
Tablet,iPad,USA
Camera,Sony,Japan

Use NF when you want to show the number of fields contained in each line:

awk '{ print NF, $0 }' file.txt

The result is:

4 Item    Model   Country Cost
4 Phone   iPhone  USA     999
4 Laptop  MacBook USA     1299
4 Watch   Galaxy  Korea   299
4 Tablet  iPad    USA     499
4 Camera  Sony    Japan   799

AWK Pattern Matching Examples

The following examples demonstrate several common techniques for finding and processing text through AWK pattern matching.

Matching Lines That Include a String

To display every line containing the letter o, run:

awk '/o/ { print $0 }' file.txt

The output is:

Item    Model   Country Cost
Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Watch   Galaxy  Korea   299
Camera  Sony    Japan   799

To determine how many lines match a particular pattern, increment a counter and display the final value from an END block:

awk '/a/ { ++cnt } END { print "Count = ", cnt }' file.txt

The output is:

Count =  4

Using Regular Expressions with AWK

AWK supports regular expressions inside the /pattern/ notation. The following expression selects lines beginning with an uppercase letter followed by lowercase characters:

awk '/^[A-Z][a-z]/ { print $0 }' file.txt

The output is:

Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Watch   Galaxy  Korea   299
Tablet  iPad    USA     499
Camera  Sony    Japan   799

Matching Particular Field Values

When you need to test one specific field rather than the entire line, use a comparison expression instead of matching the complete record with a regular expression:

# Print lines where Country (field 3) is USA
awk '$3 == "USA" { print $0 }' file.txt

The output is:

Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Tablet  iPad    USA     499

AWK can also filter records according to numeric values:

# Print lines where Cost (field 4) is greater than 500
awk '$4 > 500 { print $0 }' file.txt

The output is:

Item    Model   Country Cost
Phone   iPhone  USA     999
Laptop  MacBook USA     1299
Camera  Sony    Japan   799

To return only lines containing more than 20 characters, use AWK’s built-in length function:

awk 'length($0) > 20' file.txt

AWK BEGIN and END Blocks

AWK is capable of more than filtering individual lines. Its BEGIN and END blocks allow you to prepare settings before input processing starts and perform summary operations after all records have been handled.

What BEGIN and END Blocks Do

BEGIN and END are special patterns provided by AWK:

• The BEGIN block executes once before AWK reads any input. It can be used to initialize variables, configure separators, or print headings.
• The END block executes once after AWK finishes processing all available input. It is useful for displaying totals and summaries.

Using BEGIN Before Processing

awk 'BEGIN { print "Item\tCost"; print "----\t----" } NR > 1 { print $1 "\t" $4 }' file.txt

The output is:

Item    Cost
----    ----
Phone   999
Laptop  1299
Watch   299
Tablet  499
Camera  799

Using END for Summaries After Processing

awk 'BEGIN { print "Starting AWK processing..." } { print $1 } END { print "Done. Processed " NR " lines." }' file.txt

The output is:

Starting AWK processing...
Item
Phone
Laptop
Watch
Tablet
Camera
Done. Processed 6 lines.

AWK Conditional Logic and Control Flow

Some data-processing tasks require more than simple pattern matching. AWK includes control-flow features that can handle summary calculations, record tracking, initialization, and conditional operations. These features allow more complex processing to remain efficient and flexible.

if and if-else Statements in AWK

AWK supports if, if-else, and nested conditional statements inside its action blocks.

# Label items as expensive or affordable based on Cost
awk 'NR > 1 { if ($4 > 500) print $1, "is expensive"; else print $1, "is affordable" }' file.txt

The command produces:

Phone is expensive
Laptop is expensive
Watch is affordable
Tablet is affordable
Camera is expensive

Multiple expressions can be combined with && for logical AND or || for logical OR:

# Print USA items that cost more than 500
awk '$3 == "USA" && $4 > 500 { print $0 }' file.txt

The output is:

Phone   iPhone  USA     999
Laptop  MacBook USA     1299

Loops in AWK: for and while

AWK includes both for and while loops. A loop is useful when you need to move through each field of a particular record:

# Print each field on its own line, with its field number
awk 'NR == 2 { for (i = 1; i <= NF; i++) print "Field " i ": " $i }' file.txt

The result is:

Field 1: Phone
Field 2: iPhone
Field 3: USA
Field 4: 999

A while loop can be used in a similar way:

awk 'BEGIN { i = 1; while (i <= 5) { print "Line:", i; i++ } }'

The output is:

Line: 1
Line: 2
Line: 3
Line: 4
Line: 5

Practical AWK Use Cases and Examples

The following practical examples demonstrate how AWK can be used for common data-processing tasks directly from a terminal.

Parsing /etc/passwd with AWK

The /etc/passwd file separates its fields with colons and stores one user record on each line. Using AWK with -F: provides a straightforward way to retrieve individual fields.

# Print the username and default shell for each user
awk -F: '{ print $1, $7 }' /etc/passwd | head -5

On a typical Ubuntu system, the output may look like:

root /bin/bash
daemon /usr/sbin/nologin
bin /usr/sbin/nologin
sys /usr/sbin/nologin
sync /bin/sync

To display only users whose shell is set to /bin/bash, use:

awk -F: '$7 == "/bin/bash" { print $1 }' /etc/passwd

Processing CSV Files

Create the following sample CSV file:

cat > employees.csv << 'EOF'
Name,Department,Salary
Alice,Engineering,95000
Bob,Marketing,72000
Charlie,Engineering,88000
Diana,HR,65000
Eve,Marketing,78000
EOF

Print the name and department of every employee while excluding the header line:

awk -F, 'NR > 1 { print $1, "works in", $2 }' employees.csv

The output is:

Alice works in Engineering
Bob works in Marketing
Charlie works in Engineering
Diana works in HR
Eve works in Marketing

Summarizing System Log Files

Create a sample access log:

cat > access.log << 'EOF'
192.168.1.1 GET /index.html 200 1024
192.168.1.2 POST /login 404 512
192.168.1.3 GET /about.html 200 2048
192.168.1.1 GET /contact.html 500 256
192.168.1.4 GET /index.html 200 1024
EOF

Display all IP addresses associated with a status code of 200:

awk '$4 == 200 { print $1 }' access.log

The output is:

192.168.1.1
192.168.1.3
192.168.1.4

To count how many 404 errors appear in the file, run:

awk '$4 == 404 { count++ } END { print "404 errors:", count }' access.log

The result is:

404 errors: 1

Calculating Column Totals

To calculate the total of every value in the Cost column of file.txt while ignoring the heading row, use:

awk 'NR > 1 { sum += $4 } END { print "Total cost:", sum }' file.txt

The output is:

Total cost: 3895

You can also calculate an average:

awk -F, 'NR > 1 { sum += $3; count++ } END { print "Average salary:", sum / count }' employees.csv

The output is:

Average salary: 79600

Saving AWK Output to a File

When you need to store the results produced by an AWK command, redirect the output with the > operator:

awk '/a/ { print $3 "\t" $4 }' file.txt > output.txt

Confirm the saved content with cat:

cat output.txt

The file contains:

USA     1299
Korea   299
USA     499
Japan   799

AWK vs. sed vs. grep: Choosing the Right Tool

Choose grep when the main goal is to determine whether a pattern occurs and return the matching lines. Choose sed when you need to replace or remove text. Use awk when the task depends on individual fields, value comparisons, calculations, or output generated from several columns.

For more detailed information, consult tutorials covering the grep command and the sed command.

AWK FAQs

1. What Is the AWK Command in Unix?

AWK is a powerful text-processing utility available on Linux and Unix systems. It is designed for pattern scanning and reporting and processes files or streams one line at a time. For every input line, AWK automatically separates the record into fields according to a delimiter, with whitespace used by default. You can then apply custom pattern-and-action rules to filter records, retrieve columns, calculate numeric values, create reports, and perform other operations. Because AWK works directly with individual columns, it is particularly useful for structured or delimited information such as CSV data, configuration files, and logs.

2. How Do You Run the AWK Command in Linux?

A typical AWK command uses the following syntax:

awk 'pattern { action }' filename

For every line in the specified file, AWK evaluates the pattern and executes the related action when the pattern matches. If the pattern is omitted, the action is performed for every line. Input can also be streamed into AWK from another command with a pipeline such as command | awk '{ action }', which makes AWK convenient for shell pipelines. AWK can process several input files at once, and command-line options such as -F can define a field separator while -f can load a larger AWK program from another file. This flexibility makes AWK useful for many forms of text processing and automation.

3. What Does awk '{ print $1 }' Do?

The command awk '{ print $1 }' displays the first field or column from each input line. By default, AWK uses whitespace to separate fields, so $1 refers to the first whitespace-separated word or value.

For example, if an input line contains Name Age Country, this command outputs only the Name portion of the line. It provides a quick way to extract the first column from structured data.

4. What Does awk '{ print $2 }' Do?

The command awk '{ print $2 }' retrieves and displays the second field from every input record. For example, when processing a line such as John 30 Engineer, AWK prints 30 because $2 points to the second field. The same mechanism can be used to extract any desired column from tabular or whitespace-separated information.

5. What Is the -F Flag in AWK Used For?

The -F option defines the field separator AWK uses when dividing each input line into fields. AWK normally separates fields using whitespace, but -F allows you to supply another delimiter character or regular expression. For example, awk -F: '{ print $1 }' /etc/passwd tells AWK to treat the colon character (:) as the separator. This is useful with files such as /etc/passwd, whose fields are separated by colons, and makes AWK adaptable to many kinds of structured data.

6. What Are NR and NF in AWK?

NR and NF are two important built-in AWK variables. NR means “Number of Record” and stores the number of the record currently being processed. Its value increases as AWK moves through the input. This makes it useful for numbering output or applying operations only to particular lines.

NF means “Number of Fields” and contains the number of fields present in the current record. It is especially helpful when different lines contain different numbers of columns or when you need to refer to the last field through $NF. Both variables are commonly used when filtering, formatting, and controlling AWK processing.

7. What Is the Difference Between AWK and gawk?

gawk is the GNU implementation of the AWK programming language and is the standard AWK interpreter on many modern Linux distributions. Traditional AWK follows the POSIX specification, while gawk adds further capabilities including improved Unicode and multibyte-character handling, additional string and arithmetic functions, built-in networking functionality, and expanded command-line options. Most programs written for standard AWK also work with gawk, while the GNU implementation provides extra functionality for more advanced text-processing requirements.

8. When Should You Use AWK Instead of Python or sed?

AWK is well suited to quick one-line operations involving column extraction, field-based filtering, or inline text processing, particularly inside shell pipelines or when working with structured tabular information. When a task requires more advanced programming logic, complex data processing, or communication with other systems, Python may be more appropriate because of its large collection of libraries and its maintainability for larger programs. In comparison, sed is particularly effective for simple substitutions and line-by-line edits in files or text streams. In general, AWK is useful for fast field-aware processing, Python is suitable for more complex scripting, and sed works well for straightforward text replacement.

Conclusion

AWK is a simple yet powerful text-processing tool for Linux and Unix systems. Whether you need to extract a particular column from a file, calculate totals, examine log data, or decide whether AWK, grep, or sed is the most appropriate utility for a task, the examples in this tutorial provide a strong foundation for working with AWK.