AWK Command in Linux and Unix: Syntax, Examples, and Use Cases

The AWK command is a text-processing utility included with Linux and Unix systems. It processes input one line at a time, separates each line into fields, and uses pattern-action rules that you define to filter, modify, or report on the data. AWK takes its name from its creators, Alfred Aho, Peter Weinberger, and Brian Kernighan. It has been part of Unix since 1977 and continues to be one of the most dependable tools available to Linux administrators.

AWK is particularly useful when you want to extract columns from structured text, analyze log files, work with CSV data, or create quick reports directly from a shell pipeline without having to write a complete script.

In this tutorial, you will learn AWK syntax, explore practical examples involving pattern matching, field separators, and built-in variables, and understand how AWK differs from other text-processing utilities such as grep and sed.

Key Takeaways:

• AWK handles text one line at a time and divides each line into fields such as $1, $2, …, and $NF according to a field separator.
• The standard syntax is awk 'pattern { action }' filename. When no pattern is provided, the action is performed on every input line.
• The -F option defines the field separator, such as -F, for CSV files or -F: for colon-separated files.
• Important built-in variables include NR for the current line number, NF for the number of fields in the current line, FS for the input field separator, OFS for the output field separator, RS for the input record separator, and ORS for the output record separator.
• The BEGIN block executes before input lines are processed and is useful for initializing variables or displaying headings.
• The END block executes after all input has been processed and is well suited to displaying summaries, totals, or final calculations.
• AWK is highly effective for extracting columns, carrying out arithmetic on fields, and transforming or summarizing structured text such as log files and CSV data.
• Use grep for simple searches and sed for direct text-editing operations, while AWK is generally the stronger choice when fields, column calculations, or customized output are involved.

Run Linux workloads on scalable and reliable cloud computing infrastructure designed for developers.

Prerequisites

To follow this tutorial, you will need:

• A Linux or Unix system where AWK is installed. Most distributions include GNU AWK (gawk) by default. You can check your installation with awk --version.
• Basic familiarity with Linux command-line usage.

What Is the AWK Command in Linux and Unix?

AWK is both a pattern-scanning tool and a text-processing language. You provide rules using the form pattern { action }, and AWK applies those rules to every line it receives as input. If a line satisfies the pattern, the associated action is performed. If no pattern is specified, the action is executed for every line.

GNU AWK (gawk) is currently the most commonly used AWK implementation and serves as the default awk implementation on many Linux distributions. It expands on the original POSIX AWK specification with features including improved Unicode handling and additional string functions. You can check the installed version with:

awk –version

On a typical Linux system, the output can look like this:

GNU Awk 5.2.1, API 3.2, PMA Avon 8-g1, (GNU MPFR 4.2.1, GNU MP 6.3.0)

How AWK Differs from grep and sed

All three utilities process text, but each one focuses on a different type of task. grep finds lines that match a pattern and displays them. sed performs substitutions and editing operations on a stream of text. AWK can perform similar operations while also understanding fields, supporting calculations, and generating formatted reports.

A quick comparison:

For additional information about the individual utilities, see tutorials covering the grep command and the sed command.

AWK Syntax and Basic Structure

Processing structured text files on Unix or Linux can involve repeating the same parsing operations many times. AWK provides a simpler method: define the patterns and actions you require, and allow the utility to perform the processing.

The AWK Command Format

The basic AWK command format is:

awk options ‘pattern { action }’ input-file

You can also redirect the resulting output into another file:

awk options ‘pattern { action }’ input-file > output-file

Frequently used options include:

• -F defines the field separator, for example -F: when processing colon-delimited data.
• -f loads AWK instructions from a script file rather than specifying them directly on the command line.
• -v assigns a variable before AWK begins processing input.

Pattern and Action Blocks Explained

An AWK rule consists of two components: a pattern and an action enclosed in curly braces. Either component may be omitted:

• If the pattern is left out, AWK performs the action on every line of input.
• If the action is omitted, AWK automatically prints every line that matches the pattern.

# Print every line (no pattern, print is the default action)

awk ‘{ print }’ file.txt
# Print only lines that match a pattern (no explicit action needed)

awk ‘/error/’ file.txt

How to Run the AWK Command in Linux

AWK is useful when handling structured text and you need a fast way to select, transform, or summarize columns. Before moving to more advanced techniques, start with the basic methods for executing AWK commands on Linux.

Setting Up a Sample File

The examples throughout this tutorial use a file named file.txt. Create the file so you can reproduce the examples:

cat > file.txt << ‘EOF’

Item Model Country Cost

Phone iPhone USA 999

Laptop MacBook USA 1299

Watch Galaxy Korea 299

Tablet iPad USA 499

Camera Sony Japan 799

EOF

The file contains four fields: Item, Model, Country, and Cost.

Running AWK on a File

Provide the filename as the final argument of the command:

awk ‘{ print $0 }’ file.txt

$0 represents the complete input line. The command produces:

Item Model Country Cost

Phone iPhone USA 999

Laptop MacBook USA 1299

Watch Galaxy Korea 299

Tablet iPad USA 499

Camera Sony Japan 799

Running AWK on Command-Line Input

Instead of reading data from a file, you can pipe command output directly into AWK:

echo “Alice Engineering 95000” | awk ‘{ print $1, “works in”, $2 }’

The output is:

Alice works in Engineering

Running AWK from a Script File

For longer AWK programs or instructions you plan to reuse, store the rules in a separate file and execute them with the -f option:

# Save rules to a script file

echo ‘{ print $1, $4 }’ > print_item_cost.awk
# Run it

awk -f print_item_cost.awk file.txt

The output is:

Item Cost

Phone 999

Laptop 1299

Watch 299

Tablet 499

Camera 799

AWK Field Separator and Column Operations

AWK separates every input line into individual fields according to a delimiter. Unless another separator is configured, any whitespace, including spaces and tabs, is treated as the delimiter.

Using the Default Field Separator

AWK numbers fields beginning with $1, while $0 always represents the full line.

To display the second and third columns, use:

awk ‘{ print $2 “\t” $3 }’ file.txt

The result is:

Model Country

iPhone USA

MacBook USA

Galaxy Korea

iPad USA

Sony Japan

Specifying a Custom Field Separator with -F

Use the -F option whenever input uses a separator other than whitespace. This is frequently required for CSV data and system files such as /etc/passwd, where fields are separated with colons.

# Print the first field (username) from /etc/passwd

awk -F: ‘{ print $1 }’ /etc/passwd | head -5

The output is:

root

daemon

bin

sys

sync

Printing Specific Columns with $1, $2, and $NF

$NF is a built-in reference to the final field of the current record, regardless of the number of fields that record contains.

# Print the first and last fields

awk ‘{ print $1, $NF }’ file.txt

The output is:

Item Cost

Phone 999

Laptop 1299

Watch 299

Tablet 499

Camera 799

AWK Built-In Variables Reference

AWK includes several predefined variables that provide information about the current input record and allow you to configure input and output formatting.

NR, NF, FS, RS, OFS, and ORS Explained

Practical Examples Using Built-In Variables

Use NR to display every line together with its line number:

awk ‘{ print NR, $0 }’ file.txt

The result is:

1 Item Model Country Cost

2 Phone iPhone USA 999

3 Laptop MacBook USA 1299

4 Watch Galaxy Korea 299

5 Tablet iPad USA 499

6 Camera Sony Japan 799

You can change the delimiter used for output with OFS. This example generates comma-separated output:

awk ‘BEGIN { OFS=”,” } { print $1, $2, $3 }’ file.txt

The output is:

Item,Model,Country

Phone,iPhone,USA

Laptop,MacBook,USA

Watch,Galaxy,Korea

Tablet,iPad,USA

Camera,Sony,Japan

Use NF to print the number of fields found in every input line:

awk ‘{ print NF, $0 }’ file.txt

The output is:

4 Item Model Country Cost

4 Phone iPhone USA 999

4 Laptop MacBook USA 1299

4 Watch Galaxy Korea 299

4 Tablet iPad USA 499

4 Camera Sony Japan 799

AWK Pattern Matching Examples

The following examples demonstrate common ways to identify and process text using AWK pattern matching.

Matching Lines That Contain a String

To display every line containing the letter o, run:

awk ‘/o/ { print $0 }’ file.txt

The result is:

Item Model Country Cost

Phone iPhone USA 999

Laptop MacBook USA 1299

Watch Galaxy Korea 299

Camera Sony Japan 799

To determine how many lines match a particular pattern, increment a counter and display it from an END block:

awk ‘/a/ { ++cnt } END { print “Count = “, cnt }’ file.txt

The output is:

Count = 4

Using Regular Expressions in AWK

AWK accepts regular expressions directly inside the /pattern/ form. To match lines beginning with an uppercase letter followed by lowercase letters, use:

awk ‘/^[A-Z][a-z]/ { print $0 }’ file.txt

The result is:

Phone iPhone USA 999

Laptop MacBook USA 1299

Watch Galaxy Korea 299

Tablet iPad USA 499

Camera Sony Japan 799

Matching on Specific Field Values

If you need to match the contents of an individual field rather than search the complete line, use a field comparison instead of a regular expression:

# Print lines where Country (field 3) is USA

awk ‘$3 == “USA” { print $0 }’ file.txt

The output is:

Phone iPhone USA 999

Laptop MacBook USA 1299

Tablet iPad USA 499

Numeric comparisons can also be used for filtering:

# Print lines where Cost (field 4) is greater than 500

awk ‘$4 > 500 { print $0 }’ file.txt

The output is:

Item Model Country Cost

Phone iPhone USA 999

Laptop MacBook USA 1299

Camera Sony Japan 799

To return only lines containing more than 20 characters, use AWK’s built-in length function:

awk ‘length($0) > 20’ file.txt

AWK BEGIN and END Blocks

AWK provides capabilities beyond processing individual lines. The BEGIN and END blocks allow you to perform initialization before input is read and execute summary operations after all input has been processed.

What BEGIN and END Do

BEGIN and END are special AWK patterns:

• The BEGIN block runs one time before AWK reads any input. It can be used to display headings, initialize variables, or configure separators.
• The END block runs one time after AWK has processed all input. It can be used to display totals, summaries, or other final results.

Using BEGIN to Set Variables Before Processing

awk ‘BEGIN { print “Item\tCost”; print “—-\t—-” } NR > 1 { print $1 “\t” $4 }’ file.txt

The output is:

Item Cost

—- —-

Phone 999

Laptop 1299

Watch 299

Tablet 499

Camera 799

Using END to Print Summaries After Processing

awk ‘BEGIN { print “Starting AWK processing…” } { print $1 } END { print “Done. Processed ” NR ” lines.” }’ file.txt

The output is:

Starting AWK processing…

Item

Phone

Laptop

Watch

Tablet

Camera

Done. Processed 6 lines.

AWK Conditional Logic and Control Flow

Some data-processing tasks require more than straightforward pattern matching. AWK includes control-flow features that help when a workflow requires summary calculations, tracking records, initialization, or decisions during processing. Blocks and conditional logic make these operations efficient and flexible.

if and if-else Statements in AWK

AWK supports if, if-else, and nested conditional statements inside action blocks.

# Label items as expensive or affordable based on Cost

awk ‘NR > 1 { if ($4 > 500) print $1, “is expensive”; else print $1, “is affordable” }’ file.txt

The result is:

Phone is expensive

Laptop is expensive

Watch is affordable

Tablet is affordable

Camera is expensive

You can join several conditions with && for AND or || for OR:

# Print USA items that cost more than 500

awk ‘$3 == “USA” && $4 > 500 { print $0 }’ file.txt

The output is:

Phone iPhone USA 999

Laptop MacBook USA 1299

Loops in AWK: for and while

AWK supports both for and while loops. These constructs are useful when you need to iterate through all fields in a record:

# Print each field on its own line, with its field number

awk ‘NR == 2 { for (i = 1; i <= NF; i++) print “Field ” i “: ” $i }’ file.txt

The output is:

Field 1: Phone

Field 2: iPhone

Field 3: USA

Field 4: 999

A while loop can be used in a similar way:

awk ‘BEGIN { i = 1; while (i <= 5) { print “Line:”, i; i++ } }’

The result is:

Line: 1

Line: 2

Line: 3

Line: 4

Line: 5

Practical AWK Use Cases with Real Examples

The following practical examples show how AWK can handle common data-processing tasks directly from the terminal.

Parsing /etc/passwd with AWK

The /etc/passwd file stores one user record per line and separates its fields with colons. Using AWK together with -F: provides a convenient way to extract individual values.

# Print the username and default shell for each user

awk -F: ‘{ print $1, $7 }’ /etc/passwd | head -5

On a typical Linux system, the output can look like:

root /bin/bash

daemon /usr/sbin/nologin

bin /usr/sbin/nologin

sys /usr/sbin/nologin

sync /bin/sync

To display only users whose configured shell is /bin/bash, use:

awk -F: ‘$7 == “/bin/bash” { print $1 }’ /etc/passwd

Processing CSV Files

Create the following example CSV file:

cat > employees.csv << ‘EOF’

Name,Department,Salary

Alice,Engineering,95000

Bob,Marketing,72000

Charlie,Engineering,88000

Diana,HR,65000

Eve,Marketing,78000

EOF

Display the name and department of every employee while excluding the header:

awk -F, ‘NR > 1 { print $1, “works in”, $2 }’ employees.csv

The output is:

Alice works in Engineering

Bob works in Marketing

Charlie works in Engineering

Diana works in HR

Eve works in Marketing

Summarizing System Log Files

Create a sample log file for the following examples:

cat > access.log << ‘EOF’

192.168.1.1 GET /index.html 200 1024

192.168.1.2 POST /login 404 512

192.168.1.3 GET /about.html 200 2048

192.168.1.1 GET /contact.html 500 256

192.168.1.4 GET /index.html 200 1024

EOF

Print every IP address associated with a 200 status code:

awk ‘$4 == 200 { print $1 }’ access.log

The output is:

192.168.1.1

192.168.1.3

192.168.1.4

Count the total number of 404 errors:

awk ‘$4 == 404 { count++ } END { print “404 errors:”, count }’ access.log

The result is:

404 errors: 1

Calculating Column Totals

To calculate the sum of all entries in the Cost field of file.txt while ignoring the first row, use:

awk ‘NR > 1 { sum += $4 } END { print “Total cost:”, sum }’ file.txt

The output is:

Total cost: 3895

You can also calculate an average:

awk -F, ‘NR > 1 { sum += $3; count++ } END { print “Average salary:”, sum / count }’ employees.csv

The result is:

Average salary: 79600

Saving AWK Output to a File

When you need to store the result generated by AWK, redirect the output with the > operator:

awk ‘/a/ { print $3 “\t” $4 }’ file.txt > output.txt

Use cat to confirm the contents:

cat output.txt

The output is:

USA 1299

Korea 299

USA 499

Japan 799

AWK vs. sed vs. grep: When to Use Each Tool

Choose grep when your main requirement is to determine whether a pattern occurs and return the matching lines. Choose sed when you need to replace or remove text. Use awk when your task involves individual fields, comparisons, calculations, or generating output based on values from several columns.

FAQs

1. What is the AWK command in Unix?

AWK is a powerful text-processing utility available on both Unix and Linux systems. It is intended for pattern scanning and reporting and can process files or streamed input one record at a time. For every input line, AWK automatically separates the record into fields according to a delimiter, with whitespace used by default. You can then apply custom rules, known as pattern-action pairs, to filter records, extract fields, perform calculations, create reports, and carry out other text-processing operations. Because AWK works directly with columns, it is especially useful when processing structured or delimited information such as logs, CSV files, and configuration data.

2. How do you run the AWK command in Linux?

AWK is normally executed with the following syntax:

awk ‘pattern { action }’ filename

For each line in the specified file, AWK evaluates the pattern and executes the associated action when the pattern matches. If the pattern is omitted, the action is performed for every line. Data can also be passed into AWK through a pipeline using command | awk '{ action }', which allows AWK to integrate easily into shell command chains. AWK can process several input files in one command, and options such as -F let you define a field separator while -f lets you load a larger AWK program from an external file. These capabilities make AWK suitable for many automation and text-processing tasks.

3. What does awk '{ print $1 }' do?

The command awk '{ print $1 }' displays the first field, or first column, from each input line. By default, AWK divides lines according to whitespace, so $1 represents the first word or value found after the line has been separated into fields.

For example, if an input line contains Name Age Country, this command displays only the Name field from that line. It is commonly used as a fast method for extracting the first column from structured data.

4. What does awk '{ print $2 }' do?

awk '{ print $2 }' extracts the second field from every line and displays it. For example, when processing a line such as John 30 Engineer, AWK prints 30 because $2 represents the second field. This provides a straightforward way to retrieve a particular column from tabular or whitespace-separated information.

5. What is the -F flag in AWK used for?

The -F option defines the field separator AWK uses when splitting each line into individual fields. AWK normally separates fields using whitespace, but -F allows you to specify another character or regular expression. For example, awk -F: '{ print $1 }' /etc/passwd tells AWK to treat the colon character (:) as the field delimiter. This is useful when processing files such as /etc/passwd, where fields are separated by colons. The option makes AWK adaptable to many types of structured data.

6. What are NR and NF in AWK?

NR and NF are two important variables built into AWK. NR means Number of Record and contains the current record or line number. Its value increases as AWK moves through each input line. This makes it useful for adding line numbers to output or applying actions only to selected lines.

NF means Number of Fields and contains the total number of fields in the current record. It is particularly useful when lines contain different numbers of columns or when you need to reference the final field through $NF. Both variables are commonly used to filter, format, and control how AWK processes input.

7. What is the difference between AWK and gawk?

gawk is the GNU implementation of the AWK programming language and is the standard AWK interpreter on many modern Linux distributions. Traditional AWK follows the POSIX specification, while gawk adds functionality such as stronger Unicode and multibyte-character handling, additional string and arithmetic functions, built-in networking capabilities, and expanded command-line options. Most programs created for standard AWK also run with gawk, while the additional GNU features make more advanced text-processing tasks possible.

8. When should you use AWK instead of Python or sed?

AWK is well suited to quick one-line operations involving field extraction, column-based filtering, and inline text manipulation, particularly within shell pipelines or when processing structured tabular information. When a task requires advanced programming structures, complicated data transformations, or integration with other systems, Python can be a better option because of its extensive libraries and maintainability for larger programs. sed, by comparison, is particularly effective for simple substitutions and other line-by-line editing operations on text streams or files. In general, use AWK for fast field-oriented processing, Python for more complex scripting, and sed for straightforward text replacements.

Conclusion

AWK is a relatively simple yet powerful utility for processing text on Linux and Unix systems. Whether you need to extract a particular column from a file, calculate totals, analyze a log file, or decide whether AWK, grep, or sed is the most suitable tool for a task, the examples in this tutorial provide a strong foundation for working with AWK.