How to Use AWK to Manipulate Text in Linux

Linux utilities commonly follow the Unix design philosophy: tools should remain small, use plain-text input and output, and work well together as modular components. This approach has given Linux powerful text-processing utilities such as sed and awk.

awk functions both as a programming language and as a text-processing tool. This makes it particularly useful for structured, line-based information such as logs, tables, and basic delimited files. This guide begins with the fundamental pattern-and-action syntax, compares awk with sed for typical text-processing tasks, and then covers field-based filtering, built-in variables, and formatted output.

You will also learn how associative arrays can count, group, and aggregate values during a single pass, how awk can be combined with other commands in pipelines, and how it can be included in shell scripts for reusable automation. Practical parsing examples and frequently asked questions provide additional guidance for working with awk.

Key Takeaways

• Use the pattern-and-action model in awk to identify relevant lines and transform only matching records.
• Work with input as fields or columns by default. Variables such as $1 and $2 represent individual fields, while $0 represents the complete record.
• Define FS and OFS when working with delimited data such as /etc/passwd or CSV-like files so input and output remain consistent.
• Use BEGIN for initialization tasks such as configuring separators or displaying headings, and use END for summaries such as totals and aggregated results.
• Variables including NR, NF, FILENAME, and FNR help create record-aware logic, troubleshoot unexpected input, and process several files reliably.
• Associative arrays are useful for fast counting, grouping, summing, and other single-pass aggregation tasks without requiring a separate sort or uniq operation.
• Use -v when transferring shell values into awk so spaces, quoting, and special characters are handled more safely.
• Use awk for streaming, line-oriented text processing and lightweight reports, but consider another tool when parsing rules become more complicated, such as CSV files containing quoted commas.

AWK vs sed: When Should You Use Each?

awk and sed are standard Unix text-processing utilities and are frequently discussed together because both operate on text streams. Their purposes are different, however, and the better choice depends on the type of processing you need to perform.

In general, sed is a stream editor designed for straightforward transformations of lines, whereas awk is a pattern-scanning and data-processing language that is particularly effective with structured or column-oriented information.

Key Differences

When to Use sed

Choose sed when you need simple transformations that apply to individual lines of text. It is particularly effective for quickly changing content without having to interpret the internal field structure of the data.

For example, the following command replaces every occurrence of the word “error” with “warning” in a file:

sed 's/error/warning/g' file.txt

In general, sed is appropriate when you need to:

• Perform search-and-replace operations.
• Remove or insert particular lines.
• Apply basic transformations to complete lines of text.

Because sed processes text one line at a time and does not inherently interpret columns or fields, it is usually less convenient for structured datasets.

When to Use awk

Use awk when your work requires extracting, examining, or transforming structured information. Unlike sed, awk understands fields and can work directly with columns in an input file.

For example, you can print only the first and third columns of a file with:

awk '{print $1, $3}' file.txt

awk is especially useful when you need to:

• Retrieve particular columns from tabular information.
• Filter records according to conditions.
• Carry out calculations and aggregations.
• Create formatted output or reports.

Since awk provides programming features such as variables, conditional statements, and loops, it can manage more sophisticated data-processing operations than sed.

Quick Decision Guide

If you are uncertain about which utility to choose, these general guidelines can help:

• Use sed for basic line-oriented text transformations that do not depend on fields or structured data.
• Use awk when working with structured information, individual columns, conditional expressions, or data-processing logic.

Example: Comparing Approaches

Assume a file contains several space-separated fields on each line and you want to retrieve only the first column.

With awk, the command is direct:

awk '{print $1}' file.txt

Performing the equivalent operation with sed requires a more involved regular expression:

sed 's/^\([^ ]*\).*/\1/' file.txt

Both commands return the same information, but the awk version is easier to interpret because it directly represents the structure of the input. This illustrates the main distinction between the tools: awk works naturally with fields, while sed works primarily with text patterns.

Both utilities are powerful, but they are optimized for different situations. sed works well for fast, uncomplicated text modifications, while awk is better suited to structured data and more advanced processing logic. Knowing the difference makes it easier to select the most efficient utility for a particular task.

Basic AWK Syntax

The awk command is included by default on modern Linux systems, so additional installation is generally unnecessary before you begin using it.

awk is most effective with text files that follow a predictable format. It works especially well with tabular information, reading the input line by line until the complete file has been processed.

Whitespace, including spaces and tabs, is used as the default field separator. Many Linux configuration files follow formats that can be processed conveniently in this way.

The basic structure of an awk command is:

awk '/search_pattern/ { action_to_take_on_matches; another_action; }' file_to_parse

You can leave out either the search expression or the action section. When an action is omitted, the default action is to print each matching line.

If there is no search expression, awk performs the specified action on every input line.

If both elements are supplied, awk evaluates the search expression against the current line and runs the specified actions whenever a match occurs.

In its simplest form, awk can be used similarly to cat to display every line from a text file.

Create a file named favorite_food.txt containing favorite foods and names:

echo "carrot sandy
wasabi luke
sandwich brian
salad ryan
spaghetti jessica" > favorite_food.txt

You can then display the complete file with awk:

awk '{print}' favorite_food.txt

The output is:

carrot sandy
wasabi luke
sandwich brian
salad ryan
spaghetti jessica

This basic example does not yet take advantage of awk‘s filtering abilities. To display only lines containing the text “sand”, use:

awk '/sand/' favorite_food.txt

The result is:

carrot sandy
sandwich brian

awk now displays only the records containing the sequence “sand”.

Regular expressions allow you to target a more specific position within the text. To match only a line beginning with “sand”, use the ^sand expression:

awk '/^sand/' favorite_food.txt

Only the matching line appears:

sandwich brian

The action portion of an awk expression can determine which information is displayed. For example, the following command prints only the first field:

awk '/^sand/ {print $1;}' favorite_food.txt

The result is:

sandwich

Each whitespace-delimited column can be accessed with a variable corresponding to its position. The first field is $1, the second is $2, and the complete current line can be referenced with $0.

Internal Variables and Expanded AWK Format

While reading a file, awk maintains several built-in variables containing information about the current input and processing state.

Important internal variables include:

• FILENAME: Contains the name of the input file currently being processed.
• FNR: Contains the record number within the current input file. When several files are processed, it restarts for each file rather than representing the overall record count.
• FS: Defines the separator used to divide each input record into fields. Whitespace is the default.
• NF: Contains the number of fields in the current record.
• NR: Contains the overall number of the current record.
• OFS: Defines the separator used between output fields. Whitespace is used by default.
• ORS: Defines the separator between output records. The default is a newline character.
• RS: Defines the separator between input records. A newline character is used by default.

You can modify these variables to suit the format of the data you are processing. Configuration is often performed during the initialization stage.

For example, the following command displays every line together with its record number:

awk '{print NR, $0}' file.txt

To display the number of fields contained in each record, use:

awk '{print "Fields:", NF}' file.txt

awk also supports optional BEGIN and END blocks. Commands placed inside BEGIN execute before input processing begins, while commands inside END execute after all records have been processed.

The expanded syntax can therefore look like this:

awk 'BEGIN { action; }
/search/ { action; }
END { action; }' input_file

The BEGIN and END keywords act as special conditions. One applies before the document is processed, while the other applies after processing finishes.

This makes the BEGIN block a convenient location for changing built-in variables. For example, /etc/passwd uses colons (:) instead of whitespace to separate its fields.

To print the first field from this file, use:

awk 'BEGIN { FS=":"; }
{ print $1; }' /etc/passwd

The output begins with entries similar to:

root
daemon
bin
sys
sync
games
man
. . .

The BEGIN and END blocks can also add descriptive information around the fields being displayed. The following command formats selected information from the file as a tab-separated table:

awk 'BEGIN { FS=":"; print "User\t\tUID\t\tGID\t\tHome\t\tShell\n--------------"; }
{print $1,"\t\t",$3,"\t\t",$4,"\t\t",$6,"\t\t",$7;}
END { print "---------\nFile Complete" }' /etc/passwd

The resulting output is similar to:

User      UID       GID       Home          Shell
--------------
root      0         0         /root         /bin/bash
daemon    1         1         /usr/sbin     /bin/sh
bin       2         2         /bin          /bin/sh
sys       3         3         /dev          /bin/sh
sync      4         65534     /bin          /bin/sync
. . .
---------
File Complete

These features make it possible to format processed information in a readable and organized way.

Every section in the expanded form is optional. Even the primary action section can be omitted if another section is present. For example:

awk 'BEGIN { print "We can use awk like the echo command"; }'

This produces:

We can use awk like the echo command

The next step is to examine how matching can be performed against individual fields.

Field Searching and Compound Expressions

In an earlier example, the line in favorite_food.txt beginning with “sand” was selected. That was straightforward because the expression matched the beginning of the entire input record.

You can also check whether a pattern occurs at the beginning of a specific field instead of at the beginning of the entire line.

Create another version of favorite_food.txt that includes a numeric item identifier before every food and name:

echo "1 carrot sandy
2 wasabi luke
3 sandwich brian
4 salad ryan
5 spaghetti jessica" > favorite_food.txt

If you want to locate foods containing “sa”, you might initially try:

awk '/sa/' favorite_food.txt

This displays every line containing the character sequence “sa”:

1 carrot sandy
2 wasabi luke
3 sandwich brian
4 salad ryan

The expression matches “sa” anywhere in a record. Consequently, it also finds words such as “wasabi”, where the sequence occurs in the middle, and “sandy”, even though that value is not in the field being examined. If the goal is to find only foods beginning with “sa” in the second column, the field must be included in the expression.

Use the following command to match only values beginning with “sa” in the second field:

awk '$2 ~ /^sa/' favorite_food.txt

This restricts pattern matching to the beginning of the second column. The $2 ~ expression tells awk to apply the regular expression specifically to the second field.

The result is:

3 sandwich brian
4 salad ryan

You can invert a regular-expression match by placing an exclamation mark before the tilde. The following expression selects lines where the food does not begin with “sa”:

awk '$2 !~ /^sa/' favorite_food.txt

The output becomes:

1 carrot sandy
2 wasabi luke
5 spaghetti jessica

To select records where the food does not begin with “sa” and the item number is below 5, combine the conditions:

awk '$2 !~ /^sa/ && $1 < 5' favorite_food.txt

This introduces the && operator, which lets you require several conditions to be true at the same time. Here, the pattern condition is combined with a second test that checks whether the first field contains a value below 5.

The result is:

1 carrot sandy
2 wasabi luke

In addition to processing files directly, awk can work with output produced by other commands.

Using Associative Arrays for Data Aggregation

Up to this point, awk has mainly been used to filter and display information. Another important feature is support for associative arrays, which allow values to be stored while input is scanned.

Unlike conventional arrays that use numeric indexes, associative arrays in awk can use strings as keys. They are therefore well suited to counting, grouping, and summarizing records because a value from a field, such as a username or department, can become the array key.

Counting Occurrences

A frequent use for associative arrays is counting how many times a particular value occurs. A counter can be increased for every key while lines are processed, and the completed counts can then be displayed from an END block.

Consider a file named fruits.txt:

echo "apple
banana
apple
orange
banana
apple" > fruits.txt

Count the number of occurrences of every fruit with:

awk '{count[$1]++} END {for (item in count) print item, count[item]}' fruits.txt

The output will be similar to:

apple 3
banana 2
orange 1

In this expression:

• count[$1]++ increases a counter associated with each unique value in the first field.
• $1 acts as the key of the associative array.
• The END section outputs the aggregated results after all input has been read.

The order of keys produced by for (item in count) can vary depending on the awk implementation. When sorted results are required, the output can be sent to sort.

Grouping Data by Field

Associative arrays can also organize related records into groups. Instead of keeping only a numeric counter, you can construct a string or another value associated with each key as records are processed.

For example, create employees.txt:

echo "HR Alice
IT Bob
HR Charlie
IT David
Finance Emma" > employees.txt

Employees can then be grouped according to department:

awk '{group[$1] = (group[$1] ? group[$1] " " : "") $2} END {for (dept in group) print dept ": " group[dept]}' employees.txt

Example output:

HR: Alice Charlie
IT: Bob David
Finance: Emma

Here:

• $1, representing the department, is used as the key.
• $2, representing the employee name, is appended to the value stored for the relevant group.

The conditional expression prevents an unnecessary leading space from appearing before the first employee associated with each department.

Summing Values

Associative arrays can also maintain running totals. This pattern is useful when one field contains a label and another contains a numeric amount.

For example, consider a file named sales.txt:

echo "Alice 100
Bob 200
Alice 150
Bob 50" > sales.txt

Calculate the total amount associated with each person using:

awk '{sum[$1] += $2} END {for (name in sum) print name, sum[name]}' sales.txt

The result is:

Alice 250
Bob 250

In this example:

• $1 serves as the key containing the name.
• $2 is added to the running total associated with that key.

Finding the Maximum Value

Associative arrays can also be used when searching for maximum or minimum values. A common method is to create an aggregate array first and then iterate through the completed array to determine which entry has the largest value.

awk '{sum[$1] += $2} END {
    for (name in sum) {
        if (top == "" || sum[name] > max) { max = sum[name]; top = name }
    }
    print "Top performer:", top, max
}' sales.txt

This technique allows useful calculations and insights to be produced directly within awk without depending on additional utilities.

Why Associative Arrays Matter

Associative arrays extend awk beyond basic filtering and turn it into a lightweight data-processing engine. They allow the program to retain state while records are being read and then produce summarized information when processing is complete.

Associative arrays can be used to:

• Create frequency distributions.
• Group and reorganize records.
• Calculate totals, counts, and similar aggregates.
• Perform basic command-line analytics.

These techniques are particularly useful for logs, CSV-style files, and machine-generated data.

Processing Output from Other Programs

awk does not require a filename as its input. It can also process the output produced by another program. For example, it can extract an IPv4 address from the output of the ip command.

The ip a command provides IP addresses, broadcast addresses, and additional details about network interfaces. To display information for an interface named eth0, use:

ip a s eth0

Example output:

2571: eth0@if2572: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:11:00:0b brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.11/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

You can match the inet line and print only the IP address with:

ip a s eth0 | awk -F '[\/ ]+' '/inet / {print $3}'

The -F option configures awk to use forward slashes or spaces as delimiters according to the [\/ ]+ regular expression. This separates a line such as inet 172.17.0.11/16 into individual fields. Because leading spaces are also considered during splitting, the address is found in the third field. Consecutive spaces are handled as one separator in this situation.

The output is:

172.17.0.11

Similar techniques allow awk to search, filter, and parse the output generated by many other command-line utilities.

Using AWK in Shell Scripts and Automation

So far, awk has mainly been shown as an individual command. It becomes even more useful when incorporated into shell scripts that automate repetitive text-processing operations.

Combining awk with Bash makes it possible to create lightweight automation that parses logs, extracts metrics, and generates reports without introducing additional dependencies.

Embedding AWK in a Bash Script

An awk command can be placed directly inside a shell script just like any other command. This is helpful when the same extraction logic needs to be reused in multiple locations or executed regularly.

For example, create a script named extract_users.sh:

#!/bin/bash

awk -F ":" '{print $1}' /etc/passwd

Make the script executable and execute it:

chmod +x extract_users.sh
./extract_users.sh

The script retrieves and displays all usernames from /etc/passwd. Although the example is simple, it illustrates how awk logic can be incorporated into reusable shell scripts.

It also demonstrates a frequently used pattern: -F ":" establishes the field delimiter, while print $1 outputs the first field from each record.

Passing Variables from Bash to AWK

Shell variables can be transferred into awk with the -v option. This makes it possible to create scripts whose behavior depends on values determined elsewhere in the shell code.

#!/bin/bash

threshold=100

awk -v limit="$threshold" '$2 > limit {print $1, $2}' sales.txt

In this example:

• The Bash variable threshold is made available inside awk under the name limit.
• awk prints only records whose second field is greater than the specified threshold.

This method lets shell-script logic control how an awk program behaves.

As a general practice, using -v is safer than inserting shell variables directly into the awk program through string concatenation.

Processing Files in Bulk

Shell scripts frequently need to process more than one file. Loops can be combined with awk to handle groups of input files.

#!/bin/bash

for file in *.log; do
    echo "Processing $file"
    awk '/ERROR/ {print $0}' "$file"
done

This script goes through every .log file in the current directory, extracts records containing “ERROR”, and prints the matches for each file.

When Bash is being used and the loop should do nothing if no files match *.log, the nullglob option can be enabled with a command such as shopt -s nullglob. This causes the pattern to expand to nothing rather than remaining as the literal text *.log.

Automating Log Analysis

Log analysis is a common practical use for this type of automation.

For example, the following script counts how many error records appear in a log file:

#!/bin/bash

logfile="app.log"

awk '/ERROR/ {count++} END {print "Total errors:", count+0}' "$logfile"

The script reads the log, increases a counter for every record containing “ERROR”, and prints the total from the END section.

Adding 0 to count ensures that the script displays 0 when no records match.

This approach can be extended to:

• Trigger alerts.
• Create daily summaries.
• Send results to monitoring systems.

Combining AWK with Other Commands

awk is commonly placed inside pipelines together with other Unix utilities to create flexible command-line workflows.

grep "404" access.log | awk '{print $1}' | sort | uniq -c

This pipeline identifies HTTP 404 records, extracts the IP address from the first field, sorts the addresses, and counts how often each one occurs.

Although awk can often replace some stages in such a pipeline, using several specialized utilities together can make a script clearer and easier to maintain.

Why Use AWK in Automation?

Using awk in shell scripts can help you:

• Remove repetitive manual data-processing work.
• Create workflows that can be executed consistently.
• Extract useful information from system-generated data quickly.
• Avoid using a larger scripting language for relatively simple processing jobs.

Because awk is available on nearly all Unix-like operating systems, scripts that depend on it are generally portable between environments.

Real-World Examples: Log Parsing and Data Extraction

The previous examples explain the mechanics of awk, but the utility becomes especially valuable when solving practical data-processing problems. Administrators and developers regularly use it to examine logs, retrieve structured fields, and produce quick reports from the command line.

The following examples demonstrate several common situations where awk can be applied.

Parsing Apache Access Logs

Web server logs are a common source of structured textual information. In many standard Apache configurations, access logs use the “combined” format. In this format, the client IP address appears near the beginning of each record, while the HTTP response status is stored later in the line.

A typical Apache access-log entry might look like this:

127.0.0.1 - - [10/Oct/2023:13:55:36 +0000] "GET /index.html HTTP/1.1" 200 2326

To retrieve the IP address and HTTP status code for every request, use:

awk '{print $1, $9}' access.log

With the log structure shown above, $1 contains the client IP address and $9 contains the HTTP status code.

To count the number of requests that resulted in a 404 response:

awk '$9 == 404 {count++} END {print "404 errors:", count+0}' access.log

The expression count+0 guarantees that 0 is printed when no matching records are found.

Extracting Data from CSV Files

Although whitespace is the default field separator in awk, another separator can be supplied easily, making simple CSV data straightforward to process.

Consider the following data.csv file:

name,age,city
Alice,30,New York
Bob,25,London
Charlie,35,Sydney

To display only names and cities:

awk -F ',' '{print $1, $3}' data.csv

This technique assumes that the CSV file does not contain quoted fields with embedded commas. When commas can appear inside quoted values, a parser designed specifically for CSV data is required.

To display records where the age value is greater than 30:

awk -F ',' '$2 > 30 {print $1, $2}' data.csv

This example shows how structured information can be queried with awk without requiring a database.

Analyzing System Files

Many Linux system files use consistent formats, which makes them suitable for processing with awk.

For example, list accounts whose UID is greater than 1000 from /etc/passwd:

awk -F ':' '$3 > 1000 {print $1, $3}' /etc/passwd

To count how many accounts meet the same condition:

awk -F ':' '$3 > 1000 {count++} END {print count+0}' /etc/passwd

The expression prints 0 if no user accounts satisfy the condition.

Summarizing Log Data

Several awk features can be combined to create compact log summaries.

For example, count how many requests were received from each IP address:

awk '{count[$1]++} END {for (ip in count) print ip, count[ip]}' access.log

This creates a frequency distribution showing the number of requests associated with each client address. The for (ip in count) loop does not guarantee a particular order, so the results can be piped through sort if consistent ordering is necessary.

Applying awk to situations like these makes it possible to automate tasks that might otherwise require more elaborate tools or manual processing.

AWK FAQs

1. What Is the Difference Between AWK, gawk, and mawk?

awk refers to the language and command interface defined by POSIX. gawk, or GNU Awk, is widely used on Linux systems and provides additional functionality beyond the POSIX specification. mawk is another implementation that frequently focuses on performance and low resource usage, although it may not provide all GNU-specific extensions available in gawk.

2. How Do I Specify a Custom Field Separator in AWK?

The -F option specifies how awk should divide an input line into fields. For example, awk -F':' '{print $1}' /etc/passwd uses a colon as the field separator and prints the first field, which contains the username.

3. What Is the Difference Between AWK and sed?

sed is a stream editor designed primarily for line-oriented operations such as substitutions, deletions, and insertions. awk is a pattern-processing language intended for field-oriented data processing, conditional operations, and basic calculations. In general, awk is more convenient for column-based information and reports, while sed is often more direct for simple text replacements.

4. How Do I Use AWK to Count the Number of Occurrences of a Word or Value?

An associative array can be used as a counter. Increase the value associated with each key while reading the input and display the results from an END block. For example, awk '{count[$1]++} END {for (w in count) print w, count[w]}' file.txt counts each unique value found in the first field.

5. Can AWK Handle Multi-Line Records?

Yes. Setting the record separator RS to an empty string with RS="" causes awk to treat blocks separated by blank lines as individual records. This can be useful when input is organized into paragraphs or multi-line blocks rather than one record per line.

6. How Do I Pass a Shell Variable into an AWK Command?

The -v option can assign a shell value to an awk variable before processing starts. For example, threshold=100; awk -v t="$threshold" '$3 > t {print $0}' file.txt makes the shell value available within awk through the variable t.

7. Is AWK Suitable for Large File Processing?

Yes. awk normally processes one record at a time instead of loading the entire input file into memory. This makes it efficient for large datasets. More complex multi-stage processing may eventually be easier to maintain with another language such as Python, but awk remains highly effective for fast extraction, filtering, and aggregation.

8. How Do I Write and Run an AWK Program Stored in a File?

An awk program can be saved in a regular text file such as script.awk and executed with awk -f script.awk input.txt. This approach is usually easier to read and maintain than a long one-line command when the program includes multiple rules, blocks, or functions.

Conclusion

You should now understand the basic techniques for using awk to filter, format, and selectively display text from both files and command output. Beyond simple printing, awk provides internal variables and BEGIN/END sections for structured processing, associative arrays for single-pass aggregation, and integration with shell scripts for reusable automation. When working with practical inputs such as logs and delimited files, the distinction between awk and sed can help you decide whether the task is primarily field-oriented data processing or line-oriented text editing.

For a more detailed introduction to awk, you can read the free public-domain book written by its creators, which explores the language in greater depth.