Mastering Curl: Complete Guide to HTTP Requests and API Testing

Curl is a versatile command-line tool and the underlying library (libcurl) for transferring data over numerous network protocols, such as HTTP, HTTPS, FTP, FTPS, SCP, SFTP, LDAP, MQTT, and more. It provides features like proxy handling, user authentication, cookies, file uploads, and SSL connections. This makes it an essential utility for developers, system administrators, and anyone interacting with web APIs or network-based services.

This guide demonstrates how to use curl to carry out various HTTP requests directly from the terminal. You will discover how to download files, specify custom headers, work with different HTTP methods, manage redirects, send JSON payloads, authenticate with servers, and troubleshoot issues in verbose mode. Whether you are testing APIs or automating network operations, curl is a valuable asset in your workflow.

Quick Reference

If you only need a concise list of common curl commands, here is a quick overview:

# Basic GET request
$ curl https://example.com

# Save output to file
$ curl -o page.html https://example.com

# Follow redirects
$ curl -L http://example.com

# Send POST form data
$ curl -d "name=John&age=30" -X POST https://example.com

# Send JSON payload
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://example.com

# Add custom header
$ curl -H "Authorization: Bearer " https://api.example.com

# Basic auth
$ curl -u user:pass https://example.com

Using Curl

This section expands on the quick reference and explains essential curl operations in detail, including file downloading, protocol handling, and request formatting. These commands are frequently used for checking connectivity or retrieving public web content.

The fundamental curl command format is:

$ curl [options] 

curl Command Options

Detailed Usage

The following sections describe each curl command in greater depth. You will find examples, syntax explanations, and usage notes to help you understand how each option works and when to apply it. These explanations complement the quick reference above for those who prefer step-by-step guidance.

Download a Webpage

Download a webpage and display its source code in the terminal.

Command Syntax

$ curl 

Example

$ curl https://www.example.com/index.html

Use FTP Protocol

Retrieve a file using FTP.

Command Syntax

$ curl ftp:///

Example

$ curl ftp://ftp.example.com/public/readme.txt

Save Output to a File

Save the downloaded content to a local file rather than printing it to the terminal.

Command Syntax

$ curl -o  
$ curl -O 

-o: Save output with a custom filename.

-O: Save output using the remote filename.

Example

$ curl -o localcopy.html https://example.com/index.html
$ curl -O https://example.com/index.html

Resume an Interrupted Download

If a download was interrupted (for example due to network problems), this option resumes it from where it stopped instead of starting again.

Command Syntax

$ curl -C -O 

-C -: Resume download.

-O: Save using remote file name.

Example

$ curl -C -O https://example.com/bigfile.zip

Enable Verbose Output

Enable verbose output to debug request and response communication.

Command Syntax

$ curl -v 

-v: Verbose mode.

Example

$ curl -v https://example.com

Making HTTP Requests

Curl provides a variety of options to interact with web services. This section explains how to inspect headers, follow redirects, change HTTP methods, and customize request headers.

Get Only Response Headers

Fetch only the response headers of a URL.

Command Syntax

$ curl -I 

-I or --head: Fetch headers only.

Example

$ curl -I https://example.com

Follow Redirects

Enable following of HTTP 3xx redirects.

Command Syntax

$ curl -L 

-L: Follow redirects.

Example

$ curl -L http://example.com

Change the HTTP Method

Set a custom HTTP method using -X.

Command Syntax

$ curl -X [METHOD] 

-X: Specify HTTP method (e.g., GET, POST, DELETE).

Example

$ curl -X DELETE https://example.com/resource/123

Add Custom Headers

Use the -H flag to add one or more HTTP headers.

Command Syntax

$ curl -H "Header-Name: value" 

Example

$ curl -H "User-Agent: Mozilla/5.0" -H "Cache-Control: no-cache" https://example.com

Set a Custom User-Agent

Use --user-agent to define the request’s User-Agent string.

Command Syntax

$ curl --user-agent "[Agent String]" 

Example

$ curl --user-agent "Mozilla/5.0 (Macintosh)" https://example.com

Send Form or JSON Data (POST)

Use curl to send form-encoded or JSON data in HTTP POST requests. This is essential for interacting with REST APIs or submitting data to a server.

Command Syntax

$ curl -d "<key1=value1&key2=value2>" -X POST 
$ curl -X POST -H "Content-Type: application/json" -d '' 
</key1=value1&key2=value2>

Example

$ curl -d "name=John&age=30" -X POST https://example.com
$ curl -X POST -H "Content-Type: application/json" -d '{"name":"John"}' https://example.com

Basic Authentication

When accessing resources protected by HTTP Basic Auth, use the -u option to include your credentials directly in the request.

Command Syntax

$ curl -u : 

Example

$ curl -u admin:secret https://example.com

Advanced Usage and Tips

This section covers advanced curl patterns commonly used in scripts, automation, and API integrations. These techniques are useful for scripting, performance testing, secure authentication, and ensuring API reliability.

Exit on HTTP Errors (--fail)

Make curl exit silently with a non-zero code when HTTP errors like 404 or 500 occur. This avoids printing error pages.

Command Syntax

$ curl --fail 

--fail: Exit silently on HTTP errors (4xx, 5xx).

Example

$ curl --fail https://example.com/missing-file

Use this in scripts to halt execution when a URL is unreachable or responds with an error.

Measure HTTP Request Time (-w)

Print the total time taken to complete a request. Useful for latency measurement and benchmarking.

Command Syntax

$ curl -w "%{time_total}\n" -o /dev/null -s 

-w: Custom output after completion.

%{time_total}: Total transfer time.

-o /dev/null: Discard body.

-s: Silent mode to suppress progress.

Example

$ curl -w "%{time_total}\n" -o /dev/null -s https://example.com

This helps measure API response time or network latency.

Upload Files (-F)

Upload files using a multipart form submission.

Command Syntax

$ curl -F "key=@filename.ext" 

-F: Send form field with file.

@filename.ext: Reads the file from local disk.

Example

$ curl -F "file=@report.pdf" https://example.com/upload

Use -F for uploading files as part of a form. Use --data-binary if you want to send raw files.

Authenticate with Bearer Token

Use bearer tokens to access secured APIs (for example OAuth2 access tokens).

Command Syntax

$ curl -H "Authorization: Bearer " 

-H: Set a custom header.

Authorization: Bearer: Required for most modern APIs.

Example

$ curl -H "Authorization: Bearer abc123" https://api.example.com/data

This is required when accessing private endpoints on services like GitHub, Stripe, or cloud APIs.

Quiet or Debug Output (-s, -v)

Adjust output visibility depending on your need.

Silent Mode Syntax

$ curl -s 

-s: Suppress progress bar and errors.

Verbose Mode Syntax

$ curl -v 

-v: Show full request and response details.

Example

$ curl -s https://example.com

$ curl -v https://example.com

Use -s for scripts and logging. Use -v when debugging failures or testing headers.

Conclusion

Curl is an essential tool for developers and system administrators who need to interact with web servers, APIs, and remote resources from the command line. This guide covered both quick-use examples and detailed explanations for common curl operations such as downloading files, handling headers, sending data, and authenticating requests.

For further exploration, refer to the official curl documentation or man page to discover protocol-specific features, scripting tricks, and advanced usage patterns.