Using Redis® Queues for Sequential Job Processing in PHP
The Redis® queue library enables you to manage a sequential stream of jobs processed in a first-come, first-served order. This method is particularly useful in web development when you need to prevent users from waiting while the system performs time-consuming tasks. For example, in a high-traffic e-commerce platform, operations like payment validation and address checks can be handled in the background using queues and worker processes. In simple terms, if your application needs to maintain order in task execution but can’t afford to delay user interaction, implementing a Redis® queue—such as for a file conversion system—can be a practical solution.
This walkthrough explains how to leverage Redis® commands RPUSH and LPOP alongside PHP and MySQL to manage user sign-ups within a MySQL database on an Ubuntu 20.04 server.
System Requirements
Before you begin, ensure you have the following resources ready:
- An Ubuntu 20.04-based server
- User access with sudo privileges
- A complete LAMP (Linux, Apache, MySQL, PHP) setup
- An active Redis® server
Installing the php-redis Module
Start by accessing your server. Next, update your system’s package index and install the php-redis package. This module enables integration of Redis® features within your PHP scripts.
$ sudo apt update
$ sudo apt install -y php-redis
After installing the module, restart the Apache service to make the new configurations take effect.
$ sudo systemctl restart apache2
Setting Up the test_db Database
In this section of the tutorial, you’ll temporarily store customer sign-up data in Redis®, and later transfer it to a MySQL database for long-term storage.
Begin by accessing your MySQL server using the root account:
$ sudo mysql -u root -p
After entering your MySQL root password and hitting Enter, create a new database named test_db:
mysql> CREATE DATABASE test_db;
Next, define a MySQL user called test_db_user and assign a strong password (replace EXAMPLE_PASSWORD with your actual value). This user will be used to interact with MySQL from your PHP application.
mysql> CREATE USER 'test_db_user'@'localhost' IDENTIFIED WITH mysql_native_password BY 'EXAMPLE_PASSWORD';
mysql> GRANT ALL PRIVILEGES ON test_db.* TO 'test_db_user'@'localhost';
mysql> FLUSH PRIVILEGES;
If you’re using MariaDB instead of MySQL, you should modify the user creation step as follows:
MariaDB> GRANT ALL PRIVILEGES on test_db.* TO 'test_db_user'@'localhost' identified by 'EXAMPLE_PASSWORD';
Now, switch the active database context to test_db using the command below:
mysql> USE test_db;
Proceed to create a customers table using this SQL command:
mysql> CREATE TABLE customers (
customer_id BIGINT NOT NULL AUTO_INCREMENT PRIMARY KEY,
first_name VARCHAR(50),
last_name VARCHAR(50),
email_address VARCHAR(255)
) ENGINE = InnoDB;
Do not insert any records into the customers table just yet. You will fill it later using a worker that retrieves data from Redis®. When finished, exit the MySQL interface:
mysql> QUIT;
Creating the PHP Script for Redis® Queue
To start, create a new PHP file named redis_queue.php in your web server’s root directory using nano. This script will collect customer details through POST requests and add them to the Redis® queue.
$ sudo nano /var/www/html/redis_queue.php
Insert the following content into the file:
<?php
try {
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
$data = [];
$data = [
'first_name' => $_POST['first_name'],
'last_name' => $_POST['last_name'],
'email_address' => $_POST['email_address']
];
$redis->rpush("customers_register", json_encode($data));
echo "Customer details accepted successfully.\n";
} catch (Exception $e) {
echo $e->getMessage();
}
This script connects to the Redis® server on localhost via port 6379. It collects the customer’s submitted data from the $_POST array and structures it into an associative array. The array is then encoded into JSON and added to the Redis® list named customers_register using the rpush method. A confirmation message is displayed upon successful queuing of the data.
Submitting Customer Records to the Redis® Queue
Typically, users would enter their details using HTML forms. However, for this tutorial, you’ll use the curl command-line tool to simulate user data submission directly to http://localhost/redis_queue.php.
Execute the following commands one at a time to queue three separate customer records:
$ curl --data "first_name=JOHN&last_name=DOE&email_address=john_doe@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=MARY&last_name=SMITH&email_address=mary_smith@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=ROE&last_name=STEVE&email_address=roe_steve@example.com" http://localhost/redis_queue.php
Each command should return the following message as confirmation:
Customer details accepted successfully.
This output verifies that the customer data was successfully queued by the Redis® server. In the next step, you’ll develop a background worker to process these entries.
Building a PHP Script for Redis® Queue
To begin, create a new PHP file called redis_queue.php inside your server’s root directory. This script will handle POST requests containing customer registration details.
$ sudo nano /var/www/html/redis_queue.php
Now insert the following PHP code into the file:
<?php
try {
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
$data = [];
$data = [
'first_name' => $_POST['first_name'],
'last_name' => $_POST['last_name'],
'email_address' => $_POST['email_address']
];
$redis->rpush("customers_register", json_encode($data));
echo "Customer details accepted successfully.\n";
} catch (Exception $e) {
echo $e->getMessage();
}
This script establishes a connection to the Redis® instance running locally on port 6379. It then retrieves input data from the $_POST array and organizes it into an associative array called $data. After encoding this array into a JSON string, the script pushes it to the customers_register list in Redis® using the rpush method. If successful, the script outputs a confirmation message.
Submitting Customer Data to the Queue
Although HTML forms would typically handle user registration in a live environment, for this tutorial, you’ll use the curl command to simulate form submissions. The data will be sent to the redis_queue.php script located at http://localhost/redis_queue.php.
Execute the following commands individually to queue three different customer records:
$ curl --data "first_name=JOHN&last_name=DOE&email_address=john_doe@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=MARY&last_name=SMITH&email_address=mary_smith@example.com" http://localhost/redis_queue.php
$ curl --data "first_name=ROE&last_name=STEVE&email_address=roe_steve@example.com" http://localhost/redis_queue.php
Each command should generate the following output:
Customer details accepted successfully.
This confirmation message indicates that the Redis® queue is successfully receiving and storing the customer data. Next, you’ll implement a worker to process the entries in the queue.
Persisting Redis® Data to a MySQL Database
Now you’ll create a background process that removes customer entries from the Redis® queue and stores them permanently in a MySQL database. This will be achieved using Redis®’s LPOP function, which follows the FIFO (First-In, First-Out) model.
Start by creating a new PHP file named redis_worker.php in the root directory of your web server:
$ sudo nano /var/www/html/redis_worker.php
Add the following PHP code into the file:
<?php
try {
$redis = new Redis();
$redis->connect('127.0.0.1', 6379);
$data = $redis->lpop('customers_register');
$data = json_decode($data, true);
if (!empty($data)) {
$db_name = 'test_db';
$db_user = 'test_db_user';
$db_password = 'EXAMPLE_PASSWORD';
$db_host = 'localhost';
$pdo = new PDO('mysql:host=' . $db_host . '; dbname=' . $db_name, $db_user, $db_password);
$pdo->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$pdo->setAttribute(PDO::ATTR_EMULATE_PREPARES, false);
$sql = 'insert into customers
(
first_name,
last_name,
email_address
)
values
(
:first_name,
:last_name,
:email_address
)';
$stmt = $pdo->prepare($sql);
$stmt->execute($data);
echo $data['first_name'] . " " . $data['last_name'] . "'s details saved to database.";
}
} catch (Exception $e) {
echo $e->getMessage();
}
This script establishes a connection with the local Redis® instance and retrieves the oldest customer entry from the customers_register queue using the lpop method. The data, initially encoded as JSON, is decoded into an array and inserted into the customers table using PHP Data Objects (PDO).
The SQL insert statement utilizes bound parameters, which is a recommended practice to defend against SQL injection threats. Upon successful insertion, the script displays a message indicating that the customer’s details were stored in the database.
Please note: This implementation is for demonstration purposes and does not include any form of input validation. For real-world applications, make sure to add data sanitation and security checks to strengthen the integrity of your system.
Validating the Redis® Worker Script
To verify that the redis_worker.php script is functioning properly, you’ll need to access the URL http://localhost/redis_worker.php. Run the following command three times to process all previously queued entries:
$ curl http://localhost/redis_worker.php
If everything is set up correctly, the output should indicate that customer records are being removed from the Redis® list and stored in the database in the same order they were added:
JOHN DOE's details saved to database.
MARY SMITH's details saved to database.
ROE STEVE's details saved to database.
Checking the Data in the MySQL Database
To confirm that all customer data has been correctly stored, log into your MySQL server using the root user credentials:
$ sudo mysql -u root -p
Once authenticated, select the appropriate database:
mysql> USE test_db;
Now run a SELECT query to display all entries in the customers table:
mysql> SELECT
customer_id,
first_name,
last_name,
email_address
FROM customers;
If successful, the output should resemble the table below, indicating that the Redis® queuing and processing system is functioning correctly:
+-------------+------------+-----------+------------------------+
| customer_id | first_name | last_name | email_address |
+-------------+------------+-----------+------------------------+
| 1 | JOHN | DOE | john_doe@example.com |
| 2 | MARY | SMITH | mary_smith@example.com |
| 3 | ROE | STEVE | roe_steve@example.com |
+-------------+------------+-----------+------------------------+
3 rows in set (0.00 sec)
When finished, exit the MySQL command-line session:
mysql> QUIT;
Automating the Redis® Worker with a Cron Job
Previously, you executed the redis_worker.php script manually using the curl command. In this step, you’ll automate that process by configuring a cron job to run the worker at regular intervals without manual intervention.
Start by opening the system’s crontab file:
$ sudo nano /etc/crontab
Append the following line at the bottom of the file to run the redis_worker.php script every minute:
* * * * * root /usr/bin/wget -O - http://localhost/redis_worker.php
Save and exit the crontab file. Then, send a new customer entry to the redis_queue.php endpoint:
$ curl --data "first_name=BABY&last_name=SMALL&email_address=baby_small@example.com" http://localhost/redis_queue.php
The command should return a success message confirming that the customer data was added to the Redis® queue:
Customer details accepted successfully.
Wait approximately one minute, then log in to your MySQL database to verify the data was processed:
$ sudo mysql -u root -p
Once inside the MySQL CLI, switch to the appropriate database:
mysql> USE test_db;
Then, check the customers table using the following query:
mysql> SELECT
customer_id,
first_name,
last_name,
email_address
FROM customers;
If everything is working, you should see the new entry in the list:
+-------------+------------+-----------+------------------------+
| customer_id | first_name | last_name | email_address |
+-------------+------------+-----------+------------------------+
| 1 | JOHN | DOE | john_doe@example.com |
| 2 | MARY | SMITH | mary_smith@example.com |
| 3 | ROE | STEVE | roe_steve@example.com |
| 4 | BABY | SMALL | baby_small@example.com |
+-------------+------------+-----------+------------------------+
4 rows in set (0.00 sec)
For critical systems where one-minute intervals are not sufficient, consider creating a long-running PHP loop that continuously polls for new data.
Conclusion
This tutorial has demonstrated how to implement job queuing using Redis® and PHP on an Ubuntu 20.04 server. You’ve learned how to push data into a Redis® queue, process it with a background worker, and permanently store the data in a MySQL database. You can adapt and expand this approach based on the needs of your specific application.