Amazon SQS producer and consumer in Node.js, step by step

An Amazon SQS pipeline in Node.js needs two small programs: a producer that sends messages to a queue, and a consumer that long-polls the queue, does the work, and deletes each message only once the work has succeeded. That is the whole pattern, and it is about 80 lines of code. The interesting part is what happens when things go wrong: a crash halfway through a job, a job that runs too long, a message that never succeeds.

This tutorial builds the pipeline from my aws-sqs-demo repository, where an Express API queues jobs and a worker writes each one to S3. Then it covers what I would add before trusting it in production.

What you are building

HTTP GET /produce?...  →  SQS queue  →  consumer  →  S3 (one JSON file per job)

• The producer turns a query string into a JSON message: /produce?orderId=123&status=pending becomes {"orderId":"123","status":"pending"}.
• The consumer receives up to 10 messages at a time, writes each to S3 using the SQS message ID as the object key, then deletes it from the queue.

You need Node.js 18 or later, a standard SQS queue, an S3 bucket, and credentials that allow sqs:SendMessage, sqs:ReceiveMessage, sqs:DeleteMessage and s3:PutObject.

The producer

The producer is an Express route and one SDK call. The SQS client is created once at module load and reused for every request:

// producer/modules/sqs.js
const { SQSClient, SendMessageCommand } = require("@aws-sdk/client-sqs");

const sqsClient = new SQSClient({ region: "af-south-1" });
const QUEUE_URL = process.env.QUEUE_URL;

const sendMessageToQueue = async (message) => {
  return sqsClient.send(new SendMessageCommand({
    QueueUrl: QUEUE_URL,
    MessageBody: JSON.stringify(message),
  }));
};

// producer/routes/produce.js
router.get("/", async (req, res) => {
  const message = req.query;
  if (!message || Object.keys(message).length === 0) {
    return res.status(400).send("Message is required");
  }
  await sendMessageToQueue(message);
  res.send("Message sent");
});

Test it:

curl "http://localhost:8080/produce?orderId=123&status=pending"

The response comes back as soon as SQS has stored the message. Whatever the job is, the HTTP request no longer waits for it. That is the point of a queue: the producer stays fast, and slow or failing work happens somewhere else.

Keep messages small. SQS caps a message at 1 MiB (it was 256 KB until 2025), and anything large belongs in S3, with only its key in the message.

The consumer

The consumer is a loop around three calls: receive, process, delete.

// consumer/modules/sqs.js
const receiveMessages = async () => {
  const response = await sqsClient.send(new ReceiveMessageCommand({
    QueueUrl: QUEUE_URL,
    MaxNumberOfMessages: 10,
    WaitTimeSeconds: 20,
    VisibilityTimeout: 30,
  }));
  return response.Messages ?? [];
};

// consumer/index.js
const processMessage = async (message) => {
  const job = JSON.parse(message.Body);
  await putJsonObject(message.MessageId, JSON.stringify(job));
};

async function main() {
  while (true) {
    const messages = await receiveMessages();
    for (const message of messages) {
      try {
        await processMessage(message);
        await deleteMessage(message);
      } catch (error) {
        console.error("Job failed:", error);
      }
    }
  }
}

Each of the three receive parameters does a specific job:

• MaxNumberOfMessages: 10 is the most SQS returns in one call. Fewer calls means fewer requests to pay for.
• WaitTimeSeconds: 20 turns on long polling. When the queue is empty, SQS holds the request open for up to 20 seconds (the maximum) instead of answering "nothing" straight away. A tight loop with short polling would make thousands of empty, billable requests an hour.
• VisibilityTimeout: 30 hides each received message from other consumers for 30 seconds. If it has not been deleted by then, it becomes visible again and someone else can receive it.

Why delete comes last

A received message is not removed from the queue. It is only hidden for the visibility timeout. The consumer deletes it explicitly, and only after processMessage has succeeded.

That ordering is what makes the pipeline safe:

• The work throws: the catch logs it and skips the delete. Thirty seconds later the message reappears and is retried.
• The consumer crashes mid-job: same result. Nothing was deleted, so nothing is lost.
• Deleting first would lose jobs: if the process died after the delete but before the work finished, the message would be gone for good.

Plan for duplicates

Standard SQS queues deliver at least once. A message can occasionally arrive twice, and a job that runs longer than the visibility timeout will be handed to a second consumer while the first is still working on it. Your processing has to be safe to repeat.

This consumer gets that almost for free. It writes each job to S3 under the message ID, and a redelivered message keeps its ID, so a second run overwrites the same object with the same content. The result is the same whether the job ran once or three times.

Most real jobs are not that convenient. Sending an email or charging a card twice is a bug. The usual fix is an idempotency record written in the same atomic step as the work; I walk through that pattern in idempotent writes in DynamoDB with a guard item.

Note what the message ID does not cover: if the producer itself retries a SendMessage after a timeout, SQS stores two messages with two IDs. If that matters, put your own job ID in the body and deduplicate on that.

Before you run this in production

The demo is deliberately small. These are the changes I would make first.

Add a dead-letter queue

Right now a message that always fails (bad JSON, a bug for one input) is retried every 30 seconds until the queue's retention period ends, which is 4 days by default. Give the queue a redrive policy that moves a message to a dead-letter queue after a few failed receives:

{
  "deadLetterTargetArn": "arn:aws:sqs:<region>:<account-id>:jobs-dlq",
  "maxReceiveCount": 5
}

Failures then stop clogging the queue, and you have a list of exactly which jobs failed.

Set the visibility timeout from the real job time

Thirty seconds is fine for one S3 write. If a job can take two minutes, raise it above two minutes, or the message reappears while it is still being processed and you get duplicates by design. For jobs of very uneven length, extend it while working with ChangeMessageVisibility.

Use roles, not access keys

The demo reads an access key and secret from .env. On AWS, run the consumer with an IAM role instead: an ECS task role, an EC2 instance profile or a Lambda execution role. Then drop the credentials option, and the SDK picks the role up through its default credential chain. Nothing secret lives in the environment, and the keys rotate themselves.

Keep the queue and the work in one region

In my demo the queue is in af-south-1 and the bucket in us-east-1, because that is where I had them. Every message crosses an ocean twice, which adds latency and inter-region data transfer charges. Put the queue, the consumer and the resources it writes to in one region.

Process in parallel and shut down cleanly

The loop handles messages one at a time. For I/O-bound work, process each batch with Promise.allSettled and delete the successful ones with a single DeleteMessageBatch call. Also handle SIGTERM: stop receiving, let in-flight jobs finish, then exit. Container platforms send SIGTERM before they stop a task, and anything unfinished is simply retried, but a clean stop avoids the duplicates.

Or let Lambda do the polling

If each job fits in Lambda's 15-minute limit, you can skip the consumer process entirely. An SQS event source mapping polls the queue for you, invokes a function with a batch, deletes the messages when the function succeeds, and scales the number of pollers with the queue's depth. You still need idempotent processing and a dead-letter queue.

Summary

• Send with SendMessage; the producer returns as soon as SQS has the message.
• Receive with long polling (WaitTimeSeconds: 20) and batches of up to 10.
• Delete only after the work succeeds; failures come back after the visibility timeout.
• Assume duplicates and make processing idempotent.
• Add a dead-letter queue, use IAM roles and keep everything in one region before production.

The full code, with setup steps for both halves, is in the aws-sqs-demo repository.

Frequently asked questions

Why does my SQS message get processed twice?

Standard queues deliver at least once. A message can arrive twice, and a message whose processing outlives its visibility timeout becomes visible again and goes to another consumer. Make processing idempotent so a repeat has no extra effect, and set the visibility timeout longer than the work takes.

What does WaitTimeSeconds do in ReceiveMessage?

It turns on long polling. Instead of returning straight away when the queue is empty, SQS waits up to that many seconds (20 at most) for a message to arrive. That cuts empty responses, which you pay for, and gets messages to the consumer sooner.

When should a consumer delete an SQS message?

Only after the work has succeeded. If the consumer crashes or the work throws, leaving the message alone lets it become visible again after the visibility timeout, so it is retried.

What happens to a message that keeps failing?

Without a dead-letter queue it is retried until the queue's retention period ends, then it is deleted. Add a redrive policy with maxReceiveCount so it moves to a dead-letter queue after a few failed attempts.