File Uploads with Multer and Large Files via Streams

How File Uploads Work

Browsers and mobile apps send files as multipart/form-data: one HTTP request split into parts, where each part is either a normal field (like restaurantId) or a file. Express's express.json() can't read this format, so you need a parser.

Multer is the most common Express middleware for it. It reads the request, saves the file, and puts the details on req.file (one file) or req.files (several).

In our food delivery app, restaurant owners upload two kinds of files:

Upload	Size	Approach
Dish photos	Small, a few MB	Multer, with size and type limits
Bulk menu CSV, kitchen videos, monthly reports	Large, hundreds of MB or more	Streams: process the file in small chunks

Multer: Basic Upload

Multer has two built-in storage engines (where the file goes):

Storage	File ends up	Good for
`diskStorage`	A file on the server's disk, path in `req.file.path`	Files you'll keep or process later
`memoryStorage`	A `Buffer` in RAM, in `req.file.buffer`	Small files you forward right away (e.g. to S3 or an image resizer)

const express = require('express');
const multer = require('multer');
const path = require('path');
const crypto = require('crypto');

const storage = multer.diskStorage({
  destination: function (req, file, cb) {
    cb(null, 'uploads/dishes');
  },
  filename: function (req, file, cb) {
    // never trust the user's filename: generate our own
    const ext = path.extname(file.originalname).toLowerCase();
    cb(null, crypto.randomUUID() + ext);
  }
});

const upload = multer({
  storage: storage,
  limits: {
    fileSize: 5 * 1024 * 1024,      // 5 MB per file
    files: 5                        // max 5 files per request
  },
  fileFilter: function (req, file, cb) {
    const allowed = ['image/jpeg', 'image/png', 'image/webp'];
    if (allowed.includes(file.mimetype)) cb(null, true);
    else cb(new multer.MulterError('LIMIT_UNEXPECTED_FILE', file.fieldname));
  }
});

const app = express();

// one photo, form field name 'photo'
app.post('/dishes/:id/photo', upload.single('photo'), function (req, res) {
  // req.file: { fieldname, originalname, mimetype, size, filename, path }
  // req.body: other text fields from the form
  res.json({ url: '/uploads/dishes/' + req.file.filename });
});

// several photos at once
app.post('/dishes/:id/gallery', upload.array('photos', 5), function (req, res) {
  res.json({ uploaded: req.files.length });
});

Method	Reads	Result
`upload.single('photo')`	One file from field `photo`	`req.file`
`upload.array('photos', 5)`	Up to 5 files from field `photos`	`req.files` (array)
`upload.fields([{ name: 'cover' }, { name: 'logo' }])`	Files from several named fields	`req.files.cover`, `req.files.logo`
`upload.none()`	Text fields only	`req.body`

Handling Multer errors. Limits and filters raise a MulterError. Catch it in your error middleware and return a clear 400:

app.use(function (err, req, res, next) {
  if (err instanceof multer.MulterError) {
    if (err.code === 'LIMIT_FILE_SIZE') {
      return res.status(400).json({ message: 'Photo must be under 5 MB' });
    }
    return res.status(400).json({ message: 'Upload error: ' + err.code });
  }
  next(err);
});

Upload Security Checklist

Risk	Fix
Huge files fill the disk or RAM	Always set `limits.fileSize` and `limits.files`
Fake file type: `file.mimetype` comes from the client and can lie	Check the file's real content (magic bytes) with a library like `file-type` after upload
Malicious filenames like `../../server.js`	Generate your own names (UUID), never use `originalname` as the path
Uploaded files run as code	Store outside the code folder, never serve uploads with execute rights; better, store in object storage (S3)
Outdated library	Keep Multer on its latest major version, which includes security fixes

The Problem With Large Files

memoryStorage and fs.readFile load the whole file into RAM. That's fine for a 2 MB photo, not for a 2 GB file:

1memoryStorage, 10 owners upload a 500 MB video at once
2  10 × 500 MB = 5 GB of RAM
3→ the Node.js process runs out of memory and crashes
4 
5Stream, same 10 uploads
6  each processes ~64 KB at a time
7  10 × 64 KB ≈ 640 KB of RAM
8→ memory stays flat no matter how big the file is

A stream handles data in small chunks as it arrives, instead of waiting for all of it. Think of drinking through a straw instead of trying to swallow the whole bottle at once.

Node.js has four stream types:

Type	Does	Example
Readable	Produces data	`fs.createReadStream`, an incoming HTTP request (`req`)
Writable	Consumes data	`fs.createWriteStream`, an HTTP response (`res`)
Duplex	Both	A network socket
Transform	Changes data as it passes through	`zlib.createGzip()`, a CSV parser

Backpressure: if the writer is slower than the reader (a slow disk, a slow client), the reader must pause until the writer catches up. Otherwise chunks pile up in memory. pipeline handles this automatically, and also cleans up every stream if one fails. Prefer it over .pipe(), which doesn't forward errors.

Streaming a Large Upload to Disk

Multer uses Busboy internally. For very large files, use Busboy directly and pipe each file straight to disk, without buffering it in memory:

const busboy = require('busboy');
const fs = require('fs');
const path = require('path');
const crypto = require('crypto');
const { pipeline } = require('stream/promises');

app.post('/restaurants/:id/videos', function (req, res, next) {
  const bb = busboy({
    headers: req.headers,
    limits: { fileSize: 2 * 1024 * 1024 * 1024, files: 1 }   // 2 GB
  });

  bb.on('file', function (fieldname, fileStream, info) {
    if (info.mimeType !== 'video/mp4') {
      fileStream.resume();                       // drain and discard
      return res.status(400).json({ message: 'Only MP4 videos allowed' });
    }

    const savePath = path.join('uploads/videos', crypto.randomUUID() + '.mp4');

    fileStream.on('limit', function () {
      fs.unlink(savePath, function () {});       // remove the partial file
      res.status(413).json({ message: 'Video must be under 2 GB' });
    });

    pipeline(fileStream, fs.createWriteStream(savePath))
      .then(function () {
        if (!fileStream.truncated) res.status(201).json({ path: savePath });
      })
      .catch(next);
  });

  req.pipe(bb);
});

Processing a Large CSV Line by Line

A restaurant uploads a menu CSV with 1 million rows. Read it as a stream, parse each row as it arrives, and insert in batches so the database isn't hit a million times:

const fs = require('fs');
const { parse } = require('csv-parse');

async function importMenu(filePath, restaurantId) {
  const parser = fs.createReadStream(filePath).pipe(parse({ columns: true, trim: true }));

  let batch = [];
  let imported = 0;

  for await (const row of parser) {              // one row at a time, backpressure handled
    batch.push({
      restaurantId: restaurantId,
      name: row.name,
      price: Number(row.price),
      isVeg: row.is_veg === 'true'
    });

    if (batch.length === 1000) {
      await MenuItem.insertMany(batch);          // 1 query per 1,000 rows
      imported += batch.length;
      batch = [];
    }
  }

  if (batch.length > 0) {
    await MenuItem.insertMany(batch);
    imported += batch.length;
  }
  return imported;                               // 1,000,000 rows -> 1,000 inserts
}

For a job this long, don't run it inside the HTTP request. Save the file, put a job on a queue (like BullMQ), respond with 202 Accepted, and let a worker import it.

Streaming a Large File Download

Sending a monthly sales report the same way avoids loading it into memory. Adding gzip compresses it on the fly:

const fs = require('fs');
const zlib = require('zlib');
const { pipeline } = require('stream/promises');

app.get('/reports/:month', async function (req, res, next) {
  const filePath = path.join('reports', path.basename(req.params.month) + '.csv');

  res.setHeader('Content-Type', 'text/csv');
  res.setHeader('Content-Encoding', 'gzip');
  res.setHeader('Content-Disposition', 'attachment; filename="report.csv"');

  try {
    await pipeline(
      fs.createReadStream(filePath),   // read in chunks
      zlib.createGzip(),               // compress each chunk
      res                              // send each chunk to the client
    );
  } catch (err) {
    next(err);
  }
});

path.basename strips any ../ from the user's input, so the request can't read files outside the reports folder.

Best for Very Large Files: Upload Directly to S3

Even with streams, every byte still passes through your server. For big files, let the client upload straight to object storage (like Amazon S3). The server only creates a short-lived presigned URL: a link that allows one upload for a few minutes.

11. App   → API:  "I want to upload kitchen-tour.mp4"
22. API   → App:  presigned URL (valid 5 minutes)
33. App   → S3:   uploads the file directly
44. App   → API:  "done, key = videos/abc.mp4" → API saves it on the restaurant

const { S3Client, PutObjectCommand } = require('@aws-sdk/client-s3');
const { getSignedUrl } = require('@aws-sdk/s3-request-presigner');
const crypto = require('crypto');

const s3 = new S3Client({ region: 'ap-south-1' });

app.post('/restaurants/:id/videos/upload-url', async function (req, res) {
  const key = 'videos/' + req.params.id + '/' + crypto.randomUUID() + '.mp4';

  const url = await getSignedUrl(
    s3,
    new PutObjectCommand({ Bucket: 'food-app-media', Key: key, ContentType: 'video/mp4' }),
    { expiresIn: 300 }                    // 5 minutes
  );

  res.json({ uploadUrl: url, key: key });
});

Which Approach to Use

Approach	Memory use	Server bandwidth	Use for
Multer `memoryStorage`	Whole file in RAM	Full file	Small files forwarded right away
Multer `diskStorage`	Low	Full file	Normal uploads: photos, documents
Busboy + `pipeline`	Constant, small	Full file	Large files you must handle on the server
Presigned URL to S3	None	None	Very large files, high upload traffic

Interview Angle

How to Explain it in an Interview

The 30-Second Spoken Pitch

"File uploads arrive as multipart form data, and in Express I parse them with Multer. I use disk storage for normal files and memory storage only for small files I forward right away, and I always set file size and file count limits, generate my own filenames, and check the real file type instead of trusting the mimetype the client sends. For large files, loading everything into memory doesn't scale: ten 500 megabyte uploads would need five gigabytes of RAM. So I stream them, using Busboy and stream pipeline, which processes small chunks, handles backpressure, and cleans up on errors. For a big CSV I read it as a stream and insert rows in batches, usually in a background job. For very large uploads, the best option is a presigned S3 URL, so the client uploads directly to storage and the file never passes through my server."

Common mistakes:

Using memoryStorage or fs.readFile for big files.
No fileSize limit, so one request can fill the disk.
Using .pipe() without error handling; a failed stream can leak file handles. Use pipeline.
Running a long CSV import inside the request instead of a background job.

Summary

1multipart/form-data → parsed by Multer → req.file / req.files
2Storage      → diskStorage (normal) | memoryStorage (small files only)
3Always       → fileSize + files limits, own filenames, check real file type
4Large files  → streams: small chunks, constant memory
5pipeline     → handles backpressure + errors (prefer over .pipe())
6Big upload   → Busboy → pipeline → disk
7Big CSV      → read stream → parse → batch insert, in a background job
8Download     → createReadStream → gzip → res
9Huge files   → presigned URL → client uploads directly to S3

Long imports belong outside the request. Next, job queues with BullMQ show how to move work like this into background workers using Redis.