Node.js axios upload large file using FormData, read all data to the memory, cause out of memory issue.

Explanation of the problem

The bug in this description pertains to a problem encountered while uploading large files using FormData in Node.js with Axios. Specifically, the issue is that the entire data is read to memory, causing an out of memory error. The code block that is intended to solve this issue is as follows:

const formData = new FormData();
formData.append('file', fs.createReadStream(path), { filepath: path, filename: basename(path) });
formData.append('videoId', videoId);
await axios.post(UPLOAD_MEDIA_URL, formData, {
  headers: {
    'Content-Type': `multipart/form-data; boundary=${formData.getBoundary()}`,
    'Authorization': `Bearer ${await token()}`,
  },
  maxBodyLength: Infinity,
  maxContentLength: Infinity,
});

const stream = fs.createReadStream(path);
const formData = new FormData();
formData.append('file', stream, { filepath: path, filename: basename(path) });
formData.append('videoId', videoId);
await axios.post(UPLOAD_MEDIA_URL, formData, {
  headers: {
    'Content-Type': `multipart/form-data; boundary=${formData.getBoundary()}`,
    'Authorization': `Bearer ${await token()}`,
  },
  maxBodyLength: Infinity,
  maxContentLength: Infinity,
  onUploadProgress: progressEvent => {
    console.log(Math.round((progressEvent.loaded * 100) / progressEvent.total));
  },
});

Troubleshooting with the Lightrun Developer Observability Platform

Getting a sense of what’s actually happening inside a live application is a frustrating experience, one that relies mostly on querying and observing whatever logs were written during development.
Lightrun is a Developer Observability Platform, allowing developers to add telemetry to live applications in real-time, on-demand, and right from the IDE.

• Instantly add logs to, set metrics in, and take snapshots of live applications
• Insights delivered straight to your IDE or CLI
• Works where you do: dev, QA, staging, CI/CD, and production

Start for free today

Problem solution for Node.js axios upload large file using FormData, read all data to the memory, cause out of memory issue.

Other popular problems with Axios

Problem: Handling CORS (Cross-Origin Resource Sharing) errors

When making cross-origin requests, the browser will automatically block the request if the server does not have the appropriate CORS headers set. This can lead to issues such as the following error message: “No ‘Access-Control-Allow-Origin’ header is present on the requested resource.”

Solution:

To solve this issue, you can use the “proxy” option in the Axios config to route the request through a server-side proxy. This proxy can add the necessary CORS headers to the request before it is sent to the server.

const axios = require('axios');

const instance = axios.create({
  baseURL: 'https://yourserver.com',
  proxy: {
    host: '127.0.0.1',
    port: 9000,
  },
});

Alternatively, you can set the headers directly on the server side.

app.use((req, res, next) => {
    res.header('Access-Control-Allow-Origin', '*');
    res.header('Access-Control-Allow-Headers', 'Origin, X-Requested-With, Content-Type, Accept');
    next();
});

Problem: Handling redirects

By default, Axios will automatically follow redirects, which can lead to issues such as infinite redirect loops.

Solution:

To solve this issue, you can set the “maxRedirects” option in the Axios config to limit the number of redirects that are followed.

const axios = require('axios');

const instance = axios.create({
  baseURL: 'https://yourserver.com',
  maxRedirects: 5,
});

Alternatively, you can set the “validateStatus” option in the Axios config to a function that will only allow certain status codes to be considered as successful. This can be used to prevent redirects that return a non-success status code from being followed.

const axios = require('axios');

const instance = axios.create({
  baseURL: 'https://yourserver.com',
  validateStatus: function (status) {
    return status >= 200 && status < 300; // default
  },
});

Problem: Handling timeouts

By default, Axios will wait indefinitely for a response from the server. This can lead to issues such as hanging requests and slow performance.

Solution:

To solve this issue, you can set the “timeout” option in the Axios config to a specific time in milliseconds.

const axios = require('axios');

const instance = axios.create({
  baseURL: 'https://yourserver.com',
  timeout: 5000,
});

Alternatively, you can set the “timeout” option in the config of a specific request.

axios.get('/longRequest', {
  timeout: 3000
});

A brief introduction to Axios

Axios is a popular JavaScript library for making HTTP requests. It is based on the promise API, which allows for a more elegant way of handling asynchronous requests. Axios can be used in both the browser and in Node.js environments and it supports all popular HTTP methods such as GET, POST, PUT, DELETE and PATCH.

Axios provides a simple and easy-to-use API for making requests. It allows you to set a wide range of options, such as headers, query parameters, and request data, in a single config object. Additionally, Axios allows you to intercept requests and responses, which can be useful for adding additional functionality such as logging, authentication, or error handling. Axios also supports features such as automatic retries, automatic JSON parsing and support for canceling requests. Axios also has a built-in feature that allows you to easily manage and maintain multiple instances of the library, each with its own set of configuration options, which is useful when working with different APIs or servers. Overall, Axios is a powerful and flexible library that can be used for a wide range of use cases, making it a popular choice among developers.

Most popular use cases for Axios

1. Axios can be used for making HTTP requests to a server in a simple and consistent way. It is a popular JavaScript library that supports both Node.js and web browsers.

axios.get('https://api.example.com/users')
  .then(response => console.log(response.data))
  .catch(error => console.log(error));

axios.interceptors.request.use(config => {
  config.headers.Authorization = `Bearer ${token}`;
  return config;
});

axios.interceptors.response.use(response => {
  if (response.data.status === 'error') {
    throw new Error(response.data.message);
  }
  return response;
});

axios.all([
  axios.get('https://api.example.com/users'),
  axios.get('https://api.example.com/posts')
])
.then(axios.spread((usersResponse, postsResponse) => {
  console.log(usersResponse.data);
  console.log(postsResponse.data);
}));