node.js - How to encode or resolve the response of request Module in nodejs

Question

Hi I use request Module in Nodejs.

There is the code below:

const response = await request.post({
  uri: "https://item.rms.rakuten.co.jp/rms/mall/rsf/item/vc",
  headers: {
    Host: "item.rms.rakuten.co.jp",
    Origin: "https://item.rms.rakuten.co.jp",
    Referer: "https://item.rms.rakuten.co.jp/rms/mall/rsf/item/vc",
    "Content-Type": "application/x-www-form-urlencoded",
    "Content-Length": buildForm(data).length,
    Cookie: cookies~,
    Accept:
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9",
    "Accept-Encoding": "gzip, deflate, br",
    "Accept-Language": "ko-KR,ko;q=0.9,en-US;q=0.8,en;q=0.7,id;q=0.6",
    "Cache-Control": "max-age=0",
    Connection: "keep-alive",
    "sec-ch-ua": `"Google Chrome";v="87", " Not;A Brand";v="99", "Chromium";v="87"`,
    "sec-ch-ua-mobile": "?0",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "same-origin",
    "Sec-Fetch-User": "?1",
    "Upgrade-Insecure-Requests": 1,
    "User-Agent":
      "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36",
  },
  formData: buildForm(data),
})

const buildForm = (data) => {
  const keys = Object.keys(data);
  const form = [];

  for (let i = 0, l = keys.length; i < l; i++) {
    if (typeof data[keys[i]] === "string") {
      const data_value_buf = Buffer.from(data[keys[i]]);

      const data_value_convert = encoding.convert(
        data_value_buf,
        "EUCJP",
        "UTF8"
      );
      form.push(keys[i] + "=" + encoding.urlEncode(data_value_convert));
    } else if (typeof data[keys[i]] === "undefined") {
      form.push(keys[i] + "=" + "-");
    } else {
      form.push(keys[i] + "=" + data[keys[i]]);
    }
  }

  return form.join("&");
}

the data is Json type.

When I call the request, I got the broken string response.

so, I added encoding: null, in request post option, like

const response = await request.post({
  uri: "https://item.rms.rakuten.co.jp/rms/mall/rsf/item/vc",
  encoding: null,
  headers: {
    Host: "item.rms.rakuten.co.jp",
~~~~

then I use Iconv Module.

const iconv = new Iconv("EUC-JP", "UTF-8//translit//ignore");
const res = iconv.convert(response);

How to show the response without broken string
Also the response headers is below

Cache-control: no-store, no-cache
Connection: close
Content-Encoding: gzip
Content-Type: text/html; Charset=EUC-JP
Date: Tue, 12 Jan 2021 05:28:50 GMT
Pragma: no-cache
Server: Apache
Transfer-Encoding: chunked
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-frame-options: DENY
X-oracle-dms-ecid: 93mhk0e_y40000000
X-oracle-dms-rid: 0:1
X-XSS-Protection: 0

Please let me know how to get normal response like html tag response

Answer 1

You made a POST request. So you're SENDING data to the server. You don't get a HTML response. You get a response CODE and correspondant headers.

If you used a GET request instead, then the server could send you a HTML response. See that I said COULD. Not necessarily will. You can get JSON, or just a return code for example.

In this case, although it's not appearing completely, you can see a 200 result code in the top part of your image. This tell us that your request was successfull. And the headers are what you translated. But the relevant information is that the server returned a 200 code.

So or you GET something or you POST something. You can't get HTML while posting. In a post you send the data in the body of the request and then receive just the result code you received, or any relevant error code.

FYI, there are several other HTTP methods besides POST and GET, but for the moment stay with those 2 and when you really understood them you look the other ones.