Do not add pages marked with `noindex` to sitemap

Is your feature request related to a problem? Please describe. Pages with <meta content="noindex, follow" name="robots" /> should not be added to the sitemap. I am not sure about how this library works, but I don’t think it actually reads the content of the files so it may be hard to detect that meta tag.

[gabrielreisn]

hey, in case someone is still missing this, the solution I’m currently using within my team is a custom transform function. If that’s ok I can open a PR with the fix

transform: async (config, path) => {
    const noIndexRegex = /<meta.*noindex/gim
    const basePath = '.next/serverless/pages'
    const filePath = `${basePath + path}.html`

    if (fs.existsSync(filePath)) {
      try {
        const data = await fs.promises.readFile(filePath, 'utf8')

        if (data.match(noIndexRegex)) {
          console.log('ignored file:', filePath)

          return null
        }
      } catch (error) {
        console.error('err', error)
      }
    }

    return {
      loc: path,
      changefreq: config.changefreq,
      priority: config.priority,
      lastmod: config.autoLastmod ? new Date().toISOString() : undefined,
      alternateRefs: config.alternateRefs || [],
    }
  },

[rserafim]

I remove like this

module.exports = { siteUrl: ‘https://www.xxxx, exclude: [’/aaa/', ‘/xxx’, ‘/yyyy’], // <= exclude here