bootstrap/site/layouts/robots.txt

# www.robotstxt.org

{{- $isProduction := eq hugo.Environment "production" -}}
{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}
{{- $allowCrawling := and (not $isNetlify) $isProduction -}}

{{ if $allowCrawling }}
# Allow crawling of all content
{{- end }}
User-agent: *
Disallow:{{ if not $allowCrawling }} /{{ end }}
Sitemap: {{ "/sitemap.xml" | absURL }}
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 18:33:28 +02:00			`# www.robotstxt.org`

Use `hugo.Environment` instead of `getenv` (#29240) This is set automatically to "development" when the local server is running, and to "production" when Hugo builds the site. 2020-02-22 08:54:43 +02:00			`{{- $isProduction := eq hugo.Environment "production" -}}`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 17:26:25 +03:00			`{{- $isNetlify := eq (getenv "NETLIFY") "true" -}}`
			`{{- $allowCrawling := and (not $isNetlify) $isProduction -}}`

			`{{ if $allowCrawling }}`
Switch to Hugo. No more Ruby dependency and most important more than 10 times less build time (at least on Windows)! 2019-01-08 18:33:28 +02:00			`# Allow crawling of all content`
			`{{- end }}`
			`User-agent: *`
robots.txt: adapt for Netlify. (#29192) Since we build with `HUGO_ENV` set to `production` on Netlify, use another variable to prevent crawling. 2019-08-02 17:26:25 +03:00			`Disallow:{{ if not $allowCrawling }} /{{ end }}`
Use `absURL` so that we get consistent URLs. (#29226) 2019-08-08 18:30:43 +03:00			`Sitemap: {{ "/sitemap.xml" \| absURL }}`