Google Crawler can not find robots.txt

[djboyd4715]

We did an upgrade/migration from 6.4 to 2105. We have noticed that there has been a decline in our google anayltics numbers.

What we found out was that master.tag was not updated to 2105 standard ( all pages are served via HTTPS ). So we did that and now we can see the meta robots tag on all the pages.

I can also access the robots.txt file from our site url.

But Google is reporting that it can not find or reach the robots.txt file. Is there any reason for this? Is there something else that needs to be reviewed?

Is it a CCV2 vs CCV1 issue?

[djboyd4715]

I posted this question to SAP help. It was suggested that I create a "static" robots.txt file and attach it to my storefront endpoint.

Now, this solution does appear to work, but there is still no answer as to why the platform's standard feature of generating the robots.txt file when it is asked for via the RobotsController is not readable by google crawler.