Why a page cannot be read · robots_disallowed
robots.txt asked us not to read the page
Your robots.txt disallows this crawler, so the page was never requested. That is recorded as a finding and never worked around.
robots.txt asks machines not to read this path. Recorded as a data point, never circumvented.
§01 · what is actually happening
What is actually happening.
robots.txt is fetched before anything else. When it disallows the path, no request for the page is made at all.
Groups in robots.txt are not additive: the most specific matching group wins outright and the wildcard group is then ignored entirely. A rule written for one named crawler can silently stop applying to everything else, and the reverse.
A disallow is published as a data point. Routing around it would make every number this product publishes worthless.
§02 · what it costs
What it costs.
If the rule was meant, none. Asking not to be read is a legitimate choice and we record it as a choice rather than a failure.
If the rule was broader than intended, the cost is the same as a refusal: assistants answering questions about your products answer from somewhere else. A disallow aimed at one badly behaved scraper is often written in a way that covers every non-browser request, including the ones that would have cited you.
§03 · what would change it
The fix.
Nothing, unless the rule was broader than you meant. A disallow aimed at one badly behaved crawler is often written in a way that covers every non-browser request, including the ones answering questions about your products.
Who does it. Whoever owns robots.txt. Reading it carefully is most of the work, because the group precedence rule surprises people.
Check your own site.
One request, one verdict, no account. If this is the finding you get back, the page above is the one that applies.
See what a machine seesThe other findings