Bing! What's Wrong With You?

Search engines are basically the "client" for websites.

While operating a website, Bing is a very troublesome "white elephant".

Of course, since Google's search engine traffic is the most important (the majesty of a 90% market share), we optimize for Google, and then assume that Bing will "automatically" crawl the Google-optimized content well. (It's probably because they don't pay much attention to a minor market share of 3%.)

robots.txt is used to inform search engines about the site's information.

Actually, we use IndexNowto automatically notify Naver and Bing in real-time whenever a new post is uploaded, so I'm doing everything I can. (Google does not support IndexNow.)

Considering the "standard protocol", websites generally define a basic text format called "robots.txt" and record basic data about robots there. It tells bots which pages should not be crawled, which bots are blocked, and where the site's "sitemap" is located.

Of course, durumis also meticulously recorded robots.txt according to the standard protocol.

However, somehow...Bing seems to ignore this...?

Why suddenly...?

Why does it try to access all pages of www. sites, even though I haven't specified that such pages exist...?

Ok. Fine. Let's assume that www.durumis.com is understandable. However, it tries to crawl various pages on various subdomains by adding "www." to the front. (Naturally, countless 404 errors are recorded in the load balancer log.)

Why does it access non-existent "/atom.xml", "/sitemap.txt", "/sitemap.xml.gz", and "/sitemap_index.html"? (It repeatedly tries this on countless subdomains.)

Searching reveals that it's not just our site, but other sites are experiencing similar issues.

Searching for complaints about this issue reveals many results.

I'm not even sure if IndexNow is working properly. When I make a request, it takes up to 4 days to crawl the page, (and I've only seen a few, so I don't know if it's crawling well...)

Search starts with crawling, followed by indexing. I have many doubts about whether it works properly.

(Isn't the answer in market share?)

While hoping that our service will be highly visible in search results, I'm also very interested in search technology. I'm not sure if Bing's approach is really correct.

I wonder why people in Korea call Bing by that nickname... (I won't use any curse words.)