The first search engine on the web was called Aliweb and this is its story

On a day like today (September 30) but in 1993, the first search engine for the web was presented: AliWeb. 

Aliweb preceded, as an idea and a product, the two 90s icons of the web organization: Yahoo and Google. Despite that, and as is often the case in the history of web services, being the first was not enough to achieve commercial, operational or even cultural success. And is that, do you remember Aliweb as you would MySpace or Geocities?

This is his story:

• The software company Nexor had a developer named Martijn koster, who in his free time and as a personal project (but within work time) developed a engine to organize and catalog websites, which in 1993 were far fewer than today. Much less.
• The project started in 1992, was publicly announced in 1993 and was even usable in 1994.
• The interface was more like a form than the magic box of the most popular versions of Yahoo or Google:
  
• Although in the background there was a engine that allowed to organize according to search criteria, in reality there was no crawl bot or something similar since there was rather human curatorship to feed the database.
• Even in an early exercise in collective intelligence, people were invited to submit their own entries for the database.
• Web search in those days was a specialty, since you had to choose the right search engine for your type of query. We can see this in the final season of the North American series Halt and Catch Fire, where a group of entrepreneurs proposes to investors the project of a search engine for medical terms intranet. That technology is seen as promising by portfolio managers and they decide to take their base to apply it to the web in general.
• Later, Koster (the creator of AliWeb), invented the concept of standard exclusion robots, which is quite similar to how modern search engines work. Contrary to a curated, human-fed database, we are rather talking about spiders searching all over the web for new content.
• It is thanks to Koster that we have robots.txt to avoid the crawling harmful.

More info here.