Introduction ¶

Today's search engines are broken. They destroyed the discoverable part of the internet. We are no longer able to find content on the internet. We are pushed towards their own content and big players like Wikipedia or stack overflow. I no longer follow this path and tried to summarize my thoughts and ideas on that topic.

I don't remember the times when there were no search engines. As far as I know people used some link lists. I actually had a book which had aggregated a large list of websites, grouped by different topics. That way I discovered parts of the web.

Today we are used to "google.com", "bing.com", and some other search engines. If you are like me, you might already have switched from those global players to some alternatives. Those alternatives might still use the global players and work as a proxy. Some alternatives have completely different concepts. Some are meta searches like searx.

My issue ¶

Today's search engines don't work. The way they work just doesn't cover my actual requirements. They work fine as a meta search for big websites like Wikipedia, stack overflow and some more. But they don't work to find anything on the web itself.

Concrete example ¶

All of those modern search engines have one big issue:

There was this nice blog post I've discovered on mastodon. Now I want to read it again, or share with a friend. The bad thing: I didn't bookmarked the blog post and can't remember the website. Probably one would search based on the content of the blog post. The blog post covered plain text email and why it is way better than HTML email. It also covered how to properly write plain text emails. I tried to search for this topic and found many search results. Top results were from microsoft.com and stack overflow. But I was not able to find this blog post. Thanks to mastodon, someone else was aware of this post and shared it again with me. The blog post I'm talking about was https://drewdevault.com/2016/04/11/Please-use-text-plain-for-emails.html.

Possible root cause ¶

I guess most people just don't care about these complains. Some people don't even search for websites, they are fine with google results and take them for granted. They don't even visit the results. Most people just don't care about the web. So why should search engines?

Search engines tell us "we try to deliver the answer to your question". But I don't have concrete questions to the internet. I want to discover the internet. I want to find blog posts, ideas, concepts and thoughts of others. And I want to find something specific on the internet. These requirements seem to be out of scope for today's search engines.

Also search engines might add the same domain with different URLs on the first pages. Wasting space to discover other websites that handle the same topic. I don't care about those 15 plain text search results on microsoft.com, I care about those 15 blog posts out there on individual websites. But those websites don't have SEO. They are not optimized for search engines. They are actually optimized for content and visitors. But that's a different target group.

My conclusion ¶

I use on site searches. E.g. I use Wikipedia search, stack overflow search, search on php.net. I will use your search if I need to find things on your site. You know your content way better, and hopefully provide proper facets.

For some sites you don't even need a search. Wikipedia has a pretty easy URL structure, so I can directly open the site I want to visit. Not even wasting resources on additional server requests just to click on the Wikipedia search result.

I add more and more RSS feeds to my own Nextcloud instance to "discover" the great content on the internet. I add bookmarks, so my browser can suggest the great blog post with "plain text" inside the URL.

And I probably will create link lists on my website. The goal would be the same as "awesome lists": make the web discoverable again. We lost this great feature on our way to make the web better.

My wish ¶

Please keep your own website, create content and share content. Use social media like mastodon to share blog posts and essays. Make your content discoverable. Use proper URLs and titles. Cross-reference to other blogs and websites as well as articles.

In case you have a search on your site, please check whether you already provide facets. Please don't mimic global player search engines on your own website. Visitors should be able to discover your content. Allow them to filter your content to specific topics. They should be able to sort content, e.g. by publishing day.

I loved online shops some years back. You could open categories and browse their content. Today's online shops mostly have searches and don't allow you to discover their content. They invented stuff like recommendations, which don't work. But they removed the possibility to just browse their content. Don't follow that path, always allow visitors to discover your content. Allow them to waste time and follow Alice into your rabbit hole.

Acknowledgements ¶

Thanks to @arh for the hint to link to joinmastodon.org, see: https://toots.alirezahayati.com/@arh/105714831472442230

@alcinnz recommends searchmysite.net, see: https://floss.social/@alcinnz/105714727242128925. This "search" allows browsing sites by categories. That way you can discover websites again.