I think there is massive opportunity for domain specific search engines though, imagine a search engine specifically designed for software engineers and developers, or one for academic research (not just papers but all online scientific content, news and discussion), or one targeting the arts. I think it’s these verticals that could be incredible.

You then potentially move towards a building “meta” search engines (if your are older than about 35 you will remember these) that work out what you are searching for and uses a domain specific engine.

Edit:

Just to add to this, people who say that “decentralised” search engines are the only way to compete with Google are not completely wrong, it’s just that it’s not about protocols and distributed indexes. It’s about a community of smaller search engines working within specific domains and collaborating (commercially) on meta search engines, prompting people to search on each others engines if it would be better for that search.

We almost need an “Open Search Co-Op” which smaller search engines can join to share technology and refers users to each other.

It would just take at astronomical amount of capital to challenge Google at generic Internet search, and the few organizations with that capital are pretty clear they don't think it's a good bet. For example, Amazon is quite content just being the place you want to go to first for the things you want to buy, as that is obviously the most lucrative area.

Even in niche areas, I think it's still an enormous task to challenge Google. Despite the fact that 90% of my searches for "WTF does this stacktrace mean?" end up on StackOverflow, I invariably start my search on Google because every now and then it gives me useful non-StackOverflow tidbits, and I know I can always drill in more with specific tags on StackOverflow later.

It's telling, that a Google search scoped to a particular web site (xyz site:website.com) almost always produces better results than that web site's actual built-in search engine. If a web site that knows in detail its own content, its own users, and its own domain, still can't make their built-in search better, how does anyone else have a chance?

Google was epic when it started but quickly ended up indexing it self in a near closed loop. Everyone else was not important, it was all about the picture in front of them that was really just a mirror. Everyone stopped making home pages, forums and weblogs because it just doesn't make sense to do it.

Shit, people looking to buy something talking among their own before doing the purchase? KILL IT WITH FIRE!

that becomes a big can of worms very quickly, and starts to put into perspective some of googles other most successful verticals: NLP, Translation, Maps ("Near me"). Of course a lot of the verticals are also a generic means to get value out of all the server resources and experts required to operate the engine and then additional related branches to those verticals (email -> drive -> docs)

You only search whitelisted sites, which removes a lot of the SEO and blogspam from the results, and makes the index is wildly smaller. Search is free, but users can also pay a cheap subscription for the ability to flag sites as spammy, submit sites they think are missing, and submit queries the site has trouble with. Someone manually goes through the top N most-repeated submissions each day (early on this is all of them) and determines whether to tweak the whitelist.

Because your initial audience has to be people who are very enthusiastic about better search results, you should identify what they're interested in and focus your efforts on initial whitelists to things they care about. That might be technical questions (MDN StackOverflow, every random framework's docs site), or maybe whatever it is journalists would want to have access to (I'm not qualified to answer).

I don't think VC-encouraged hypergrowth is possible with the above strategy, but you could probably get to ramen profitable in a reasonable time frame, with slow growth ever after.

The fate of runnaroo seems to provide evidence to the contrary.

I understand your intent with that domain-specific example but surprisingly, that reasonable intuition sometimes doesn't work...

E.g. for Amazon product search, I often deliberately use Google's engine to find the Amazon page because Amazon's own search can't do it. See example screenshots in my previous comment about that: https://news.ycombinator.com/item?id=27539199

Curious what you think these are. I prefer Bing's UI to Google's but aside from that Bing is (IMO) a dumpster fire that quite literally makes me feel negligent (towards myself) when I use it - am I missing something?

In the past, Google search results will point to some good websites you never heard of and are very low traffic. But nowadays, the first couple pages are all popular websites with huge traffic even when they are low quality on the content. You can only find low traffic but good content websites in comments from social sites like reddit and HN.

So it seems to me, to build a usable search engine, you only need to index most popular websites.

I would disagree actually. I think the majority of searches people do and the solutions they would be satisfied with require only indexing major sites. Create a better UX attached to PageRank of the top 10-100k+ Alexa ranked sites and you'd have a competitor that could focus on other improvements.

At the bottom you could search Google or DDG if you want more niche results.

I.e., optimizing UX of the p90 searches would be enough for people to move and the p90 of searches are happy with a pretty narrow range of sites. (You could insert value prop of: no ads, no tracking, and crowd-sourced refinement -- i.e., realtime upvote/downvoting -- of the top results, previews, caching, more knowledge graph stuff on the right side, deeper metadata processing of top sites (i.e., avoiding the need to 'site:*'), or some other revolutionary UX like a one sentence hypothesis solution of exactly what the person is asking for again upvoted/downvoted for refinement -- Google does this but there is a lot of room for improvement). It's fun to think about. There is an order of magnitude in both precision and accuracy that can be improved for huge swaths of common questions people ask on Google.

Agree we need a coop of search engines, but I'll add that probably an org like mozilla is best suited for it, that way it can stay non-profit and recycle any profits back into R&D. personally I believe most of these solid business plan companies can be replaced by a simpler open-source version as IMO a lot of their energy is spend in securing revenue lines & profits.

Mozilla is probably in the best position to do it, I'm just not sure they actually would do it in a way that appealed to consumers.

This doesn't address why MSFT can't compete with Google. If you solve that problem, then you really get to the heart of the solution.

Isn't this exactly the reason why one should do it, as Google is the essence of the online advertising industry and this very regulation could threaten its existence?

>I think there is massive opportunity for domain specific search engines though

Challenge with niche search engine is discoverability. Imagine you had 10 of the greatest vertical search engines available today - how do you remember which one to use and how? The beauty of a general web search is that it is an easy to remember entry point to all of what is out there.

> You then potentially move towards a building “meta” search engines

Ever wondered why is this called meta search engine, and an email software that connects to multiple email servers is called email client and not "meta email". "Search engine client" sounds like a more appropriate name.

The challenge comes from the massive investments in turning that data into a good search engine and cutting through spam. That is the investment I think Google has scaled over time. Secondly, mind share; the moat Google has is immense, although I think DuckDuckGo and Bing has shown given enough time it is possible to get mindshare slowly; though if anything this demonstrates the challenge.

It’s also a harder problem to solve than Google’s algorithm.

After two decades Microsoft seems to have brought in support, but they left the hard part to others.

They seem to be very dominant on the problem of “here’s my dataset, please provide me with an api / interface to search it”

When searching for something on HN, I get much better results using Google and site: rather than using the embedded algolia search.

I would have to think that this would be true of a corporate data set as well.

Their api might be what sets them apart, though. I’m not sure… not in that space.

Can you elaborate?

If someone searches for something on one engine and it doesn’t have the results the “meta” search engine could be checked and direct the user to another engine that would better fit their query.

Too much of the up-to-date high value information is posted onto semi-public channels. Like facebook groups, instagram posts, telegram channels, etc.

At the same time, too much of the publicly available websites are becoming clickbait and mindless marketing drivel.

Just try to research anything slightly obscure that has to be up-to-date information (from the last few years).

Like "how to get from Nairobi to Kisumu by bus". Google maps doesn't have this (not surprisingly), but what about the SERPs? Nothing there. 90% of the results are useless bus ticket sale websites that have no info, just SEO. (You can't buy this ticket online anyway).

Is Kenya just such an off-the-map destination that no backpacker has ever written about taking this route? No!

The information is out there. You'll find it inside the related facebook groups for travelers in east Africa. And if not? That's where you'll ask. But the answer? It'll never reach a SERP in the future.

Even more sad, is that if you keep looking on Google, you may find buried results from years ago. From travel forums and such. What's sad is that the info will be 10 years old, since no one uses public forums anymore. (Unless it's on reddit, pretty much the last public forum with any reach).

Would happily pay for that... even if it only has 5% of the coverage of Google, that's fine because Google is like 95% noise anyway.

It's not exactly your specifications but 2007 Mahalo attempted something like that and they shut down after a few years:

https://en.wikipedia.org/wiki/Mahalo.com

The website is no longer there so you have to use Wayback Machine or image search to see what the landing page for Mahalo search UI looked like : https://www.google.com/search?q=%22mahalo.com%22+search&tbm=...

Google’s revenue per user in the US underlines that point, taking a credit card out for an annual subscription for a $100 probably isn’t going to score a good conversion rate for anything beyond a very small userbase.

The predicted economic/behavioural constraints are inhibiting innovation. You’re forced to play Google’s game.

Like with smart speakers, why not manually add a bunch of interactions? You can do it yourself in settings by adding a phrase and desired response, why not have the team adding a bunch they would want?

I see this in discussions a lot where many times the answer is that the data doesn't exist, but thats not satisfactory to the person making the challenging.

So not only has FB hoarded the infomation, but it has no incentive to fix the group experience (e.g., prompt an input for group search during the asking bit) as it keeps engagement up.

- recent March Madness results; thread of examples, https://twitter.com/DotDotJames/status/1506826439961858052

- hot-off-the-press startup funding, https://twitter.com/DotDotJames/status/1504901028520640514

- new free trip contest announcements w/ query breakdown, https://twitter.com/DotDotJames/status/1506443371828686848

after: & before: cover many use cases, and can be used in any combo

before X date, before:2022-01-01 range of X to Y dates, after:2022-01-01 before:2022-02-01 multiple ranges, easiest to split up queries since X date, after:2022-03-14

& per OP question, can be used with site, e.g.,

after:2020-12-31 before:2022-01-01 site:news.ycombinator.com intitle:search

all posts w/ search in title during 2021 on HN

https://breezethat.com/?q=after%3A2020-12-31+before%3A2022-0...

or

after:2020-12-31 before:2022-01-01 site:reddit.com intitle:russia intitle:ukraine

all posts on Reddit during 2021 w/ Russia & Ukraine in title

https://breezethat.com/?q=after%3A2020-12-31+before%3A2022-0...

(all posts with search in title during 2021 on HN)

there's also some more advanced techniques, post on that forthcoming

The top-of-page "featured snippet":

> Kenya Airways and Jambojet Limited fly from Nairobi to Kisumu every 4 hours. Alternatively, The Guardian Coach operates a bus from Nairobi to Kisumu every 3 hours. Tickets cost $6 - $11 and the journey takes 6h 30m. Two other operators also service this route.

Then the rest of the page links to route-planning websites, a Tripadvisor discussion about bus from Nairobi, and other actually relevant results.

Google disappoints me too, but TBH I can't objectively gauge how good it is. My own opinion is just based on spotty personal experiences that I don't keep track of.

Why not ? A scrapper may not be able to find it, but we all have a Facebook account. You dont need an automated scrapper to create a search engine. Info can be manually curated or scrapped through web extensions

God no.

Nah.

[1]https://yandex.com/search/touch/?text=how+to+get+from+Nairob...

One thing a competitor needs is a new and innovative technical algorithm.

Back in 1998, Google's PageRank was an innovative algorithm that calculated relevance based on counting back links instead of parsing the word counts in embedded HTML text like other search engines. This created a noticeable improvement in quality of results.

Nobody seems to have The Next Big Idea for a better search engine yet. Somebody did a Shown HN of a new search engine based on whitelisted curated domains such as reddit discussions. But there are many technical problems with that (e.g. Goodhart's Law & Hawthorne Effect creates bad feedback loop of gaming the reddit discussions which then poisons the search engine.)

Another technical idea of crowdsourced decentralized search index creates a very slow query engine which is a hard sell when web surfers are used to Google results appearing in less than 1 second.

DDG's idea of "privacy" is interesting, but being (mostly) based on Microsoft Bing's search engine doesn't actually create a quantum leap in better search results.

What's the next breakthrough idea that extracts the good signal from all the noise of a trillion web pages?

Create 3-5 different search algorithms, with names and understandable characteristics, then let me choose which one when searching. You may have some algorithm that prioritizes recency, or rapid changes, or more interaction, and others that prioritize longevity, citations/links, others that prioritize that type of information - prioritize regional or noteworthy events over historical info, etc. Have some algorithms that penalize spam/linkfarms more heavily than others. The git/so spam farms that come up when I search for an error message are simply useless, and would love ways to switch away and try other algorithms, without necessarily having to switch entire search systems.

Have algorithms that explicitly take in to account the search/click behaviour of others. Let me choose to have the search behaviour of friends/colleagues/famous-folks explicitly included or excluded in search results ranking.

Let people easily switch between multiple algorithms, and let 1000 gardens bloom (or something like that).

Also, determining better ways to 'search' non-text stuff. If virtual worlds become a thing, finding ways to 'search' those (whatever that might mean) will be big.

Image search still has a long way to go, IME.

The way it's likely seen by many tech companies is that the more decisions you have to offload to the user, the less advanced your system is. An old car with a lot of knobs and dials and levers is not more advanced than a self-driving car with a single control, select destination. Some people will still prefer the old car, but it will be a niche market.

"You're viewing through FOOBAR lens. Click here to view results from another lens".

Doesn't have to be that complicated.

You can add three more buttons on top of the tabs that are already there, but would their value for more advanced users be worth the extra friction/confusion for average users? And how would it scale if someone wants a fourth, fifth, or tenth "lens"?

Try images.yandex.com: it's vastly better than most others. It's not too good at identifying specific people but it will find prior exact uses of an image (like TinEye) but is especially very good at abstracting what's the abstract subject, color, pose, setting etc in a photo and show you similar.

The question is if such an approach would lure a lot of people away, I could see an entry angle might be a front end to google with an app that does that.

[1] https://brave.com/static-assets/files/goggles.pdf

We now have a couple digital generations of folks who've grown up with 'web', and we've generated billions of 'documents'.

TIME ACCURATE QUERIES need to be mainstream.

"Let me search for FOOBAR and get results that would have been accurate/returned for Dec 11, 2006. Now search for FOOBAR and get results from Dec 11, 2014", etc.

However as for Google, I feel like even merely surfacing a range of relevant filter options can help their search engine, since it's what informed users are already doing by structuring a query when the first one produces crappy results. E.g. These results mostly contain links to Pinterest.com, would you like to repeat the search with these omitted? (or even just being able to collapse these site-by-site.)

Additionally I think modern search could be broken down into a range of sub-tools that exist beyond google or even online. One such example is searching our web history, perhaps browsers could produce a privacy-preserving way of storing information about each webpage and its images that the user can then search later (by either terms or images.)

IMO, all startups have a sweet-spot where they've figured out what the market needs, and they can be profitable, but they haven't yet sucked that bone dry to where they're forced to extend more tentacles to grow their revenue. For Google, for me, that was about the 2007-2012 time frame.

Any other competitor would have the same financial pressures eventually. Long term, it's impossible to "not be evil".

> They stop using it if the results are poor.

These are not laws of nature. There are so many reasons why most people would still use $X even if the performance of $X was underwhelming.

At least for me, I find myself adding “site:Reddit.com” all the time I need a good answer and not just a SEO-tricked ad page. Vector Search could be a technical solution for this. Although it’s fairly new and there is still a lot of research to do.

Seems like with enough actual humans I should never have to see another geeksforgeeks page come up in my search results.

Here's another recent Show HN that demoed a new search engine:

Show HN: Goopt – Search Engine for a Procedural Simulation of the Web with GPT-3

https://news.ycombinator.com/item?id=30443738

I know this was offered as a bit of joke. But it prompted a really thought-provoking discussion and seemed to me to suggest a genuine paradigm shift in search or information organization. It is a novel way to extract a useful signal from all the noise. It brought to my mind "Dr. Know" from Spielberg's AI movie:

https://www.youtube.com/watch?v=x0QkgAuEPbk

Will the signal be a good one? It will need to get better and inevitably will. The thing that worries me: will it even matter?

For example, someone comes up with a 'Finder' thing that's barely a comprehensive directory of the internet, and that, along with 'site search' aka for FB, Amazon, does better for the use case.

Most people are not 'searching' the web, they're often typing URLs or searchers for things they want answers to.

When I type a technical thing I'm really just searching 'stack exchange'.

On mobile, we use apps, which are different than pages.

Apple could come up with something that renders Google less than necessary.

Google will not be replaced directly.

Not really, because Google can do algorithms. For a new competitor to really succeed they have to start with something Google can't do. Perhaps:

- device/platform related. Google owns chrome & android but a competitor could emerge somewhere out of their reach, like only on iPhone, only on airpods, alexa, oculus, etc.

- being worthless to 99% of the market to make it 10X better to a 1% audience (e.g. developers, nurses, or even more niche like college athletes, jewish moms, etc)

- privacy/ad related

Delicious sort of tried it but never executed well on the search side.

In a way it's a personalized PageRank algo with users linking to pages they like and to other users they follow. This always made more sense to me than a global authority ranking.

When looking for a babysitter you want recommendations from friends you trust, not some global ranking of popular babysitters.

Most long tail queries work well enough with Google and indeed most of the benefit of this approach would be in the head where ranking matters more. Furthermore nothing prohibits users from liking domains, not just individual URLs.

To me this is going to be manual ranking, but the biggest issue with manual ranking is not that you have to do it manually, but how do you know if you can trust users who rank pages. The Wikipedia system is a good start but not enough, and the second issue is how do you prevent its business model to kill its quality

Do the algorithms weight links based on visibility or relevance? A link at the top of a sorted content site HN should probably be worth more than one at the bottom, and one buried in invisible metadata somewhere should be worth 0.

But AFAIK Google no longer uses PageRank, or at least not the same version as we know it.

You can't compete with that by building a better search engine. Even if you had the infrastructure, capital, and you offered Apple and Mozilla more money than they get from Google, Google could beat your offer.

The only way someone could beat Google, or even just compete with them, would be if they could make everyone understand that they can change their default search engine and give them a reason to do it and have Google screw up how they respond. I wouldn't bet on it happening.

Meanwhile on Android, there's no practical way to completely move towards ddg for searched info unless you basically get rid of some voice and search features.

Without laws, Google set up an ecology of everything search related hits google one way or another. On Apple there are ways around it because Apple isn't a search company.

Having said that......... Most people don't care. It's why Microsoft pushes so hard for ie. Because they get to set the default which nobody will change.

It's heavy handed, but if you view search indexing as a natural monopoly, it's relatively simple for how effective it would be at curtailing market power.

Huh? I don't understand what this means.

I also think that's why it's very hard for competitors to compete. As soon as one is going to get credibly menacing they'll improve on the quality again to preserve their position.

I think that has happened with McDonald's for instance. A few years ago the quality was at it's lowest, the food definitely started to feel fake, air filled and not satisfying. Since the small local competition of hand crafted hamburger popped up a bit everywhere, and credible international competitors ramped up (5 guys, in n out...) they had to improve the quality. And my impression is that it's been a bit more than a year that the quality dramatically improved, and their hamburgers actually taste like food again.

I think it's simply the quality versus profits cursor and it can be adapted any time.

Huh, maybe I don't eat there enough to notice differences, but on the occasions lately that I've had a McDonald's burger, I haven't noticed the quality being anything but terrible. Not entirely unpleasant, but just not at all a high quality meal, even compared to some of the other fast food burger places.

Organic search results only ever profit Google if they answer the user's question, because it makes that user more likely to use Google again and click an ad in the future. If a user searches something and the top result doesn't answer their question, then Google hasn't done it's job and the user is less likely to use Google again and therefore less likely to click an ad in the future.

Surely the most profitable option for Google is to answer people's questions quickly and easily, whether you think these questions are answered by Wikipedia instead of a business site targeted to answering that question is a different story.

I just don't understand how Google's goal of "answering people's questions" is somehow opposite to "providing quality" for users. I think for most of HN, they have a very narrow view that "quality" means antiquated wiki-type sites.

This is simply false. It would only be true if there was a better alternative to Google, which there isn’t.

Obviously the best for Google would be to send the user to a website with the best content and that contains Google ads but that's often not both the case.

It's an optimization problem, and generally you cannot optimize for two variables, profits and user satisfaction.

Maximizing only for profits would probably lead to completely garbage results so they probably have to compromise.

Beyond that, Google’s dominance isn’t going away due to their own “monopoly-like” positioning, their business relationships, the inertia of their massive public adoption, all of these applied within three or four other vital areas (ie: YouTube), their ability to pivot in response to anything novel that appeared to undermine their position, no apparent stories of Google executives hosting puppy kicking parties for the entire company to Satanic Panic everyone away from their products, the problem of promising companies being acquired because the owners (VC or founder or otherwise) are happy to be bought, the general “ick” factor of someone like Facebook attempting to enter the fray, some other things none of us have ever considered, random luck, and the initial conditions of the universe.

You have to have good results against the inconceivably vast amount of content out there, you have to be easy to use, you have to be free, and you have to be able to do all of this and more for the yeeeeeears it would take to wiggle into the space and expand while resisting the pressures above. Doable, wildly improbable.

However, I guess big tech companies have become part of the superpower games (USA vs. China etc.). Breaking up Google might just mean a Chinese company takes over. Can't trust the other governments to enforce similar market conditions.

So yeah, like you said, conditions of the universe :)

Well-crafted arguments showing where Google (or ilk) are anti-competitive are likely to gain some traction.

In my mind it shouldn't make sense to found a company with the explicit goal of being purchased by one of the tech giants in a few years.

Many are never even really trying to get s sustainable business model and venture capital is fueling this machine.

One point of view is that it’s a more efficient way for the tech giants to develop new features. An internal team trying to do greenfield work will inevitably be slowed down by bureaucracy, where a startup can iterate more quickly without all the friction of things like performance reviews, HR exercises, and if I’m being cynical, pesky issues like user data protection frameworks.

It’s risky, but the payoff for founders is significantly larger than what an equivalent employee would get for leading an internal project.

You can break up the phone systems because the child companies can provide similar levels of service. But how do you break up a single search algorithm?

There is an obvious problem here - companies who don't advertise with Google could easily find themselves blocked from appearing on the first pages of Google search results. Would anyone really be surprised to find that the internals of the Google search algorithm have a weight factor that assigns better scores to sites that advertise with Google? Hence requiring Google to expose the internals of the algorithm to public scrutiny really seems like the only way forward. Yes, those are 'trade secrets'. So are the contents of your favorite hot sauce, but government regulators require those contents to be displayed on the product label (for good reasons).

On top of all that, Google's also under pressure from governments and media corporations to push their information content (aka propaganda, influence, etc.) to the top of search results, burying anything like independent content in those areas (world events, domestic politics, etc.) far down. This is particularly obvious on Youtube incidentally, but Google has the same problem. Some of this can be avoided using the 'verbatim' option and some interesting word choices in your search string, but it's a fairly tedious exercise.

Again, exposing the internals of the Google search algorithm would be interesting here, as it seems clear certain 'authoritative sites' are assigned better scores in the search ranking - not because they have more backlinks or more accurate content, but merely because of governmental and media pressure.

https://www.quora.com/Is-it-possible-to-beat-Google/answer/M...

A few competitors such as you.com are in the works. I think there’s a lot of potential, but the ranking algorithm is going to need to be decentralized (aka user-configurable) in some way. This configuration would also need to be sharable, maybe something akin to an App Store, because not all users can be asked to be technical enough to configure everything.

There is an enormous amount of wasted value stemming from the disconnect between what Google thinks the user wants to see, and what the user actually wants to see.

If I had to guess why this happened, I’m betting that Google and others had a notion that AI and other advances could eventually approximate a user’s needs to a degree that would keep up with the demand. But the monopoly set in and there was no longer an incentive to innovate that far. I’m sure Google tried, but not to a degree that would have happened if the competition had really stepped it up, or even existed. However, I’m sure by now there’s less optimism that the above effort is even a solvable problem, or one worth solving. One-size-fits-all is extremely suboptimal, but that paradigm is Google at its core. Without a paradigm shift, it’s diminishing returns all the way.

What Google is good at, however, is monetizing what they do have. If you’re trying to challenge them in the monetization game, with the same business model competing for the same customers (advertisers), you most certainly will lose. Seriously, don’t do that. Get to unicorn status first.

Focus on the tech, make sure the business model isn’t competing with the giants, and say no to any buyout offer from Big Tech.

Results being what Google wants the user to see seems to be the real problem.

Youtube for example in the past would show suggestions from small channels that were interesting and fun, but now you can't even find them in search using an exact video title. I've seen Google News completely shut out a breaking news story for half a day and the only reason I can think of was the ranking AI saw some 'bad' words in every story - even from NYT/WaPo, whose articles were among the many dozens on Bing News. Presumably they've applied their anti-"I hated it" ideas to Search as well.

So now is a great time to compete with Google because they're handicapping themselves and even though they could just turn off the bias and return good results, they probably would rather lose than stop trying to influence users.

What do you mean? What configuration options would you like?

Suppose you're a gamer - You could configure search results related to gaming to be limited to games you own, games you want, the subset of platforms or genres you care about, games your friends own, and so on. It wouldn't be a priority for most search companies to make something like that, but it would certainly provide some value for certain people. Decentralized configuration would allow for communities with given interests to configure and distribute customizations for that audience, in a way that would appeal to that audience. It's not that farfetched of an idea, very realistic both from a tech and adoption POV.

You.com is starting the building blocks of such a thing, but I don't know how far they're going to push the envelope. https://about.you.com/apps/

Can you elaborate?

"Decentralized configuration would allow for communities with given interests to configure and distribute customizations for that audience, in a way that would appeal to that audience."

> This configuration would also need to be sharable, maybe something akin to an App Store, because not all users can be asked to be technical enough to configure everything.

Or, as with any other software, there's this thing called a default, which usually works for the average user.

Having results come up that are also explicitly labelled "You are viewing results through the FOOBAR lens - click here to use another lens".

Our results are already skewed by factors that we don't have much visibility in to - being more explicit/transparent about it, while giving people some ability to influence/change the lens they view results through would be useful.

However I just can't rely on the search results. It's rare for me to get a decent result for a search so I switch between chrome. I'm not sure why because I thought DDG used Google search.

It's 2022, and Google search is just as shitty as their GDrive search.

Or, roughly, "The goals of the advertising business model do not always correspond to providing quality search to users"

- Ernest Hemingway, paraphrased

In fact, great organic search could lower ad revenue. I imagine there's some balance there.

The high number of posts on the subject suggests real decline, and stopping that decline won't be helped by a rose-tinted assessment of the current situation.

Maybe I don't have very high standards, or maybe it is the way I write search queries, but I almost only use DDG and the results are good enough for me. I just find what I am looking for. There usually is just no reason for me to use something else, so why would I want to put my privacy into the lion's cage?

The few times I landed on Google's result page during the last couple of years, I found the results more confusing than helpful. Maybe I unlearned how to use Google.

We assume people in general are dissatisfied with Google, because there are blog posts about it, and some people on HN agree, and because we have certain strongly held beliefs and technical knowledge that lead us to this conclusion. What we have not validated is whether people outside the tech community are as dissatisfied as we are. If they aren't willing to switch, then there won't be a notable competitor to Google.

We know that Big Tobacco maniuplates their formulas to make things unnaturally addicting and are incredibly bad for people's health. People continue to smoke.

We know that processed foods are just not healthy for people, yet because of it's readily being available and cheap prices, people continue to eat it.

We know that the Earth is not flat, but people continue to believe that it is.

Just having the knowledge that [Google|Facebook|TikTok|$socialMediaPlatform] is manipulating what you see/read/consume in ways that benefit them and at times is actively harmful to the users doesn't mean people will stop.

https://kagi.com/

How much will Kagi cost?

Kagi will be completely free during the beta-test period for all users. Once we officially launch, we are dedicated to providing the best possible search experience to our users at a reasonable price. We plan to offer entry level plans for as low as $10/month, unlimited plan at around $20-$30/mo as well as have bundles (to include Kagi email and other services), organization/team plans, family plans and annual payment discounts available.

We understand that for some user this may sound prohibitively expensive, but unfortunately we are not in the position to set the price point by consensus or market expectation, but by realistic cost of providing the service at a given level of quality in a way that potentially ensures sustainability and serving our customers long-term.

You're going to need a quantum leap improvement over Google to unseat their positioning. It has to be very substantial to overcome all the various moats they have, not least of which is consumers being used to using Google, the brand awareness.

The next great search engine will emerge from a niche and conquer one segment after another from there. It won't be a massive general search engine that shows up one day (which is what the Google watchers have been waiting for forever - that new behemoth comprehensive competitor is never going to arrive fully formed). There's a decent possibility consumer Web search will be a later stage addition to said new niche competitor, consumer Web won't be its primary or initial target. They'll add on general consumer Web search as a "we might as well" offering once they conquer enough niches.

I love this because I want it to be true. However, iirc Google wanted to sell itself to Yahoo! for a million dollars and about five years later again for a billion dollars.

I know there are quite a few millionaires here but for me, five million dollars would change my life. I can’t imagine being able to turn it down.

Even if by some lucky circumstance Google doesn't become aware of the new kid conquering their beachhead, as they start down this expansion path Google will acquire them or throw resources at competing. A deliberate strategy of only going after segments that Google avoids (like social) almost inevitably attracts the attention of some other MANGAM.

Still, given enough dice rolls someone will eventually have a streak, but we'll be waiting a while for that to happen.

I realize I am a sample size of one, but I run this exercise about twice a year, and sadly always go back to Google.

As much as people claim otherwise, it's not that Google has no competition. People are trying! Google's product is legitimately good, despite the widespread perception of declining quality.

Private, anonymous and independent search index. Generally, search has a lot of attention on it right now, lots of great efforts to compete with G.

https://www.justice.gov/opa/pr/justice-department-sues-monop...

The only thing that would have a chance is if Apple developed a search engine and used it as the default instead of using Google. People are busy/ignorant and do not care and will eat whatever search results they are given.

But this might be a risk to Apple based on how many people use Gmail on their Apple phones.

It is not about quality anymore, it is only about market domination. The only thing that will bring about new search engines will be anti-trust legislation.

Google search itself is irreducible, at least in any productive way.

You can know there is a problem without knowing how to fix it. That should not stop you from talking about the problem.

If it's not covered on HN, I'll append "from:reddit" instead (they're not always wise but as at least they're not getting paid per word written, and the upvote system gives me a sense of what real people think). If I want a broad survey of a topic, Wikipedia or some other wiki are usually great. If I want a really deep dive, I'll append "book", usually in conjunction with "from:news.ycombinator". Lastly, I sometimes add a time filter, e.g. "after:2021".

Clearly this is an awkward pattern. But it really works, and I'm having trouble thinking of something that would be meaningfully better. I suspect most people have found ways of using Google that work well enough for them. So I'm afraid Google Search will stick around for awhile.

I tried DDG for a couple of weeks and it was just awful. For almost all non-trivial searches i had to result to !g. I finally switched back after two weeks or so and suddenly I'm much happier with the quality of Google search results.

Obviously this is just one man's opinion.

I think the technical challenges are big enough to make this a really difficult task.

If you're willing to pay for search there is an alternative emerging: Kagi. It's not a Google competitor, though. It's a niche product for people (like me) who are willing to pay a significant amount of money for access to a search engine whose creators make money by providing value to their users rather than by providing their users to advertisers.

IMO Kagi can only appeal to people who care about search enough to pay and don’t search enough to make it too expensive, which is probably a contradiction.

> We plan to offer entry level plans for as low as $10/month, *unlimited plan at around $20-$30/mo" as well as have bundles (to include Kagi email and other services), organization/team plans, family plans and annual payment discounts available.

I don't know where you're getting 5c per search from, but that unlimited plan would cap it at something I could live with and I doubt I'll need it.

I said a single search 'can cost' 5 cents, but in reality it depends on the search and how many filters you apply.

You can view the cost of your searches on your account. Each 'user search' can be multiple 'charged searches', which is why the cost can be >5c for some searches (e.g. if you search Dog, then click images and size -> large this is 3 searches charged to your account not one, and if you click through various options to refine your image search each click is chargable for instance).

If you click on the top result, find out it's not what you wanted so hit back, then click on the second result, this is charged as two searches if your browser prompts a reload - which depends on the page you visit (even though I would class it as a single search).

If you have a better business model, that does not involve building yet another ad-supported search engine, we are all ears!

I don't think most users would think this was 3 distinct searches, if it's just going through the steps you are required to take to get the single search you want.

Secondly, the media would not allow it because many top media executives own Google shares. If a better Google competitor came along, media companies would do everything to suppress information about it so that nobody would know it exists.

The declining quality of Google results seems pretty clear, but only in relative terms. That is, Google now compared to Google in the past. In an absolute sense the service remains pretty great.

A confounding factor is that maybe the quality decline is in fact in internet content in general. That is, Google per se is as good as ever, it's just it's become much more difficult to find good results.

So, simply, the answer is perhaps that no matter the effort a competitor could put in they simply cannot outdo Google at their core competency. Perhaps for fundamental reasons, but even if not in some niche, surely then for sheer funding and scale reasons when trying to expand beyond that niche.

If a competitor to Google emerges, it'll be in some non obvious thing that is not core web search. Basically, when core web search becomes less important and Google can't pivot quickly enough to the new more important thing to compete.

A lot of it started happening when they took away the ability to search for exact terms, etc.

You can still do this. Put quotes around the terms.

But that's it, it was just "there" near the bottom. They put in all this fancy AI understanding and ML and NLP effort but when I go out of my way to tell the search that "this word right here is super important and critical", they just go ahead and append "page.includes(word)" as a filter to their super algorithm. Instead of using that signal to drive the search. No better than when sites used to stuff their html pages with keywords to trick search engines.

#grumpy

Managing to grow enough in an unappreciated search niche while staying under the radar in order to better withstand that kind of eventual attention seems unlikely. Targeting a segment that Google has been burned on (like social search) may help, but that probably just draws attention from a different MANGAM.

Eventually, someone will be both smart and lucky enough to carve out some of the search space, but don't hold your breath, it is going to take so much luck it won't be soon, and may look accidental.

Remember, search is not just web search. It includes vast array of things such as maps, images, videos, knowledge datasets, discussion groups, real-time news and so on. Each of these segments is an enormous effort on its own requiring massive capital investments. Relevance algos these days are mostly driven by thousands of tweaks, ML models and legions of rules. It’s a complex beast that takes thousands of PhDs and years to perfect. Even than performance is pretty subpar than most expectations. So, what chance do you have as a new entrant in same game? The key is that you don’t want to play same game if you want to win.

A lot of companies have came and gone announcing themselves as Google competitor and trying to play same game. Things like DDG survive only because they can lower the COGS by offloading real work to other people like Bing which itself is fine example of what you might be able achieve if you only had a tiny fraction of Google’s budgets. If you consider dollar for dollar capital investment, Bing is actually quite good.

This is not to say there will never be a viable competitor for Google search. I just don’t think it can be through the traditional framework of index serving. No one really knows what other alternatives can arise in future. One very possible thing is language models. If we can figure out how to scale up serving of massive language models that effectively “memorizes” whole index in them, they can provide quality, capabilities and experience that cannot be matched by simple index serving. If this is viable route, I think we are still at least 3-5 years away.

Google owns one of the major mobile platforms, where it is the default search engine. It pays competing Browsers like Firefox to be the default and is so tied in public perception that "to google" is a verb.

Also Google is fast and mostly reliable.

Also Google search integrates with other services like Maps for localized search so that one is torn back to them easily.

And if you were to get close to it they have tons of money to fight you.

The interesting aspect is that outside the broader search domain their approach often enough doesn't work. They didn't get social networking, they didn't get messaging (except mail) ... so they way to beat them likely is to find new segments and occupy that space (like Zuckerberg wants to do with "Metaverse," whatever that shall be)

If we are talking about search then it will likely be something AI based probably operating under a different model than Google search. AI is a huge force probably akin to the Internet.

Today, it’s totally possible for a single person or a small team to build a domain specific search engine that indexes 10s or even 100s of million documents.

Building the index v0.1 that kind of working is not hard. But maintaining the index to handle countless edge cases is tedious and need non-stop investment (eg, infrastructure + paying salary for talents)

Also, a search engine is more than “keyword matching”. You need to do search result ranking. And it gets exponentially difficult to do as # of indexed documents increases.

As a side note, I'm a publisher and Google Discover traffic alone has made me about $1k in the last week alone to just one single article by itself

https://developers.google.com/search/docs/advanced/mobile/go...

Some of the content on there is, however, fairly obvious clickbait, but there's also a super easy (working!) 'not interested' button in there.

1. Data about what the population is interested: More people use Google today, giving Google more data to train its AI with.

2. Server capacity: It's cheaper for Google to store 1 image than for me to store 1 image, because it stores many more images. Same for compute, such as training AI. Google can get one A100 GPU cheaper than I can.

3. Ads: If I had the exact same search quality and even the exact same number of users, Google will make more money from the same operation, because it has more advertisers.

They set up their business well!

Returning blue links is a thing of the past, as the Web of yesterday is long gone. Blue links always were about surfing i.e. following hyperlinks just for the sake of it since the main premise was most of them were of high quality and quickly proliferating.

All that is gone now and the links are a promotional thing how to get paid in one way or another. This is why Google results have been deteriorating, regardless of tens of trillions of archived pages on the Web. Google has lost the principal ranking signal years ago.

The next huge scale smart information system will be based on dense vectors (a few hundred dimensions) such as in AI but the key will be much bigger scale, of (tens of) billions of vectors. Contemporary AI works won datasets 4-5 orders of magnitude smaller, getting bogged down in gigantic transformer models such as GPT-3 with 175B+ parameters that take weeks and millions of dollars just to train. One might wonder what is innate knowledge of such a huge model, and it is not much as one can see for themselves as GPT-3 is now open (until Apr 1).

The future will be based on embeddings that are NOT contextualized i.e. no separate vectors for different senses in superpositions. Such systems will not be based on ads nor tracking as the resources required will be orders of magnitude less than what is currently required at Google.

Speed of access, an Internet browsing engine , contracts with non-GGL browsers, cloud data centers, maps/locations/navigation, fiber cables/internet pipelines, social networks and personal management software, AI/ML research, personal devices OS ...

Impossible is nothing, but any incumbent will likely take in capital linked to Google. GGL is dominant across the world's hundred and something countries. Excpect places where it is banned(China) or shunned(Russia), anywhere it is equivalent to a monopoly of internet services.

From a technical perspective, a challenger can rise up. From an economic standpoint, a challenger would not last long (maybe locally if favored by regulation). Even the most successfull others: Apple or Amazon would not dare take on Google's Ads or Search Engine. Microsoft tried with Windows Phones and Bing, then ended failing to dominate the market. Google is not singular business. It has so many tentacles in the tech sector that wave a web of interlocking tech products. Growth numbers in Search are boosted by integration in all of Google's own products and the rest of internet users and companies.

To answer your question is to figure out what advantage it has on any competitor starting from zero... Sadly, its network scale is too large to allow an equal challenger.

Setting aside all of the specifics related to building and running the engine itself, how do you make money doing it? Unless you are servicing a very specialized and lucrative vertical that doesn't mind paying, everyone else gets their search results for free so you generally can't charge users for it. This typically leaves advertising (which Google arguably owns the market for re: search) and/or selling user data (which is both unpopular with users and depending on jurisdiction, illegal). Figure out a new way to monetize search and you may have something...

This is a problem with today's world dominated by mega-tech companies: to compete in many areas of it you need to essentially be a monopolist in some other area so you can afford to compete in something like search as a loss leader at least until you get established. Look at how much Microsoft spent on Bing[1]... and they got how much of the market? Facebook wasn't initially profitable either, but they had a distinctly different approach to aggregating a pool of data Google couldn't search and users they couldn't monetize.

[1] https://money.cnn.com/2011/09/20/technology/microsoft_bing/i...

  * Google (and all of FAANG) have a stupid high percentage of the worker pool.
  * Google's biggest asset is that they are Google.  Kind of like competing with Coke.  Intrinsic quality of what they do is kind of irrelevant.
  * General lack of value in most things internet.  Google is getting a lot of value out of what they do, but they are having to put stupid levels of manpower into squeezing out that value.

[1] https://search.brave.com

[2] https://brave.com/static-assets/files/goggles.pdf

There is also Kagi

https://kagi.com

Also there is no actual path for them to be profitable (their pricing strategy isn’t viable imo) and they are currently loosing money on every search.

It's not _just_ that: https://kagi.com/faq#Where-are-your-results-coming-from

Anyways, Kagi results vastly outperform Google results in quality for me. I'd happily pay the rather steep price they are currently targeting, should that end up being how expensive it is.

We have a path to profitability. Indeed, the entire business model is engineered to be sustainable. Whether the price will be accepted by the market (we do not need many users to be profitable, maybe around 50,000) is another matter. I am sorry if you do not feel like on. But one thing we will not do, is loose money on every search.

You have a theoretical path to profitability, but as you haven’t yet charged a penny it’s not proven to be viable. I genuinely hope you prove me wrong, and turn on paid subscriptions and it’s a huge success. If it’s not, better to learn that lesson now while you still have a chance to change course.

For instance how can you offer a better maps for PND if you can't elicit traffic data from the density and speed of android smartphones that are around of 80% of phones traveling around the world?

To defy Alphabet you do not need a better Google, you need a different kind of solution, for instance instead of competing in modern web crap to lock down users and surveil them you can propose classic desktop computing with decentralized tools. Try looking for instance at Jami, Retroshare, ZeroNet, you can integrate them in a suite and say "hey, instead of depending on Zoom, Meet or Teams, proprietary services with limits that might change, surveil, etc use this system, there is no SPOF, no service behind".

Some have tried something, for instance DeltaChat seems to be a WA clone, but it's actually a MUA, in that case it doesn't took off much because most people simply do not care, you have to know your public. Starting from CS and humanities courses [1] to attracts students and plant the seed of something new, being prepared to face years before a success simply because no empire born quickly and when some are against you it's even more complex.

The old adage: people have the power, but they do not know how to use it it's unfortunately very true, you can only bend people, use them, to elicit a slice of power :-)

[1] most CS students haven't enough skills to comprehend the world, they just explore few aspects ignoring the rest and that's why dictatorships like STEM, because they help to generate "ancient Greek's 'useless idiots' to be employed" instead of Citizens (disclaimer: I am an engineer, I learnt than personally in years)

There will be a dramatic change in how people ask questions and access information.

Maybe it will be personal assistants. I doubt it.

Perhaps these kinds of Neuralink implants? Could be...

Whoever nails it, will be one of the near-future trillionaires.

Also while the quality of search has declined it is nowhere near the point where casual users would be bothered enough to switch

The implication of OP's question and perhaps the heart of the matter is whether Google is unfairly applying some impediment or anti-competitive behavior. My personal opinion is that they're not. I'm not a software guy so maybe I don't know the facts. I would very much like to be educated on that here, the same way I've tried to educate people here about display industry facts on the ground.

That Google is still at 85% show that people really want to use Google, it is not just "whatever comes with the computer".

If bing was successful at it, for example, it would cause the sentence "I tried to google it but ended up needing to Bing it" to make tangible sense to someone who has used both services.

Even if you create a better search engine, there is too much of a marketing gap to compete on that alone. "use bing because the results are better!" simply doesnt motivate enough people to switch off google.

Quite frankly, it is rare that google does not deliver expected results for the generic use case.

If you have a great idea and access to deep enough pockets to develop it, there's still the problem that much of the information people want to find are behind various walled gardens and not suitable for public search. Incrementally more helpful arrangement of the same publicly available data that Google already has, isn't worth the effort of challenging the behemoth

Google’s death will come not from some upstart search engine, but from many walled gardens that each have perfectly good topical search inside them. We’re already in a case where different kinds of information are checked at different destinations.

Google will keep trying valiantly to more aggressively turn into a knowledge base instead of a search engine, and ultimately will probably settle at being an interactive encyclopaedia.

The internet has moved past search engines. Now if you want information you need to know or figure out where to ask the question.

Can't just buy time on someone else's cloud network and hire a couple of coders to compete with that.

You could throw a lot of money at it but not compete on years of investment.

My brain is swiss-cheese these days, I wish I could remember the name of that one-man startup a decade ago that actually had a lightweight high-speed crawler and was making a serious attempt at competition. Was covered around here several times. But obviously they didn't succeed if I can't remember the name.