Investors Are Whispering About a ‘Crazy’ Fix for Google’s Monopoly

The idea is that the US would require Google to make its Search index publicly available for anyone to use.

Bots and indexes

A Search index is created by a bot crawling the web and collecting keywords and other information from sites. Google then stores all that in a database. When you type a query into Google’s Search engine, the company’s technology zips through this index to find matching and similar keywords, which are tied back to the relevant websites. Then Google shows you the most relevant results.

It’s a bit like how you find a topic in a big book by looking it up in the index at the back, rather than reading the whole book every time you need a specific bit of information.

Google’s Search index is the biggest in the world. Without it, the company’s search engine wouldn’t work very well at all.

Rivals have tried to build alternative search indexes for decades. It’s really hard, partly because websites don’t like lots of random bots crawling all over their content. Sites often block these other bots. But they can’t block Google’s because the company’s Search dominance means that sites would lose massive traffic if they did that. You can see this in action today with several top websites blocking OpenAI’s new Search bot, but not Google’s.

Making Google’s dominant index an open public resource would level the playing field in a dramatic way because rivals wouldn’t have to build their own indexes from scratch.

The Search index as part of the public commons

Wouldn’t this be taking something that belongs to Google, though?

Not really. The Search index is a collection of keywords and other data from websites that have been created by other people and other companies — not Google. Google doesn’t own this stuff. Instead, it makes everyone else’s content available through its Search engine.

So it’s not too much of a stretch to require that Google make the index part of the public commons, that set of cultural and natural resources shared by everyone for the good of society.

Google’s mission statement is “to organize the world’s information and make it universally accessible and useful.” What better way to make this more accessible than by sharing this index of the world’s information without restraint?

An open index could spark new search approaches

If this happened, I could imagine hundreds of new search products taking different, inventive ways to fetch, rank, and present all the information lurking in this huge index of data.

Judge Amit Mehta’s monopoly decision identified Google’s data advantage as a big reason alternative products like Microsoft’s Bing are sometimes lower quality. So, creating a more competitive environment would require more equality in data. In the same way Bing powers DuckDuckGo, Google’s open index could power other rival search engines that could offer different approaches to Google’s search engine.

Year API

This would be done through an open API, or application programming interface, which lets different applications communicate and share information.

Google could charge money for giving index access to large commercial players, while providing the index free to researchers and other smaller organizations, along with maybe a low price for startups.

An MVNO analogy

One analogy of how this can work is from the wireless industry. Mobile Virtual Network Operators, or MVNOs, offer cellular service for your smartphone. But they don’t own their own network of cell towers and other expensive equipment. Instead, they pay to use the networks of Verizon, AT&T, and T-Mobile.

This arrangement has created a huge array of competing cellphone plans and providers, each offering different prices and features. One example of many is Credo Mobile, which uses Verizon’s network but supports progressive social policies.

The 3 US telecom giants have invested billions of dollars building huge networks. They spend billions on upkeep of all this gear, too. They clearly own these assets. And yet, society has worked out a way for these assets to be shared more widely to encourage more competition and consumer choice.

In fact, Google already understands the benefits of making these types of valuable resources publicly available. The company has its own MVNO, called Google Fi, which pioneered the idea of ​​using multiple cellular networks at the same time.

Now, if only this type of inventive competition could be applied to online search.

I asked Google and the DOJ about this idea. They both declined to comment.