Community search engine

Community Search Engines are an approach to build highly relevant search indexes and collaborative content spaces for communities.

The interface is similar to agent chat interfaces – chatting and refining to get what you're looking for – as well as personal notes / bookmark collections¹ – browsing and interacting with "your stuff" to find what you've stored before.

I'm using the label search engine as a term that I think will resonate with the mass market: you go there to find stuff.

What are we trying to fix?

Google has broke the social contract of search. Even without AI slop being placed as the first "result", the results returned lead to top ranking pages that themselves are low quality SEO garbage.

I can't easily find results that are useful or relevant to me any more.

OK, I'll ask my friends for what they think! But, on corporate social platforms such as Twitter, Instagram, or Threads, the algorithm is centrally controlled and can't be scoped to your friends.

You're not sure that your request will be seen by your connections, and your connections won't necessarily see your post.

Discovery and Distribution

This covers two aspects:

• discovery – browsing and finding what you're looking for, ideally with a tune-able, transparent algorithm
• distribution – being able to share "your stuff" in a way that it's possible for others to in turn see it and discover it

For both, we think that Open Social Protocols offer a way forward. We also think distribution is up stream of discovery aka DREAM.

If your information isn't seen because it isn't indexed or shared, or is algorithmically black holed or suppressed, it can't be discovered.

Emerging Tools

DXOS has their DXOS Composer suite. Local first content, LLM models, and composition of a number of different tools, from Discord API connections for indexing, to audio / video transcription.

The Patchwork system by Ink & Switch stores data local-first, cached in your browser. It relies on Automerge for sync. It doesn't have much in the way of a search index at all, and until Keyhive is integrated, doesn't have a permissions system. It can do multiplayer collaboration by sharing links to content.

Groundmist is a proof of concept that uses Ink & Switch's tiny essay editor with local-first, private data (a precursor to Patchwork, it also uses Automerge) and combines it with ATProtocol for public publishing as well as structured data via the AT Protocol Lexicon system.

Tonk has just released the Tonk CLI and TonkbookLM. Uses Automerge. Specifically mentions Obsidian as a source of local data.

How do we build community search engines?

I envision a Personal Notes & Publishing Stack that enables people and communities to collaborate, pool information, and add context and relevance.

Further Reading

Dark Forest and Cozy Web: communities are moving to cozy web spaces, and/or have need for members only content. The open internet has degraded search incentivized by click throughs to show ads, and commercial social platforms

Brazilianization of the Internet: forming communities, which can be considered a kind of commons, trending towards Cozy Web spaces. Does this cause a kind of elitism, and/or further deteriorate public spaces?

Erin Kissane's Against the Dark Forest specifically argues that we need to protect the publis social internet and not all retreat into Cozy Web corners. The entire article is dense and amazing and I could quote every paragraph.

How Algolia uses Electron to improve internal productivity: I’m still inspired by this description of Algolia implementing company wide search across internal tools, with desktop integration. This should be a tool for organizations of all kinds.

Previously

I rewrote the page that was here previously to refactor it extensively.

I applied to SoP 2024 Application with some ideas around the original formulation of using events and digital communities as a way to bootstrap the search engine, without tackling the Personal Notes & Publishing Stack.

Part of community is "flow" (Stock and Flow) and Proto Apps is how I think social graphs / communities / flows of information form.