An introductory article that connects algebraic datatypes, combinatorial structures, and generating functions. Includes a bibliography on the fascinating literature around this.
We have started this project where we explore novel interface ideas by taking current interface patterns and experimenting with them to understand what they will be like if we strike a different trade-off with the design choices. Would love to have your feedback and thoughts if you get a chance to check it out.
From what I understand from the demo on the website, it's not a Large Language Model.
The following is how I think it works :
They are probably using diffusion model conditioned on the input prompt to organize the spaces of link.
Search engines in the deep learning era usually embed responses (here links) and queries (here text prompt) in some joint space.
And to get the response they usually do an approximate near neighbor search.
Here they probably replace this neighbor search by a diffusion process.
This is akin to building a learned index. The diffusion process is an iterative process that progressively get you links closer to your query. This diffusion process is kind of a learned hierarchical navigation small world.
Because you need your response to be an existing link at the end of the diffusion process you must project to the discrete space of existing links. There are two schools of thoughts here : If you did your diffusion in a continuous space you can do an approximate near neighbor search in the buckets around to do this projection. Alternatively you can stay in discrete space. You do your diffusion along the edges of a graph. Something akin to train your network to play wikipedia speedrun but on the whole internet.
But diffusion model can be more powerful by not embedding them in the same space (you can do still it but you can do something more powerful).
The problem of embedding in a same space is that with this embedding process you define what is a relevant answer instead of learning the relevancy from the data.
With a diffusion generative model, among other things, what you can do instead to build your database is for each link you read the associated page and you use GPT-3 to generate n queries that would be appropriate to your document (or portion of document). Then you use the diffusion model to learn the mapping query to link with this generated example pair (generated query, link).
Diffusion models solve the mode collapse problem. Aka one query can have multiple different responses weighted by how often they appear in the training data. So they are the natural candidate for building a search engine.
Let's take the wiki-speedrun diffusion approach that is easier to understand.
At query time : You embed the input prompt to a vector.
You start from the wikipedia home page. You use your diffusion model network( that takes as input the current page you are on, and the input prompt vector), and you predict which link to follow (or whether or not you should stop because you have arrived). That takes you to a new current page, and you use your diffusion network again to pick the next link to follow. After doing this n (~20 times?), you have arrived on the relevant page.
Concerning the signals you can use : You have a lot of design freedom. You can basically add any information that you can embed, to pass it as input to the diffusion network. One such signals can be for example a context vector that represent your previous queries.
In a similar way during the organization of the links space, you have lot of freedom to define what is a relevant query. You can score it semantically (if you don't have a lot of real user signal), or if you have already plenty of users that have made plenty of queries, you learn the mapping with the pair (successful query of the user, link).
