Co-author here - first paper from my postdoc accepted to NeurIPS main conference.
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
I was really impressed by the scale of the lacuna tool you created, but then I dug into the niche topics (steganography, covert communication) that constitute my PhD research area and my expectations were tempered.
There’s a lot of noise in the research literature. Lots of papers get published that really don’t contribute much of anything to the field, but the existence of a bunch of papers often sends a signal that some concept is important and needs to be studied.
I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
We haven't done concrete measurements yet, but my intuition is that the quality of Lacuna proposals is a function of density of topics / research directions. Lacuna is most dense around AI topics; for neighboring ones (e.g. bio, physics), I'd join you in tempering expectations.
> I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
Yes, I think this can work. We use a similar idea to categorize what papers have been influential or foundational -- look at the ones that have changed the trajectory (in time) of a research direction significantly.
Thanks for checking it out! We're working on a collaborative discussion / editing feature mediated by Althea (our AI agent). That should provide a way for domain experts like yourself to contribute to improving it. Hopefully we'll have that soon!
ya - I think my intuitions around velcro are firmly intertwined with my meat-space actuators and perceptual equipment.
In other words, I experience high-resolution multi-modal qualia all day and it's really noisy and I think a lot of new ideas and concepts come from obsessively exploring a small subset of those details. Maybe that's just me posthoc rationalizing my phd, though.
I think if an LLM had read any discussions on how best to remove burrs from pet coats then it might eventually get to Velcro - I believe the inventor of Velcro got the idea from burrs?
> "I believe the inventor of Velcro got the idea from burrs?"
I think there's a provocative human-intelligence analog to be found there: "humans will never discover anything that doesn't already exist somewhere in nature. We can only recombine concepts we've seen—never invent truly new ones".
It's said that mathematics is "unreasonably effective" at describing the natural world. But what about the other way around? What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?
I'm working on a piece inspired in large part by William Thurston's "On proof and progress in mathematics"[1]. Should publish it next week.
Part of why math is effective is that its extraordinarily compressible. And you can take those ideas and apply them elsewhere (composition works well). The natural world has a lot of detail and is therefore hard to compress. That's why it's a good source of new ideas. Consider the relation between the McCulloch–Pitts neuron (MCP lol)[2], developed to model biological spiking neural networks, and an inspiration for modern artificial neural networks.
> What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?
Interesting. My intuition here is quite the opposite, that math is vastly overpowered for physics. I don't think we invest a lot of work in figuring out math that isn't useful to describe the natural world.
One part of the explanation is that we focus primarily on AI research, and a substantial chunk of physicists gravitate (heh) towards AI research, so we end up capturing more of their work.
Another part of the explanation could be that (AI +) physics papers are just easier to access, so we end up processing more of those.
Great question - the "universe" of venues we address is centered on peer-reviewed machine learning like ICML, ICLR, NeurIPS, COLM because (a) we have expertise in this area and so can therefore judge the quality and (b) it's all open access.
Therefore, the further you get away from the topic of ML, the less dense our coverage. Frankly, it's a tough to editorial choice to determine which research directions and therefore which proposals to include. ML is quite multi-disciplinary. And people who have a few papers in ML but many in (e.g.) medicine get upset when their profile is sparse. But we can't possibly produce good proposals in medicine due to its tendency to be closed-access.
I did that with Claude research. It collected tons of studies from two or more dozen platforms and crosschecked whether my hypothesis was already considered anywhere and it created this gigantic set of data where it showed me all kinds of fields where the many parts of my hypothesis--that Claude rephrased over and over based on contiguous expansions and elaborations in layman's terms--were already or actively researched and how much of it was proved in theory and/or practice. It turns out the specific connections I made were unique and the underlying theory existed, much of it peer-reviewed and in application. I did not check whether some of those applications, peer-reviews and several thousands of studies were challenged but people were applying the science and there was demand for more and lots of work to be done, requiring specialists from diverse fields.
I just quickly checked the 25,000 new research proposals and found a few that indeed touched my hypothesis and the underlying theories but no direct connection.
Honestly, as so often found in criticism of academia, your approach seems very technical and alienated--pun intended--from the people, their lives and interactions with each other and the world, including tech, and it thus seems that your objective is "estranged" from the very thing you are trying to find.
If Claude's and other LLMs surprise when they find rare occurrences of ideation and inferential trains of thought is worth anything, then that it's an appreciation of training sets lack of humanity and their individual thinking.
Here's some idea I likely will never get to: "linguistic conceptualization of psycho-social reverse pareidolia" or "when, why and how humans were taught to not identify meaningful connections". The "doesn't look like anything to me" equivalent so imperative to the narrative of the TV Show Westworld, created by Jonathan Nolan
and Lisa Joy.
As much of a merry-go-round, cough, it is, science philosophy and personal narratives and perspectives is where scientific novelty resides, so getting LLMs to extract more out of the cattle, sorry, people, might not be such a bad idea, after all.
Thanks! I wish we could take full credit for the paper name, but it's based on a line of work from James Evans (Chicago-based computational social scientist) investigating "Alien artificial intelligences" https://x.com/profjamesevans/highlights
The idea is to allocate academic reviewer time based on the estimated quality of a submitted paper. But critically, you should maintain uncertainty bounds in your estimate. It's kind of like an UCB (upper confidence bound) bandit approach but inspired by cascaded speech translation (where you really should not throw away information at any point in your pipeline).
Congratulations, this is a really cool approach and result. I lead a research team and I'm very interested in tooling for improved collaboration, especially between human and agent researchers.
Do you have any evidence that this approach scales to cross-domain research?
I'd say science is somewhat fractal in this sense. A research area contains topics, those topics have sub-topics, those sub-topics have competing lines that "talk" to each other, etc. So if it works in core ML that means it's working for RL and deep learning and computer vision and NLP -- which is kind of cross-domain.
That said - we believe the method does need to have sufficient data to model the conceptual density. So it'll not work in closed-science areas.
One other thing I will share: we have been working on a way to enable verified academics with sufficient topical domain expertise to engage in small group discussions that update these documents. You can think of it like Althea (our agent) being an area chair and moderating a discussion to determine how to update pages. The reason we think this will work is that you can have an anonymous discussion with the AI system (and be blunt), but we can weigh your expertise when updating the documents.
Oof - it's not like goodfire - we have quantitative results and a clearly defined method!
But yes, we might be bad at marketing. Internally the term we use more often is "concept elements".
The intuition is that if you take a complex treatment of a topic (like an academic paper), you should be able to break it down into some more abstract self-standing ideas. Then, critically, these units are composable and easily embeddable (i.e. they don't have proper names and numbers).
Here's a couple examples of these idea atoms pulled from our paper https://lacuna.tiptreesystems.com/work/the-alien-space-of-sc...: "Scientific ideation often conflates the logical coherence of a research direction with its likelihood of being discovered by the current community. This oversight makes it difficult to identify research paths that are scientifically sound yet unlikely to be discovered due to the existing distribution of expertise and collaboration networks."
Another example: "A generative model estimates the plausibility of concept combinations by treating conceptual units as tokens and calculating their length-normalized log-likelihood. By training on random orderings of these units, the model learns to identify which components are likely to co-occur in a valid research contribution regardless of their sequence."
> I made it to the first figure in the paper before I raised an eyebrow at this term.
I came up with that name, and fair eyebrow. Think of an "atom" as a unit of a working vocabulary (a "token"), not a Platonic form. It's a way of compressing the corpus (paraphrases of the same idea across papers get merged into one symbol, a bit like a tokenizer's vocabulary).
We also talk about some limitations of this framing in page 11 (towards the top).
I love this. My personal completely unsupported theory is we’re cognitively capable of conceiving of less than 1% of what’s true/accurate about the universe, and cognitively capable of understanding 1% of that 1%
- art style
- a doctor might see indications of a health issue that Lisa has
- what it says about the society of that time
- geology of the landscape
- contemporary production of the color pigments
- what the picture does with you when you look at it
- and so on
Good Sci-Fi novels bring along some of these - aliens that have access to different concepts of reality, e.g. the dark forest hypothesis. The hard part is that you actually have to come up with that understanding for writing it, instead of just hand-waving it.
Very cool - the Alien Science work actually had a precursor in image generation. Cultural Alien Sampler: Open-ended art generation balancing originality and coherence https://arxiv.org/abs/2510.20849
I guess it makes sense that art folks like creativity!
I think the way computers developed is kind of an interesting example of this in the real world. Babbage basically realized the idea of a modern computer, but tried to build a too-complicated 'supercomputer' (40 decimal digit precision!) because he didn't know how incredibly useful a much simpler, more limited computer could be. We had to wait another 75 years and spend an incredible amount of money to develop relay and vacuum-tube supercomputers, before realizing that the idea could be scaled down, and something like a PDP-8 / PLC / microcontroller was incredibly useful for industrial control and, with sufficient cleverness, even do a lot of scientific computing.
I think if you went back in time you could pay some power loom engineers to build you a pretty decent 4-8 bit microcontroller operating at 1-5 Hz (comparable to power looms at the time) and computer-controlled textiles would have paid for the computer revolution 75-100 years ahead of time.
There's a plausible version where no matter what weird detours and dead-ends a human society takes on the tech tree, as long as there continue to be wars every few generations, the needed progress keeps getting made and the dead-ends or even just over-reaches get pruned afterward.
You can go down the wrong rabbit holes for a long while in peace-time (e.g. string theory), but war-time is a crucible for correct ideas.
I agree with you, though my own motivation is not centered on war, but (a long tail of) existential risks facing humanity, mostly centered around collective action problems.
A bit of a tangent, regarding innovation. It's interesting to think about counterfactuals in history. There are questions like what if solar panels became widespread 100 years earlier? Could we have avoided global warming and all the wars over coal and oil? (Not possible because lack of purity technology in materials and understanding of matter.) But are there other things where some easy discovery was waiting just around the corner but it wasn't discovered for some reason? Or something was discovered in one society but not in another for a long time, from similar starting conditions. Washing hands certainly is something that one could go back with, probably thousands of years, to bring immediate benefits. Some cultures have done it but some have absolutely refused it for a long time.
And then applying that kind of thinking to today. Looking back to 2026 from 2126, what would a historian facepalm over? Or what is already being done in one field or social scene but not in another, that would be greatly beneficial? The classic recent example of success is applying airline pilots' checklists to surgery. Airline pilots probably are a little bit like aliens, from surgeon point of view. How many people had already tried similar things, but failed with the institutions and people's inability to receive innovation? Maybe with AI tooling we can facilitate better innovation diffusion. It's not without its dangers of course.
I'm on the other hand reminded of a certain Lem sub-story from A Perfect Vacuum, which was about geniuses whose discoveries were too divorced from the mainstream to be useful / worth following upon.
I've filtered the generative proposals for biology [0] because that's a field I'm a bit familiar with. I think it's a bunch of nonsense, tho the confident language really gaslights me.
Yea, so the way this was built causes regions of density (core machine learning topics) to be good and sparse areas like biology to be worse. There are some areas of intersection like Medical Coding, which anecdotally Joseph Paul Cohen (was a mentor of mine and a postdoc with Yoshua Bengio back in the day) reviewed while we were at COLM and he thought was good enough to share around his research lab. All that said, research proposals will be hit or miss.
The reason that biology is sparse is because we are processing full papers from open access venues. Machine learning papers have a long history of open access publication, whereas other areas still have a lot of concepts locked up. I suspect our method would probably not work very well if you only provided abstracts (like AI2 ScholarQA).
Our clusters and therefore our sampling of new ideas is based on density in concept space, and a lot of the key concepts don't end up in the abstract.
Go open science! All that said, the right partnership with a commercial publisher would probably make this work well in that regime.
We built a way to sample diverse ideas in machine learning - ideas that make scientific sense, but that no existing research community is positioned to propose.
In short, the way it works is like this:
1. Take 16,068 LLM papers (NeurIPS, ICLR, ICML, ACL, EMNLP, 2017 to 2025) are broken into 273 "idea atoms". Every paper, and every candidate idea, becomes a set of atoms.
2. Train a Coherence Model: a 42.8M-parameter transformer trained on papers as sequences of atoms. Basically a language model whose tokens are ideas.
3. Train a Cognitive-Availability Model: a dual encoder trained on the publication histories of 39,791 researchers, which predicts where each person could plausibly go next.
4. Use those models to score sampled tuples using reciprocal rank fusion (you want ideas that go together well, but that are not cognitively available to anyone).
We tested it in a bunch of ways. One interesting one was that we took all the papers at the end of 2024, then checked which high-scoring tuples showed up in 2025 papers. It worked well - one examples is an ICLR 2025 oral, goes from rank 1,489 with coherence alone to the top 25 of 1.14M samples.
A funny one: if you ask Claude to come up with an ML research idea, it is often about interpretability. If you then run an autoresearch pipeline with some A100s on the idea, the result tends to be interpretability 8/10 times. Our Alien Sampling method stayed at 0 interpretability ideas even after letting claude cook on them.
We also paid 50 people with graduate degrees in CS/AI to judge the research reports and found that Alien’s reports had the best average quality rank and produced 3 of the top 5 reports.
Oh and then we scaled it way the hell up to a couple million papers and 10,000,000's of concept elements and generated 25,000 new research proposals that you can search over here: https://lacuna.tiptreesystems.com/explore?view=hypotheses
Code (MIT): https://github.com/alejandrohdez00/alien-space-of-science
There’s a lot of noise in the research literature. Lots of papers get published that really don’t contribute much of anything to the field, but the existence of a bunch of papers often sends a signal that some concept is important and needs to be studied.
I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
The one thing I could recommend however is rigging up the mcp (https://github.com/tiptreesystems/lacuna-research-mcp) to your code agent and trawling the database for ideas worth thinking about.
> I find that often a big cluster of papers around a particular topic, none of which seems to move the needle much, is actually an indication that we need a new approach and writing more and more incremental papers just digs deeper into a local minimum.
Yes, I think this can work. We use a similar idea to categorize what papers have been influential or foundational -- look at the ones that have changed the trajectory (in time) of a research direction significantly.
https://blog.karliner.net/posts/chatgpt-and-the-velcro-test/
In other words, I experience high-resolution multi-modal qualia all day and it's really noisy and I think a lot of new ideas and concepts come from obsessively exploring a small subset of those details. Maybe that's just me posthoc rationalizing my phd, though.
I think there's a provocative human-intelligence analog to be found there: "humans will never discover anything that doesn't already exist somewhere in nature. We can only recombine concepts we've seen—never invent truly new ones".
It's said that mathematics is "unreasonably effective" at describing the natural world. But what about the other way around? What if mathematics is unreasonably ineffective at escaping the convex-hull of concepts and patterns found in nature?
Part of why math is effective is that its extraordinarily compressible. And you can take those ideas and apply them elsewhere (composition works well). The natural world has a lot of detail and is therefore hard to compress. That's why it's a good source of new ideas. Consider the relation between the McCulloch–Pitts neuron (MCP lol)[2], developed to model biological spiking neural networks, and an inspiration for modern artificial neural networks.
[1] https://arxiv.org/abs/math/9404236 [2] https://en.wikipedia.org/wiki/Artificial_neuron
Interesting. My intuition here is quite the opposite, that math is vastly overpowered for physics. I don't think we invest a lot of work in figuring out math that isn't useful to describe the natural world.
edit: clarification: the content itself is fine, in fact I'm curious!
But I wonder why when someone types 'medicine' there is only 6 proposals while when someone type 'physics' there is 447 proposals?
One part of the explanation is that we focus primarily on AI research, and a substantial chunk of physicists gravitate (heh) towards AI research, so we end up capturing more of their work.
Another part of the explanation could be that (AI +) physics papers are just easier to access, so we end up processing more of those.
Therefore, the further you get away from the topic of ML, the less dense our coverage. Frankly, it's a tough to editorial choice to determine which research directions and therefore which proposals to include. ML is quite multi-disciplinary. And people who have a few papers in ML but many in (e.g.) medicine get upset when their profile is sparse. But we can't possibly produce good proposals in medicine due to its tendency to be closed-access.
I just quickly checked the 25,000 new research proposals and found a few that indeed touched my hypothesis and the underlying theories but no direct connection.
Honestly, as so often found in criticism of academia, your approach seems very technical and alienated--pun intended--from the people, their lives and interactions with each other and the world, including tech, and it thus seems that your objective is "estranged" from the very thing you are trying to find.
If Claude's and other LLMs surprise when they find rare occurrences of ideation and inferential trains of thought is worth anything, then that it's an appreciation of training sets lack of humanity and their individual thinking.
Here's some idea I likely will never get to: "linguistic conceptualization of psycho-social reverse pareidolia" or "when, why and how humans were taught to not identify meaningful connections". The "doesn't look like anything to me" equivalent so imperative to the narrative of the TV Show Westworld, created by Jonathan Nolan and Lisa Joy.
As much of a merry-go-round, cough, it is, science philosophy and personal narratives and perspectives is where scientific novelty resides, so getting LLMs to extract more out of the cattle, sorry, people, might not be such a bad idea, after all.
The idea is to allocate academic reviewer time based on the estimated quality of a submitted paper. But critically, you should maintain uncertainty bounds in your estimate. It's kind of like an UCB (upper confidence bound) bandit approach but inspired by cascaded speech translation (where you really should not throw away information at any point in your pipeline).
Recommendation: rig up the mcp and have claude or codex trawl the corpus for the kinds of ideas you might care about.
``` claude mcp add --scope user lacuna-research -- uvx lacuna-research-mcp ```
or
``` codex mcp add lacuna-research -- uvx lacuna-research-mcp ```
Do you have any evidence that this approach scales to cross-domain research?
That said - we believe the method does need to have sufficient data to model the conceptual density. So it'll not work in closed-science areas.
One other thing I will share: we have been working on a way to enable verified academics with sufficient topical domain expertise to engage in small group discussions that update these documents. You can think of it like Althea (our agent) being an area chair and moderating a discussion to determine how to update pages. The reason we think this will work is that you can have an anonymous discussion with the AI system (and be blunt), but we can weigh your expertise when updating the documents.
I made it to the first figure in the paper before I raised an eyebrow at this term.
Then I thought of "Already Clean": https://already-clean.goodfire.com
I cant help but think of Plato's forms -
But yes, we might be bad at marketing. Internally the term we use more often is "concept elements".
The intuition is that if you take a complex treatment of a topic (like an academic paper), you should be able to break it down into some more abstract self-standing ideas. Then, critically, these units are composable and easily embeddable (i.e. they don't have proper names and numbers).
Here's a couple examples of these idea atoms pulled from our paper https://lacuna.tiptreesystems.com/work/the-alien-space-of-sc...: "Scientific ideation often conflates the logical coherence of a research direction with its likelihood of being discovered by the current community. This oversight makes it difficult to identify research paths that are scientifically sound yet unlikely to be discovered due to the existing distribution of expertise and collaboration networks."
Another example: "A generative model estimates the plausibility of concept combinations by treating conceptual units as tokens and calculating their length-normalized log-likelihood. By training on random orderings of these units, the model learns to identify which components are likely to co-occur in a valid research contribution regardless of their sequence."
I came up with that name, and fair eyebrow. Think of an "atom" as a unit of a working vocabulary (a "token"), not a Platonic form. It's a way of compressing the corpus (paraphrases of the same idea across papers get merged into one symbol, a bit like a tokenizer's vocabulary).
We also talk about some limitations of this framing in page 11 (towards the top).
If you take a look at the https://en.wikipedia.org/wiki/Mona_Lisa , it can be seen from so many aspects
- art style - a doctor might see indications of a health issue that Lisa has - what it says about the society of that time - geology of the landscape - contemporary production of the color pigments - what the picture does with you when you look at it - and so on
Good Sci-Fi novels bring along some of these - aliens that have access to different concepts of reality, e.g. the dark forest hypothesis. The hard part is that you actually have to come up with that understanding for writing it, instead of just hand-waving it.
If you can't conceive it, then you can't take a percentage of "it", since you've not conceived that "it".
https://x.com/re_skob/status/2113043120128483349
I guess it makes sense that art folks like creativity!
The Road not Taken by Harry Turtledove
https://pastebin.com/aJQfubrK
Makes one wonder if there were technologies that did something similar to humanity in the real world…
I wonder if all the investment in smartphones and LLM’s may have taken away from focus on other things…
I think if you went back in time you could pay some power loom engineers to build you a pretty decent 4-8 bit microcontroller operating at 1-5 Hz (comparable to power looms at the time) and computer-controlled textiles would have paid for the computer revolution 75-100 years ahead of time.
You can go down the wrong rabbit holes for a long while in peace-time (e.g. string theory), but war-time is a crucible for correct ideas.
I agree with you, though my own motivation is not centered on war, but (a long tail of) existential risks facing humanity, mostly centered around collective action problems.
And then applying that kind of thinking to today. Looking back to 2026 from 2126, what would a historian facepalm over? Or what is already being done in one field or social scene but not in another, that would be greatly beneficial? The classic recent example of success is applying airline pilots' checklists to surgery. Airline pilots probably are a little bit like aliens, from surgeon point of view. How many people had already tried similar things, but failed with the institutions and people's inability to receive innovation? Maybe with AI tooling we can facilitate better innovation diffusion. It's not without its dangers of course.
[0] https://lacuna.tiptreesystems.com/explore?view=hypotheses&q=...
The reason that biology is sparse is because we are processing full papers from open access venues. Machine learning papers have a long history of open access publication, whereas other areas still have a lot of concepts locked up. I suspect our method would probably not work very well if you only provided abstracts (like AI2 ScholarQA).
Our clusters and therefore our sampling of new ideas is based on density in concept space, and a lot of the key concepts don't end up in the abstract.
Go open science! All that said, the right partnership with a commercial publisher would probably make this work well in that regime.