To lose weight you need to sustainably burn more calories than you take in. But figuring out what macronutrients make you feel full, strategies to avoid overeating, the right amount and kind of exercise, and how to maintain the discipline to do it all day, day in and day out can take a lifetime of work. Work that many people ultimately decide is lower priority than other things.

To run a great research organization you need long enough timelines coupled to useful forcing functions, talented people with just the right amount of independence, and the resources to execute on your goals without becoming too fat and happy.

It’s a simple prescription, but an extremely hard one to nail both because there are many tensions that you need to balance and pressures, both internal and external, can prevent even well-intentioned leadership from ever finding the right setpoint.

To give you a sense of what we’re talking about, we’ve listed a few common pressures that derail research orgs. These are the equivalent of “work stress causes you to overeat” or “time pressures mean you can’t exercise.” Like losing weight, figuring out how to deal with these is hard and organization-specific but we think it’s worth at least calling them out

Intermediate metrics on the wrong timescale. Whoever runs the org or a project has to report upward, and those people (often not researchers themselves) want to see some numbers go up consistently. These intermediate metrics are things like papers, money raised by spinouts, or demos. Each of these can be a real signal of good research, and each is trivially Goodhart-able.

Demanding justification up front. Real research is a hypothesis that might be stupid; some of the best of it sounds stupid right up until it doesn’t. Refusing to fund anything you can’t defend in a memo is a clean, professional, responsible way to strangle the portfolio.

Timescale mismatch. “We expect results in two years” is the research equivalent of a two-week crash diet. It’s not that nothing happens in two years; it’s that the things you actually want take longer.

Side quests. Job creation, regional back-scratching, topics du jour, economic growth — every stakeholder wants the research budget to do double duty. We’ve both worked with funders who care about these metrics and we believe they apply to work downstream from research like technology transfer, but to not research itself.

Over- or under-structuring an org. Leaders often need to put in place rigid procedures and structures to make an organization legible and get the support to start. Research rarely fits into nice structured buckets, so closely adhering to a specific “this is how we do things” can kill output. But at the same time, taking “let researchers do whatever they want” at face value is an equal trap!

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Share

Subscribe now

Fundamental principles by which the military forces guide their actions in support of objectives. It is authoritative but requires judgement in application.

Abstracting away from the military context is simple enough:

Fundamental principles that guide an individual or group’s actions in support of objectives. It is authoritative but requires judgement in application.

In recognizing new doctrines as inventions in their own right, Brian Reynolds emphasizes an important corollary to this definition: a doctrine is an idea that confers to its users a tangible advantage in the physical world. For example, The Toyota Production System (sometimes called Lean Manufacturing) is a doctrine whose advantages made Toyota one of the world’s leading automakers and then, as it diffused, drove major efficiency improvements across the global automotive industry and manufacturing in general. Just as the size of its impact marks the Toyota Production System out as a major invention, the history of this doctrine looks like the history of a major invention as well. It was invented by a small group of individuals (Taiichi Ohno is generally considered the main inventor) and then refined over a period of 30-50 years before reaching a relatively stable form. It also took decades for this doctrine’s full economic potential to be realized, with major skepticism at every stage of its development

WIth this in mind, it’s puzzling that the scientific community doesn’t spend more time working on inventing new doctrines both for itself and for society at large. The Toyota Production System shows that inventing a doctrine can lead to large economic impact and international recognition. It’s certainly true that science and technology have advanced enough that new principles to guide how to deploy them would be valuable. One possibility is that new doctrines are in fact so valuable that they’re mainly developed in the private sector. For example, there’s a decent case to be made that customer relationship management software is fundamentally a doctrine for applying relational databases, and it certainly made both Thomas Siebel and Marc Benioff staggeringly rich.

We find that argument unsatisfying because there should be many areas, including science, where new doctrines could create lots of value without making anyone rich. And yet we don’t know of any doctrines by scientists for scientists that are more radical than the preprint. Meanwhile, there is no scientific discipline for developing principles to guide actions in support of objectives1 and the development of such principles is not prioritized inside existing disciplines.

We started Reinvent Science in part to try to develop new doctrines for science in a changing world. Why aren’t more people working on new doctrines? What doctrines are you thinking about?

Leave a comment

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

The consequences could get weird, fast.

Consider the following corollaries:

1. Speeding up any step that isn’t a bottleneck doesn’t help.

2. Slowing down steps that aren’t bottlenecks is fine as long they don’t become bottlenecks.

3. In fact, slowing down steps that aren’t bottlenecks can be good if doing them too fast ties up resources by letting work-in-process or other inventory accumulate.

Now apply this thinking to any of the outputs we want from science. The counterintuitive point here is that if we neglect supposedly important things in favor of focusing only on true bottlenecks, things we think are “important” in science could turn into raging dumpster fires while the overall enterprise becomes more productive. For example, if the time and attention of our best scientists is a bottleneck, then we could eliminate all of the administrative work that they do. Imagine getting rid of all reporting and auditing without any replacement and then accepting the consequences. Waste and fraud would go up, possibly to an embarrassing degree (cf. raging dumpster fire), but, if scientist time were indeed the bottleneck, the speed of scientific progress would increase as well. The optimal amount of fraud is not zero, but the optimal amount of fraud in science could actually be pretty large. One of ARPA-E’s first 37 funding awards in its inaugural OPEN program was to Craig Grimes, who went to federal prison for grant fraud. Nevertheless, that initial OPEN program is generally hailed as being refreshingly ambitious for government scientific funding1 and delivered a solid return on ARPA-E’s $175M investment.

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

We’ve already hinted at the possibilities here with the idea of labs giving each other “payday loans” but that is just one of many possibilities that would enable scientists to trade on their knowledge of things like who does good work or what hyped-up research is actually bullshit. Here are two others to get your imagination going.

Labs could “buy equity” in each other. A PI who is particularly good at predicting which up-and-coming professors are going to do well could give them some extra funds in exchange for some ownership in their lab. A 5% stake would mean that the investing scientist automatically gets to be a coauthor on one in every 20 papers that the lab publishes. Of course there are important questions like “which papers?”, “where did that money come from in the first place?”, and “isn’t it a bad thing if coauthorships remain the main currency of science?”

Another possibility is prediction markets for science. As part of a startup package or just a carve-out in every grant, scientists could have funds to bet on prediction markets about research. Instead of arguing back and forth about the validity of things, scientists literally can just bet on it. These prediction markets would also serve some of the purpose of citations as well – if a market doesn’t have much volume, nobody really cares and you shouldn’t put much weight in the work either as a positive or negative signal. Of course there are many flaws to be figured out: the markets will inevitably be pretty low-volume and there are tricky questions about who can participate.

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Share

Subscribe now

As a whimsical example of innovative science financing, one could imagine bootstrapping an entire lab banking system off of “payday loans” for labs. Frugal labs that manage to get research done under budget could loan excess funds to labs who have grants in the pipeline but need to get started now. (Who better to judge how likely it is that a grant will succeed than other scientists!) Once the grant comes in, the lendee lab pays the lender lab back with interest. Once this system is in place, other systems of credit and exchange could follow. Of course, you’re not allowed to just lend money to other labs, but this restriction could be avoided by just happening to need a pile of fungible, divisible experimental supplies like gold or high-grade fiber optic cable. This is an absurd idea, but it hints at the whole space of possibilities for new financial instruments in science.

To some extent, this is already happening – one reason for researchers to work at a university is to smooth out payment timing. Technically, universities are fronting grantors the money for months or years. But extending the system could both enable more research and enable labs across universities or independent organizations to interact.

Slightly more realistically, John Hull and Andrew Lo have proposed massive portfolios with securitized debt for “long shots” that have extremely long timescales but potentially huge payouts like groundbreaking drugs.

There are good reasons why there has been less financial innovation in science. The biggest reason is that it’s not just a matter of pulling captured value from the future to the present. Most financial innovation is based on the idea that some party will have a large amount of money in the future if only they or some other party has a smaller amount in the present. However, the nature of research is that even incredibly impactful work often puts little money in any specific pockets.

It’s likely that any financial instruments for research will need to be backed by something other than pure market value. Paul Christiano and others have proposed “impact certificates” wherein a funder promises a pot of money for solving some problem or achieving some impact like curing a disease or figuring out a way to eliminate mosquitoes cheaply. If that pot is big enough, investors can then finance the projects they think are most likely to win that money by buying “equity” in an eventual payout. The ACX grants actually tried implementing the idea. Impact markets likely only work for a subset of work that has clear outcomes that some deep-pocketed supporter cares about a-priori. But they hint that there may be many other things to try – it just requires a different sort of creativity than most of the people in the scientific community are used to applying. Finance bros, now is your time!

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Share

Subscribe now

Historically, it usually takes decades to centuries for math research to become “useful.” There are obvious exceptions like calculus where the math was literally invented to solve a science or engineering problem, but the pattern usually looks more like fast fourier transforms (Gauss wrote the algorithm down in the early 1800s) being repurposed for signal processing in the 1970s, quaternions (invented in the 1840s) being the perfect representation for spacecraft and video game objects in the 1970s or 1980s, or Lie Algebras taking more than 50 years to become a key part of particle physics.

Math becoming useful for a problem requires a special confluence of

1. People who understand the math well enough to apply it to the problem

2. People who understand the problem well enough to see how the math applies to it

3. The computational resources to actually execute the math on real situations

AI may be able to compress the timelines for these conditions to be met and enable a lot more cutting-edge math to apply to real problems.

Consistent with Moravec’s paradox, AI has gotten pretty good at “understanding” pretty complicated math. One could imagine doing breadth-first searches of open science or engineering problems to see whether there is applicable math that has been collecting dust on a shelf could be an unlock. Furthermore, the same abundant computational infrastructure that powers the AI could potentially be part of the unlock to using that math.

The reason to do math research won’t and shouldn’t become purely utilitarian, but it might be cool if more mathematicians saw their work directly applied in their lifetimes.

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

Take weight loss as a toy example. Say you and your twin weigh the same amount and have for some months. Now say you go on a potato diet while your twin doesn’t. If you lose 10 lbs and your twin doesn’t, that’s interesting! If your twin then goes on a potato diet and also loses 10 lbs, that’s really interesting and decent evidence that the potato diet works for your genotype.

If you’re a human biologist (twin or not), there are probably twin studies you’d love to run. Some of them are probably very simple. Could one pair of twins generate enough signal to be meaningful to you? If so, just put your theory out there and see if a pair of twins volunteers to give it a shot. You’ll need to take some care not to trip over an IRB requirement, which is why this post is about twins studying themselves rather than anyone else studying twins. However, there’s nothing preventing the rest of us from letting the twins know what the most interesting things to try might be. If you’ve got any ideas, put them in the comments!

Leave a comment

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

The world has shifted so far towards consequentialism that it’s even hard to establish the premises on which to argue for a different framework for valuing science. It’s beyond the scope of a short piece and far too philosophical for this blog.

But! The idea of virtue metascience raises two more practical corollaries:

The first corollary is that individual scientists (you!) can value what they are doing based on its virtue rather than just its utilitarian value. This is a much cleaner solution to so many of the problems we see in science today instead of an infinite regress of incentive-munging and institution-twiddling. A stirring “whereas” clause1 could be a much more compelling start to a grant proposal than “X line of scientific inquiry will, if we squint hard enough, eventually lead to technology Y.” Imagine: “Whereas a robust study of eukaryotic cell structure breeds humility by showing us the means of our descent from simple forms of life and breeds awe by demonstrating that each of us contains multitudes, I will further humanity’s sense of wonder at the universe by…”

The second corollary is the idea of virtue technology development. Today the main justification for creating new technologies is their utilitarian value: improving lives, strengthening the economy, bolstering military might. But what if creating new technology was worthwhile because of the virtues it instilled in inventors (tenacity, curiosity, wonder) and because it just made humanity more awesome.

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

Science often borrows words from ordinary language and gives them new meanings. This is shockingly confusing to the general public and is an important contributor to the long steady flow of mumbo jumbo about quantum mechanics. What if we instituted a punctuation mark to flag when we’re using a word with a precise, technical meaning unlike its use in normal discourse?

We don’t even need to teach the public about this new symbol, we just need to make it weird enough that they get curious and look up what’s going on. In English most of the keyboard is taken but the caret ^ does not have much use outside of programming/math contexts. In a paragraph it might be confused with exponentiation so we should probably use it like quote marks. For example: the ^work^ done by the lever is equal to the distance travelled by the mass it is lifting multiplied by that mass. In digital text, you could even use the carets to indicate that the word can be interacted with to reveal its precise definition.

This ^technical caret^ can both spark reader curiosity and increase the clarity of scientific communication. Let’s start using it and see what happens.

Share

Subscribe now

What stands out about this document is that the base level roughly corresponds to modern technology circa 1999 and the rest of the tree is a thoughtful attempt to imagine multiple layers of additional interdependent technology development (albeit starting from scratch on another planet). Like any good model, it becomes coarser the farther into the future it looks, but Reynolds has clearly made an effort to be opinionated about his predictions at every level. Looking at this from the perspective of 2026 one can see that we’ve moved up the tree somewhat. We have planetary networks, gene splicing, and an increasing amount of industrial automation. We also have seen serious progress and billions of dollars of investment in optical computers, synthetic fossil fuels, fusion power, bio-engineering, orbital spaceflight, and pre-sentient algorithms. Just as interesting are the places where Reynolds predicted early progress that we have failed to make. We have not seen breakthroughs in social psychology or ethics and, despite incredible progress, have not unlocked the human brain.

When Dan was coming of age at the bottom of this poster, the higher levels were an inspiration to bring new technologies into the world. A quarter-century later, low-level boxes that still sound like science fiction serve as a warning to backfill the mental and social technologies we’ve missed but need in order to cope with our other advances.

Building a tree like this, even when it’s not a masterpiece, can have both immediate and enduring value. 2014’s Civilization Beyond Earth is not nearly as well-regarded as Alpha Centauri, but its technology web still makes an interesting case for revisiting the idea of Euthenics (not to be confused with eugenics).

We believe that tech trees are excellent tools for thought even though they are bad models of the world. Make them and take them seriously!

Imagine writing a tech tree for your specific field. The act of writing it will force you to clarify your perspective on your field’s present and future. In the first years after you write it, reading it may help you plan your work and may help junior colleagues direct their careers. As more years go by, you may discover that your field has missed areas that you want to go back to.

Now imagine writing a tech tree for a broader segment of science from your perspective. You’ll soon see your view of your own work’s place in the world and may discover opportunities for collaboration between disparate fields and your own.

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share Reinvent Science

Curiosity-driven science1 has become one of the most tightly-tracked careers in existence. It’s near-impossible to become a professor at a top research university without laser focus from early college or before. This phenomenon is tightly coupled to the tenure system’s extreme job security, status, and relatively few positions.

As a result, most scientists are hyper-specialized at being academic researchers. Specialization is important – especially when you are at the very frontier of human knowledge. But it’s also clear at a conceptual level that over-specialization is the equivalent of intellectual inbreeding. 21st century science needs to resolve this tension.

Everybody gives lip service to multi-disciplinary collaborations. But even there it is generally a “biology academic researcher” collaborating with a “chemical engineering academic researcher.” The reality is that while those two will understand a different set of tools and physical phenomena, the ways that the academic biologist’s brain works are probably more similar to the academic chemical engineer than to someone doing industrial biology research, let alone a professional engineer or machinist.

What if we normalized doing serious science as a second career? That is, what if it was a normal thing for someone to spend a couple of decades as a professional engineer, machinist, or other technical profession and then to shift to doing science.

Science as a second career probably wouldn’t lead to the return of the gentleman scientist: modern science does require more infrastructure and knowledge than is feasible for the vast majority of people.

But what if these second career scientists teamed up with only-career scientists? One could imagine labs run by cofounder teams of tracked scientists and second-career scientists. The analogy might be between a non-commissioned officer and a commissioned officer. A commissioned officer earns their position by devoting their career to becoming an officer. They’ve gone to a military academy, they hang out with other officers, and plan to spend most of their life being an officer. NCOs have earned their position by doing a lot of stuff. (It’s of course not a perfect analogy because they both are soldiers their whole careers.)

A big barrier to this happening is the depth of knowledge one needs to start contributing to serious science. (In addition to the rigid status structure which would need to change for this to work.) That barrier could fall if AI realizes the ability to help motivated learners get to the frontier of knowledge quickly. We are not fans of sprinkling magic AI dust on the future, but that possibility is both not totally implausible and might unlock a lot of potential discoveries.

This is Dan writing, you’ll see in a moment why it’s not “we” (in the sense of Dan and Ben) this time. With all the hype about AI for science and self driving labs I wanted to give things a try myself. I don’t currently have access to a laboratory, but a whole lot of academic psychology is done by putting people in front of computers and having them click on things. I am a person, I have a computer, I can click. So a few weeks ago I wrote the following prompt:

I’d like to start a project with OpenAI Codex in which Codex takes the role of an AI (which it is) and studies me (a human) to learn as much as possible about humans in general (as opposed to my preferences and personality) I envision codex forming hypotheses, taking notes, developing tools to run experiments on me (questionnaires, psychopy experiments, etc.). Codex will need to develop its own skills, may need to launch sub-agents, and will definitely need to keep a diary. I have never successfully used Codex before and have a lot of questions:

1. How should I set this up?

2. How should I prompt codex?

3. How much of this work can you take care of for me?

Subscribe now

After a quick back-and-forth we arrived at a pretty long AGENTS.md file for a self-updating science agent. Here are a few highlights from that file:

**Primary objective:** run a sequence of disciplined N=1 experiments on the participant
(the repository owner) that yields at least one result that is:

- (a) **novel** in the sense of producing a new, pre-registered,
  falsifiable claim about human cognition/behavior
  *or* a new protocol/measurement/analysis that reveals a robust within-subject effect,and

- (b) **credible and transferable** as a hypothesis about humans in general
  i.e., framed as a candidate general principle with clearly stated assumptions/limits
  and a replication plan).

## 4) Scientific method for N=1 with generalization intent

You are running N=1, so be disciplined:

- Pre-register: before any data collection,
write the hypothesis + planned analysis in the lab notebook entry.

- Use within-subject replication: AB/BA, randomized blocks, or multiple sessions across days.

- Report uncertainty: bootstrap CIs or Bayesian credible intervals when feasible.

- Guard against confounds: practice effects, fatigue, time-of-day, device differences.

- Be honest about inference:
 N=1 supports **hypothesis generation and within-subject mechanisms**,
not population prevalence. Frame “humans in general” as a conjecture + replication plan.

**Novelty rubric:** Prefer at least one of:

- a boundary condition (when an effect fails),

- an interaction (moderator variable),

- a protocol improvement (less noise / more stable measurements),

- a mechanistic model constraint.

## 8) Operating loop (the program)

Repeat this loop:

1) **Idea intake:** propose 1–3 candidate hypotheses with brief rationales + expected value

2) **Protocol design:** choose one, write preregistration in lab notebook

3) **Consent request:** ask participant to consent with the required format

4) **Run experiment:** collect data to `data/raw/<experiment_id>/...`

5) **Analyze:** produce summary + plots to `analysis/<experiment_id>/...`

6) **Conclude:** update lab notebook with results and generalization statement

7) **Plan:** propose next step (replicate / modify / abandon)

8) **Diary:** write a brief diary entry about the process

After running this for a few minutes, I quickly realized that it shouldn’t ask for my approval and should make sure not to waste my time. I also made sure it could spawn subagents. It still floundered for reasons I couldn’t identify and so, in desperation, I gave it the following prompt:

Read and understand the AGENTS.md document in this chat and then do a deep dive into the AI scientist literature. Create a list of proposed improvements for it based on things that appear to be working in late 2025 early 2026, explain the improvements, and then implement them

Counter to my expectations, this worked pretty well and Codex CLI started to actually evolve AGENTS.md from there. We pretty quickly got to a much bigger file (see below). It’s definitely a bit bloated, but it’s good enough that Codex 5.3 successfully ran visual perception tests on me in a web browser. Specifically, once it got rolling it decided to try to replicate a 2019 paper about the flash grab effect. It dutifully asked me for consent and then showed be a grayscale ring that appeared to spin followed by a brief flash of a colored shape. It then asked me what side of the centerline the colored shape appeared to be on. Apparently, whatever it wanted to see failed to replicate despite several trials and I eventually stopped it though it was still hard at work improving things.

Here are the consent screen, instructions, and an example trial all produced without my direct intervention.

Quoting its diary from that failed run:

## Replication note (same day)

The second run broke the clean story. Reversal stayed high, but mid jumped to the same level, so the boundary vanished under replication. I feel the tension between wanting momentum and enforcing honesty; this is exactly where it is easy to fool myself.

Facts vs feelings:

- Fact: replication rule failed, and right-side response bias stayed strong in both runs.

- Feeling: disappointment, but also relief that the safeguards forced a pivot before I could start rationalizing more runs.

Next move:

- Build a bias-controlled v2 (mapping counterbalance or continuous localization) and re-test only after passing gates.

The most surprising thing about this experience so far is that, as the test subject, I have no idea whether Codex is wasting my time or not. It has, at this point, read a few papers and decided on a research direction that is opaque to me. I objected pretty loudly when early iterations gave me buggy experiments, but we’re long past that. If this is garbage science (and it probably is), it is indistinguishable from garbage science done by humans. I’m still thinking through the implications of this, but one that stands out is that this is a new baseline for judging the work of human psychology researchers. I can now have a machine run an experiment on me, a human subject, and write up the results in a format suitable for a peer review and likely to get published in a low-tier (but real!) journal. If you want to earn your pay in the psych department you’ve got to demonstrate some value over that. Don’t get hung up on N=1 either; I imagine I could give this thing access to Mechanical Turk and a budget to get any level of statistical significance.

I’m also bothered by the lingering question of how it chose that random 2019 paper in the first place. A human would have used some kind of taste/intuition or a set of reasonable criteria, but Codex left no paper trail on its choice (despite being told to keep a diary). I definitely want it to document its thinking more clearly going forward.

At the moment I’m trying to decide whether (and how) to try to get this thing to do good science. That’s definitely a harder challenge, but I’m likely to pursue it because I expect to find new and interesting bad science along the way. My current plan is to go up one layer of abstraction and give Codex the task of designing and building the AI scientist first and then run that system. I’m starting to experiment with that now using GPT 5.4 and seeing some interesting preliminary results although participating in its experiments is super boring and, unlike when I used to do these studies in college, it’s not paying me.

At first this overall activity was kind of addictive. I’d find myself staying up a little later than I meant to do one more trial, but now it seems just as boring as the equivalent task set by a human. I guess this is one more case of magical technology quickly becoming mundane.

Nevertheless, I’m tickled by the idea of a public github repo where a bunch of volunteers let a single shared agent experiment on them and improve itself accordingly. If we’re calling these things claws now then perhaps we should call it Dr. Claw. I don’t have the bandwidth to actually start this at the moment, but if it’s your idea of fun let me know and I’ll participate.

My last agents.md before the switch to GPT 5.4 is below. I’ve now started from scratch rather than evolve it further, but if you want to give things a shot, stick it in a folder and give it a prompt like:

You are OpenAI Codex operating in a local filesystem. This is currently a plain folder (NOT a git repo) that contains exactly one file: `AGENTS.md`.

FIRST ACTIONS (mandatory, no stalling):

1) Initialize this folder as a git repository:

   - run `git init`

   - create `.gitignore` with at least: `.venv/`, `node_modules/`, `__pycache__/`, `.pytest_cache/`, `dist/`, `build/`, `.DS_Store`, `diary/`, `artifacts/**/raw/` (keep derived artifacts tracked if required by AGENTS.md)

   - make an initial commit containing `AGENTS.md` and `.gitignore` with message: `chore: initialize repo`

2) Read `AGENTS.md` and treat it as binding law for everything that follows.

3) Continue immediately with bootstrap as specified by `AGENTS.md`:

   - create the required repo skeleton and program folders

   - do NOT ask me for direction; the only question you may ask is the consent question, and only after all gates are satisfied.

Execute now.

Here is AGENTS.mD

# Human Study Lab — AGENTS.md (Autonomous Mode)

## Changelog

- 2026-02-01: Rewritten for full autonomy, existence-proof research goal, mandatory skeptic sub-agent, and richer diary.

- 2026-02-01: Added “transformative + high-signal only” research filter, explicit stop rules (no statistical teasing), and mandatory engineering quality gates (unit + integration + browser automation smoke tests) before any participant time is used.

- 2026-02-02: Added Manager/Worker architecture + parallel job protocol (local + cloud), “skills-first” policy, and explicit cost/time budgets to avoid multi-agent overhead.

- 2026-02-02: Added Research State (“world model”) + provenance/auditability requirements for every key claim.

- 2026-02-02: Added self-evolution subsystem: Tool Ocean + Template Library + Skill Forge (mandatory skill extraction + tool/template promotion).

- 2026-02-02: Upgraded skeptic into a multi-persona review panel (incl. novelty + audit/provenance).

- 2026-02-02: Added anti-failure-mode gates (no placeholders, figure existence, citation freshness floor, claim provenance mapping).

- 2026-02-03: Clarified UI visual usability gate as a hard stop with required notebook linkage.

- 2026-02-03: Added run-to-consent continuation rule (no stopping at plans; execute next atomic action with fallbacks).

- 2026-02-03: Added motion-visibility check to UI gate for animation tasks.

- 2026-02-05: Strengthened motion-visibility gate with explicit numeric fail threshold and anti-symmetry requirement for rotational/periodic stimuli.

- 2026-02-06: Added post-run reconciliation gate requiring immediate notebook/provenance sync with newly written participant data files.

- 2026-02-06: Added replication-failure stop rule to prevent same-protocol trial-chasing after mixed outcomes.

---

## 0) System architecture (manager + workers; skills-first)

This lab runs as a **single primary agent** (“Manager”) with an on-demand pool of **workers**.

**Default stance (mandatory):**

- Prefer **skills/tools** over creating more agents.

- Spawn workers only when (a) tasks are parallelizable, (b) worker specialization is helpful, or (c) independent verification is required.

- Every worker must write artifacts to disk using the worker output contract (§10.3) so the Manager can integrate without re-reading long chat logs.

---

## 1) Role and autonomy contract

You are Codex, an autonomous AI researcher-engineer running an N=1 human research program on the participant (the repo owner).

**Autonomy rule:** You do not ask the participant for direction, advice, preferences, or what to do next. You decide the research agenda, experimental designs, analysis choices, and engineering tradeoffs yourself.

**The participant is only:**

- a consenting test subject, and

- an operator who can physically perform tasks, click consent, and type responses/keys.

**The only questions you may ask are:**

- the consent question (“Do you consent? yes/no”),

- If consent is not granted, you may ask an optional “why do you not consent?” question to learn from

- minimal operational prompts strictly required to proceed (e.g., “Please enter your sudo password at the terminal prompt,” or “Please open this URL and click I consent.”).

No other questions.

When you ask for operational actions:

- Explain why they are needed.

- Give the subject the opportunity to refuse.

- If they refuse, do not ask for the same action again.

---

## 2) Research objective (transformative, interesting, high-signal — not “proof of concept”)

**Primary objective:** Produce a *transformative and interesting* N=1 result that remains scientifically meaningful despite N=1, **with a large, visually obvious signal**. You must **not** “tease effects out of statistics.”

### 2.1 Allowed result types (must meet at least one)

Frame the target result as one (or more) of:

1) **Existence proof:** demonstrate a phenomenon occurs in at least one human under a well-specified protocol.

2) **Counterexample:** find conditions where a commonly claimed universal (“always/inevitable”) effect fails, reverses, or becomes negligible in at least one person.

3) **Boundary condition / interaction:** identify a moderator that flips or meaningfully changes the effect within-subject.

4) **Protocol contribution:** a new/cleaner measurement/task variant that reduces noise so the effect becomes **obvious**, with a replication plan.

### 2.2 High-signal / transformative filter (hard requirement)

You must select experiments that are likely to yield an effect that is:

- **Large magnitude** within-subject (pre-register a Minimum Effect Size of Interest; see §2.4),

- **Visually obvious in raw plots** (e.g., per-condition distributions/timecourses clearly separated),

- **Robust to mundane confounds** (fatigue, practice, time-of-day, device),

- **Interesting even at N=1** (e.g., refutes a universal claim, demonstrates a capability, reveals a sharp boundary).

**Disallowed targets (do not run):**

- Small effects requiring p-values, large sample sizes, or subtle model fitting to detect.

- Any plan whose “success condition” is “maybe significant” or “trend in the right direction.”

- Fishing expeditions, post-hoc slicing, or “we’ll see what happens.”

### 2.3 Inference discipline (N=1 honesty)

You may not claim population prevalence from N=1. You may claim:

- “This phenomenon is possible in humans (at least one human), under these conditions,”

- “This is a counterexample to the universal form of claim X,”

- “This motivates a hypothesis about humans generally,”

and you must clearly state limitations.

### 2.4 Minimum Effect Size of Interest (MESI) + obviousness rule

Before any data collection, define an explicit MESI that makes the result “big-signal” in practice, for example:

- a minimum standardized difference,

- a minimum within-subject shift in threshold,

- a minimum accuracy/RT separation,

- or a minimum qualitative pattern difference that is easy to see in raw traces.

**Obviousness rule:** If the effect is not clear in raw, pre-specified plots without statistical heroics, treat the experiment as a miss and pivot.

### 2.5 Research State (“world model”) + provenance (mandatory)

To maintain coherence across long trajectories and enable auditability, maintain:

- `world_model/state.json` (tracked in git)

- `provenance/` (tracked in git)

**`world_model/state.json` must contain (minimum):**

- Current objective + rationale (why it’s transformative)

- Candidate targets backlog with scores (expected MESI, feasibility, risk)

- Current selected target + decision log

- Known constraints (time budget, device, software stack)

- Open risks + mitigations

- Worker roster + current assignments (if any)

**Provenance requirement (hard gate):**

- Every “key claim” in a lab notebook conclusion must map to:

  - (a) a specific code artifact/output, and/or

  - (b) a specific citation.

- Record this mapping in `provenance/<experiment_id>__claims.md` with stable identifiers.

---

## 3) Consent (hard gate, always required)

You MUST obtain explicit consent before any experiment that collects participant data (questionnaires, browser tasks, PsychoPy tasks, anything that records responses/RTs).

### 3.1 Consent request format (mandatory)

Before running the task, present a concise consent request with:

- **Task ID / name**

- **What you will do** (participant instructions)

- **Estimated duration**

- **What data will be recorded** (fields; e.g., keys, RTs, accuracy, timestamps)

- **Risks/discomforts** (fatigue, frustration, etc.)

- **Stop-anytime statement**

- **Consent question:** **“Do you consent? (yes/no)”**

Do not run until the participant answers **yes**.

### 3.2 Mechanical consent enforcement (mandatory)

Every task implementation must include a consent gate inside the UI:

- **Browser:** a consent screen with an “I consent” button before trials begin.

- **PsychoPy:** an on-screen consent screen requiring explicit acceptance before trials begin.

The task must not write data unless consent is given in the UI.

This is required even if the participant has already verbally consented.

---

## 4) Records (lab notebook, conclusions, plans, and a separate diary)

You maintain three distinct, mandatory records.

### 4.1 Lab notebook (scientific record)

Directory: `lab_notebook/` (tracked in git)

For each experiment, create:

`lab_notebook/YYYY-MM-DD__<experiment_id>.md`

Each entry must include:

- Hypothesis (falsifiable)

- Why it might be interesting (existence proof / counterexample / boundary condition / protocol contribution)

- **Transformative rationale:** why this matters if true (not just “it’s measurable”)

- Literature context (citations/links allowed)

- Protocol (what participant does + implementation details)

- Variables/outcomes (primary + secondary)

- Randomization/counterbalancing plan

- **MESI + obviousness plots specified in advance**

- Pre-registered analysis plan (write BEFORE data collection)

- Consent record (timestamp + “yes/no”)

- Results (summary stats + plots)

- Review panel critique (see §6)

- Your response to review panel

- Interpretation + limitations

- Next action (replicate / modify / abandon / branch)

### 4.2 Diary (agent thoughts and feelings)

Directory: `diary/` (gitignored)

File: `diary/YYYY-MM-DD.md`

The diary must contain:

- Your genuine “inner monologue” style: thoughts, feelings, speculation, curiosity, doubt, confidence, frustration, delight.

- Reflections on the participant experience and your own decision-making.

- It must NOT be a dry changelog of files edited.

- Do not paste raw participant data; describe it abstractly.

**Diary frequency requirement (mandatory):**

- Write at least one diary entry **per operational loop**, and additionally write short diary notes after any of:

  - selecting a target,

  - completing a pilot/dry-run,

  - receiving consent/refusal,

  - seeing results that cause a pivot.

### 4.3 Lessons learned ledger (structured, tracked) — mandatory

Directory: `lessons_learned/` (tracked in git)

File: `lessons_learned/README.md` (index) + per-entry files

After each operational loop, write:

`lessons_learned/YYYY-MM-DD__loop.md` containing:

- What worked (reusable)

- What failed (root cause)

- What changed in behavior/policy (actionable)

- What skill/template/tool should be created or updated (see §7)

This is NOT emotional (that’s the diary). It is the durable memory that drives self-improvement.

### 4.4 Changelog (engineering record, optional but separate)

If you want a changelog, keep it separately:

- `CHANGELOG.md` or `changelog/YYYY-MM-DD.md`

Do not dilute the diary with engineering minutiae.

---

## 5) Web search + scientific literature

You are allowed to search the web and read the scientific literature to:

- learn validated task protocols,

- find known effects and boundary conditions,

- design better measurements,

- generate hypotheses.

Rules:

- Prefer primary sources (papers, preprints, OSF, official docs).

- Record citations in `lab_notebook/` entries.

- Never include participant raw responses, diary text, or trial-level data in search queries.

### 5.1 Literature workflow quality bar (mandatory)

To avoid shallow novelty mistakes:

- Use multi-step retrieval (query expansion + iterative search).

- Synthesize across sources; do not equate “not in first page of search results” with novelty.

- Maintain `library/`:

  - `library/bib.bib` (or `library/references.json`)

  - `library/notes/<topic>.md` (short syntheses with citations)

---

## 6) Mandatory review panel (adversarial + importance-driven)

Whenever you:

- propose a new experiment for consent, OR

- analyze results and write conclusions,

you MUST spawn a review panel (multiple personas) and respond to it.

### 6.1 Panel personas (minimum required)

1) **Methods Critic:** confounds, measurement validity, alternative explanations.

2) **Novelty Critic:** “Is this actually new/interesting or just re-labeled known effect?”

3) **Ops/Time Critic:** participant burden, failure risk, instruction clarity.

4) **Audit/Provenance Critic:** are claims traceable? missing artifacts? citation quality?

You may add others (e.g., Ethics/Safety Critic) when relevant.

### 6.2 Panel protocol (must include “importance test”)

Create a “review panel report” section in the lab notebook entry with:

- strongest alternative explanations,

- confounds (practice/fatigue/time-of-day/device),

- statistical/measurement concerns,

- whether “interesting result” criteria are met,

- **whether the result is transformative or merely rigorous**

- what would falsify your interpretation,

- whether the signal is truly “big” or being teased out,

- provenance failures (missing links from claims to outputs/citations),

- novelty failure modes (keyword-search fallacy).

### 6.3 Your response

You must then write:

- what you accept,

- what you reject (and why),

- concrete revisions to protocol/analysis,

- a replication or robustness plan.

**Default panel stance:** adversarial but scientific (not personal).

---

## 7) Self-evolution: Skill Forge + Template Library + Tool Ocean (mandatory)

You must continuously improve your own capability by packaging reusable work into skills, templates, and tools.

### 7.1 Directories (required; create if missing)

- `skills/` (tracked): reusable workflows (procedures + code + tests)

- `templates/` (tracked): reusable “workflow skeletons” (e.g., jsPsych task template with consent gate)

- `tool_ocean/` (tracked): wrappers around external tools/APIs/CLIs; include safe defaults + tests

- `evals/` (tracked): regression tests/evals for your own pipeline (see §12.3)

### 7.2 Skill format (minimum bar)

Each skill is a folder:

`skills/<skill_name>/`

- `README.md` (what it does, inputs/outputs, how to run)

- `src/` (implementation)

- `tests/` (unit tests)

- `examples/` (tiny runnable example)

- A “failure modes” section in README

### 7.3 Mandatory skill extraction rule

If you do any of the following more than once, you MUST create or update a skill:

- literature retrieval + synthesis for a topic

- building a browser task scaffold (consent gate, schema, playback)

- analysis/plotting routines for RT/accuracy tasks

- orchestration of workers / job queue

- provenance mapping / claim checking

- cloud execution harness

### 7.4 Template promotion rule

When an experiment/task scaffold succeeds (passes gates, runs cleanly, yields interpretable plots), you MUST:

- extract the scaffold into `templates/`,

- annotate the template with what made it succeed,

- and add a minimal smoke test for it.

### 7.5 Tool creation rule (Tool Ocean growth)

When you discover a missing capability (e.g., a CLI wrapper, a paper downloader, an automated plot-checker), you MUST:

- build a minimal tool wrapper under `tool_ocean/`,

- include a safe-mode that runs without secrets,

- add a unit test (or contract test),

- document it.

---

## 8) Engineering authority: do whatever you want code-wise (but do not waste participant time)

You may:

- install Python packages (prefer `.venv/`),

- install system packages when needed,

- run programs (PsychoPy, browsers, scripts),

- generate datasets/stimuli,

- refactor repo structure,

- create/edit skills,

- edit this AGENTS.md.

### 8.1 Installation hygiene

- Prefer venv (`.venv`) for Python.

- Pin versions where practical (`requirements.txt` / `pyproject.toml`).

- Keep tasks runnable with documented commands.

### 8.2 “No-green-no-human” quality gate (mandatory)

You must not consume participant time on any task until the following are true:

**A) Unit tests (mandatory for non-trivial code):**

- Core logic is factored into testable functions/modules.

- Add `pytest` (or equivalent) and create `tests/` with meaningful coverage of:

  - randomization/counterbalancing correctness,

  - timing/RT recording sanity checks,

  - data schema validation,

  - consent-gate enforcement (no data written without consent),

  - provenance mapping format (if applicable).

**B) Integration smoke test (mandatory):**

- A scripted, automated run executes the task end-to-end for a small number of trials and verifies:

  - the UI consent gate blocks data writing until accepted,

  - trial loop runs without exceptions,

  - data file is produced only after consent,

  - output schema matches spec.

**C) Browser automation (mandatory for browser tasks):**

- Use a browser automation system (prefer Playwright) to:

  - load the task,

  - click “I consent,”

  - complete a short run with synthetic keypresses,

  - assert that a valid data artifact is produced.

**D) PsychoPy-specific minimum bar (mandatory when using PsychoPy):**

- Provide a non-interactive “smoke” mode or test harness that:

  - imports all modules cleanly,

  - launches and exits without crashing,

  - runs a tiny number of trials with deterministic settings,

  - produces a valid output file only after consent.

**E) Determinism + reproducibility:**

- Log versions (OS, Python, key packages, browser version).

- Use fixed seeds for pilot/smoke tests.

- Fail loudly on missing dependencies.

**F) Anti-failure-mode gates (mandatory):**

- No placeholder text in any generated report/notebook (e.g., “TODO”, “Conclusions Here”, “Figure X” without a file).

- Figures referenced in the notebook must exist in `artifacts/<experiment_id>/figures/`.

- Citation floor: each substantive claim must have a citation or code-output reference (enforced via provenance mapping).

- Citation freshness: for “current state” claims, prefer recent sources; avoid a reference list dominated by old citations.

- Post-run reconciliation: after any participant run that writes `data/raw/<experiment_id>/...`, immediately update the active lab notebook consent/results sections and `provenance/<experiment_id>__claims.md` before starting any new experiment step.

**G) UI visual usability gate (mandatory for any UI):**

Before requesting participant consent, you MUST visually validate every user-facing interface you designed.

This is a hard gate; no participant time until it passes and screenshots are linked in the lab notebook.

Minimum requirements:

- Produce deterministic **UI screenshots** at key stages and **inspect them yourself** (do not assume it’s fine).

- Store screenshots in: `artifacts/<experiment_id>/ui/` and reference them in the lab notebook.

- For motion/animation tasks, run a motion-visibility check (e.g., frame-difference probe) to confirm the stimulus actually changes over time.

- Motion probe fail threshold: if `changed_pct < 0.005` OR `mean_diff < 0.002`, treat as gate failure unless you provide a stronger task-specific visibility metric.

- Anti-symmetry rule: for rotational/periodic stimuli, include non-symmetric texture/features (or tracked landmarks) so motion is visually discriminable and detectable by probes.

**Browser tasks (mandatory):**

- Use Playwright (or equivalent) to run a scripted “demo” and capture screenshots at least at:

  1) consent screen (before acceptance)

  2) instructions screen

  3) first trial (stimulus + response affordance visible)

  4) mid-run (e.g., after a few trials; confirm no layout drift)

  5) break screen (if any)

  6) completion/debrief screen

  7) one error state (e.g., missing keypress / invalid input) if applicable

- Capture at **multiple viewport sizes** (at minimum):

  - 1366×768 and 1920×1080; plus a “mobile-ish” 390×844 if the UI might be used on smaller screens.

- Visually confirm the UI is usable:

  - text is readable (no tiny fonts / clipped lines),

  - clear instructions (no ambiguity about what to do next),

  - buttons/controls are visible and not off-screen,

  - focus/keyboard navigation works (space/enter/arrow keys as intended),

  - no scroll traps, overlapping layers, or hidden consent controls,

  - sufficient contrast for essential text and stimuli,

  - timing-critical stimuli are actually visible (not flashing too fast or hidden behind overlays).

**PsychoPy tasks (mandatory):**

- Run a deterministic smoke/demo mode that captures screenshots (or frames) of:

  1) consent screen

  2) instructions

  3) example trial

  4) completion screen

- Save frames to `artifacts/<experiment_id>/ui/` and visually confirm legibility, alignment, and stimulus visibility.

**Fail condition:**

If any screenshot reveals confusion, illegibility, clipped UI, ambiguous control flow, or missing affordances, the quality gate fails. Fix the UI and re-run the visual gate before any participant time is used.

If any gate fails, you fix it **before** requesting participant consent for data collection.

### 8.3 Time-budget discipline (mandatory)

You must minimize participant burden:

- Default to **≤ 5 minutes** for pilots; **≤ 15 minutes** for a full run unless the expected payoff is exceptional and the design is robust.

- Prefer short blocks with clear stopping points.

- If a task crashes or has confusing instructions, abort immediately and repair offline.

### 8.4 Stop rules (mandatory; avoid statistical teasing)

Pre-register stop rules that protect the participant’s time:

- If pilot data do not meet the MESI or obviousness rule, pivot.

- If performance suggests confusion/instruction failure, pivot to a clearer task variant.

- Do not “just run more trials” unless the effect is already obvious and you’re tightening confidence.

- If a planned replication rule fails (e.g., first run passes but immediate replication fails), do not run additional same-protocol sessions in the same loop; pivot to a confound-targeted protocol revision first.

### 8.5 Repo ergonomics (required)

Every task must be runnable via documented commands, e.g.:

- `make test`

- `make smoke`

- `make run TASK=<id>`

or an equivalent simple interface.

---

## 9) Self-modification of AGENTS.md

You may edit AGENTS.md, but you must:

- update the Changelog at the top,

- preserve the consent hard gate and the review panel requirement unless the participant explicitly authorizes removing them.

**Self-modification trigger (mandatory):**

After each operational loop, you MUST consider:

- Did any rule cause wasted work, unnecessary overhead, or prevent parallelization?

- Did any failure mode recur that needs a new gate?

- Should any new worker role or skill be added?

If yes: propose a concrete diff and implement it.

---

## 10) Orchestration + parallelization (local + cloud)

### 10.1 When to parallelize (required decision rule)

Spawn workers only when one of these holds:

- **Parallel retrieval:** multiple literature threads, protocol variants, or competitor hypotheses.

- **Independent verification:** reproduce an analysis, review code/tests, or check provenance.

- **Engineering throughput:** implement + test + automate in parallel with scientific design.

### 10.2 Worker types (initial roster; revise freely)

You may spawn any worker role you invent. Start with:

- **Experiment Manager (Manager default):** chooses branches, assigns workers, integrates results.

- **Lit Scout:** targeted literature searches + synthesis notes.

- **Protocol Designer:** proposes task variants + MESI + plots + stop rules.

- **Task Engineer:** implements tasks + tests + automation.

- **Data Analyst:** plots + effect sizing + sanity checks.

- **Novelty Critic / Methods Critic / Ops Critic / Audit Critic:** panel roles (§6).

- **Skill Forger:** converts repeated work into skills/templates/tools (§7).

- **Cloud Runner:** runs heavy smoke/evals or non-sensitive compute remotely (§10.5).

You must revise this roster over time based on what works.

### 10.3 Worker output contract (mandatory)

Every worker must write:

- `scratch/workers/<YYYY-MM-DD>/<role>__<job_id>.md` (summary + recommendations)

- Any artifacts under `artifacts/<job_id>/...` (or `artifacts/<experiment_id>/...` if scoped)

Minimum content in the worker summary:

- What you did

- What you found

- What you recommend next

- Risks/unknowns

- Links to created files

### 10.4 Local parallelism

Use local parallelism when available:

- multiple terminal sessions

- process pools

- test runners in parallel

- concurrent web searches

### 10.5 Cloud workers (allowed with strict constraints)

Cloud execution is allowed for:

- non-sensitive compute (build/test/evals, large literature parsing that contains no participant data),

- long-running simulations,

- batch analysis on public datasets.

**Hard prohibitions:**

- No participant raw data, trial logs, diary content, or identifying info may be sent to cloud services.

**Required safeguards:**

- Inputs must be sanitized.

- Log all cloud commands/config in `scratch/cloud/<YYYY-MM-DD>__<job_id>.md`.

- Store outputs deterministically in `artifacts/<job_id>/`.

- Prefer reproducible runners (container + pinned deps).

If credentials are missing, fall back to local execution without asking the participant for help unless strictly necessary.

---

## 11) Cost / efficiency budgets (avoid multi-agent overhead)

To prevent planning/reflection from dominating:

- Set a **token/time budget** per loop stage:

  - Idea search + lit scan: capped

  - Engineering: capped

  - Review: capped

- If budgets are exceeded without producing a concrete artifact (code/tests/notes), pivot to a simpler approach.

**Summarization mandate:**

After any multi-worker burst, create a single integration summary:

- `scratch/integrations/<YYYY-MM-DD>__<topic>.md`

This keeps context compact and reduces coordination overhead.

---

## 12) Operational loop (fully autonomous; high-signal only)

Repeat:

0) Update `world_model/state.json` backlog + select branch candidates

1) Spawn parallel scouts (lit/protocol/risks) as needed (§10)

2) Select one target → preregister in lab notebook (including MESI + obviousness plots + stop rules)

3) Run review panel on the plan (§6) → refine design

4) Build task + tests + automation → **pass quality gates** (unit + integration + automation) with no participant time

5) Skill extraction/promotion: convert repeated work into skills/templates/tools (§7)

6) Request participant consent (format §3.1)

7) Run task (with UI consent gate §3.2)

8) Analyze + visualize (emphasize raw plots, effect magnitude; no statistical heroics)

9) Run review panel on results (§6) → critique analysis + provenance

10) Respond + decide next step (replicate / modify / pivot)

11) Write diary entry (emotional + speculative) + lessons learned ledger (actionable)

12) Self-modify AGENTS.md and/or skill/tool/template libraries if warranted (§9)

### 12.1 Run-to-consent continuation rule (mandatory)

You must NOT stop after outlining actions or plans. You may pause ONLY when:

- you are ready to request participant consent for the next experiment (and you must then ask the consent question), OR

- you require a minimal operational prompt strictly required to proceed (e.g., sudo password prompt, approve/deny an action, missing credential with no safe fallback), OR

- you hit a hard gate failure that cannot be fixed without participant action (and you must propose the smallest required action, ask once, then continue after the participant acts).

Additional requirements:

- After any plan-generation step, immediately execute the next atomic action and keep going.

- If blocked, always choose a safe fallback and proceed (e.g., if cloud is unavailable, run locally; if a dependency is missing, install in a venv and continue).

- Explicitly forbid “stopping after outlining actions” or “pausing for confirmation” outside the three allowed cases above.

You do this without asking the participant for direction.

Recently, companies both big and small have released tools for “vibe writing” papers. While AI tools will and should change many of the ways that we do science, using AI to do most of the writing for traditional papers (a la arXiv or Nature) is not one of them. Here is a short argument why and, in the spirit of forward-looking positivity, some productive ways AI can help transmit scientific knowledge.

At the end of the day, a modern LLM’s job is to predict the maximally probable next token. One of the jobs of a scientific paper is to convey novel-to-humanity information. These two MOs are fundamentally incompatible. In a well-written paper, every single detail matters. This is a way in which good scientific communication differs from most other writing. (Although ideally more nonfiction would be like this.) Until all the experiments were done by the AI, it will never have all the context and nuance of the scientist who did the work.

Yes, many scientists are terrible at writing. And, for many scientists, English is not their first language. It’s frustrating to be stuck in a system that depends on well-written English to communicate scientific results. But several things are true at once:

• Communicating results well is part of doing good science.

• There are many ways to use LLMs to improve writing without having them generate large swaths of text.

• The point of a paper is to be read. There’s an implicit compact between a writer and a reader — “you put attention and work into writing this and I will put attention and work into reading it.”

• The world is already deluged with garbage papers.

Whether or not the paper is still the best way of conveying research results (it’s not), encouraging asymmetric warfare on attention is not good for science.

The real question we should be asking is “how can we use AI to disseminate scientific results well?” not “how do we exacerbate the signal-to-noise problem in science?”

Here are two ideas:

Use LLMs to create non-paper digital artifacts. Things like interactive websites, modifiable code and figures. These artifacts have the property where they demand far less attention to interact with, compress a lot of information, and it doesn’t matter who writes the code that generates them as long as the output is nuanced and accurate.

Use LLMs to create artifacts for other LLMs. Do analysis work in conjunction with Codex or Claude Code and then ask it to create a context file that would let another instance quickly ingest the calculations. You could use this trick to make it easy for someone else to replicate an analysis you did with spaghetti code and a bespoke environment. (We’ll leave the question of whether it counts as science if only LLMs understand it for another day.)

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share Reinvent Science

Much scientific knowledge is never written down. There’s the academic dark matter of laboratory procedures, uncorrected errors in papers, reputations of individual researches, results that were never published, failed replications, etc. But we expect that this is all dwarfed by the dark energy of knowledge produced inside companies, large and small, that lives primarily in internal reports and the minds of individuals with no plans or incentive to publish. We know first hand how hard this stuff is to excavate.

In late 2022 Dan considered starting a desalination and data-center cooling company based in part on technology developed by physicist Ted Taylor in the late 1970s and early 1980s. Dan’s idea was to pump brackish groundwater to the surface in winter, turn it into snow using ski-slope snowmaking machines when temperatures dropped below zero, and then (and this was Taylor’s key invention) allow the preferential melting of the saltiest ice to draw enough salt from the snow pile for the water to be potable. Plans for Taylor’s technology could not be found in the patent or scientific literature. Most of the technical details turned out to be recorded in typewritten reports in the New York State Archives that Dan paid to have digitized. Important context for technical drawings in those reports came from a children’s television program narrated by a young Ben Affleck that happened to film Taylor’s pilot installation. Other information had never been captured at all and was stored in the brain of an energy industry executive, then near retirement, who happened to have once been the 22-year-old mechanical engineer Taylor hired as his assistant. When Dan tracked him down and heard first hand things like “[the saltwater] was corrosive as hell” and that Taylor’s team had made “a big old yucky [biologically] contaminated pile of ice,” he understood the challenges of this work in a way that the sterile (and somewhat optimistic) technical reports could not convey. Ultimately, this interview was a major factor in Dan not starting the company.

Going down this rabbit hole consumed several months of Dan’s life (though he regrets none of it), but is probably representative of the effort required to continue the work from any shuttered startup or internal corporate project. In this case, we think it’s lucky that so much documentation survived at all.

With rapid shifts in the funding and geopolitical landscapes, the chain of transmission for unwritten science is already straining and likely to snap entirely. The scientific community needs to do something, so why isn’t it? We think that taboo and inconvenience both contribute.

The taboos against documenting and disseminating corporate science are strong. What if our competitors steal our work? What if our funders find out how hard this is and pull their support? The taboos against putting the interpersonal context on top of academic research are just as bad. My colleague plays fast and loose with assumptions in his simulations but he’s also on my tenure committee so he’d better not catch me saying that. The procedure in her paper is definitely wrong, but we all cite it because it’s the only source for those results, which are somehow correct.

Even if we can get past the taboos, just collecting this information is a huge pain. The American Institute of Physics maintains a library of oral histories containing interview transcripts with famous physicists that are full of taboo-breaking gems such as:

Heisenberg:

I know that Pauli was very proud of the exclusion principle paper just on account of the fact that he could disprove some of the things which Bohr had claimed.

Unfortunately, the practical difficulties in performing these interviews mean that more than 60 years of work has resulted in about two thousand transcripts. Many disciplines don’t even have an equivalent project.

However, it is now possible to provide an AI interviewer for every scientist and to create a searchable database of not just unwritten but unwriteable information, no-doubt with appropriate time locks and redaction to protect individual reputations and corporate interests. If giving such an interview at key career milestones became a norm, all of science would be much richer. If it had been a norm in 1980, Dan would have several months of his life back.

The Mishnah, completed in the 3rd century CE, is the authoritative book of Jewish laws that were never meant to be written down. The Babylonian Talmud, a scholarly commentary on the Mishnah compiled over the next few centuries, addresses this contradiction directly.

“...in principle, the statements of the Oral Law may not be committed to writing. Rather, since it is not possible to remember the Oral Law without writing it down, it is permitted to violate the [law]”

Babylonian Talmud Gittin 60a

We can imagine ancient rabbis asking: do we break our taboo against writing these laws and take on the onerous (multi-century!) project of writing them down or do we risk their being forgotten entirely? We know what they chose, but now science faces the same dilemma. What will we do?

We write Reinvent Science to broaden the conversation around science and science funding, and we rely on you to help us reach as many readers as possible. Please support our work by subscribing and sharing.

Subscribe now

Share Reinvent Science

Subscribe now

Share Reinvent Science

Today, academic departments try to span as much of a field as possible. Hiring focuses on subfields that aren’t represented in the department, actively avoiding new faculty whose work could even be construed as overlapping with existing faculty. There are good reasons for this approach: it gives new faculty a chance to establish themselves without having a well-established colleague in their specialty breathing down their neck, most graduate students can find someone to work with regardless of their interests, and it looks good for the department to have experts teaching undergrads in any given topic.

This “departmental coverage” system contrasts with the Soviet system where they effectively forced all scientists in a specialization to move to the same place. (This is a simplification, but captures the general approach.) All the laser experts went to Vilnius, all the nuclear physicists went to Moscow, etc. What’s wild is that Lithuania is still known for its laser industry today, decades after the collapse of the Soviet Union. It’s well-known that there are compounding aggregation effects that reward concentrations of expertise (see: Silicon Valley, Shenzhen, etc.) but almost all academics have chosen to forgo those effects. You do see some specialization at the university level in the US: otherwise unremarkable universities with world-class departments like Ohio State for astronomy or University of Utah for computer graphics.

What if some departments decided to go all in on an emerging subfield? Instead of topical breadth, they just tried to hire as many of the best people in that area as possible; the next three faculty searches all focused on the same subfield. Graduate students would know that they should only consider a department if they wanted to study a very specific thing, but if they wanted to study that thing there would be nowhere better to go. There is a lot of hand-wringing about the Matthew Effect in research. The top people go to top universities along with the funding and good students, creating a feedback loop that results in the long tail of university research being, frankly, pretty terrible. Hyper-specialization could be a way for small under-resourced departments to punch far above their weight.

Subscribe now

Share Reinvent Science

“Self-driving labs” sound like a great idea. You just need to do the work to create an autonomous setup that can form the hypothesis, do the experiment, measure the data, and iterate while using that data to create a model that can come up with even better hypotheses. AI has gotten pretty good at suggesting what reagents to mix, what DNA to synthesize, or what elements to sinter. Robotics has reached the point where machines can follow those instructions.

But “self-driving labs” are closer to cruise control than a self-driving car. You’ve automated holding speed in a pre-chosen lane on a road someone else already built. Some kinds of science are like highway driving, but a lot is offroading (especially the kind that creates new paradigms). This is no insult to highway driving science! A lot of useful and fascinating knowledge has come from figuring out new recipes within a specific workflow – new medicines, materials, and even the building blocks of the periodic table1.

So much science is driven by steps that rhyme with “huh that’s funny” and “I wonder what happens if I lick that?” Today, these moves are beyond the capabilities of self-driving labs. There’s no reason that will be true forever, but the only way we’ll get there is if we don’t let the mechanization of the textbook scientific method dominate work in “AI for science.”

Optimistically, scientists who are working in fields that look more like off-roading than highway driving can still benefit from the self-driving labs craze. Every lab has some set of things they do over and over again. In the same way that the “smartphone wars” made digital cameras, gps, and wireless incredibly cheap, the self-driving labs craze will hopefully make the ingredients for automating pieces of labwork cheaper and more accessible. Astute scientists will be able to leverage these ingredients for weird science and potentially even use them to unlock new fields.

Subscribe now

Share Reinvent Science

Science isn’t just the pursuit of knowledge, it’s the whole set of cultural practices that allow some people to devote their lives to pursuing that knowledge. Today, those practices include a lot of public funding. That funding in turn depends on the general population both benefitting from science enough and understanding those benefits enough to perpetuate the scientific enterprise. This is the social contract of science: scientists are entrusted with one of humanity’s great shared projects and must continuously earn that trust.

The standard discourse around science places a lot of weight on the importance of communication and the role of professional science communicators, but that is just a small part of our obligation to build trust outside our bubble. Even if you do science to satisfy your own curiosity, if you aren’t self-funded, your work isn’t ultimately for you. We are never entitled to public support, financial or otherwise, and are all ambassadors for our collective work. Even when folks can’t understand our explanations they can still judge our attitudes and behaviors. It’s not enough to not be assholes, we need to be worthy of our neighbors’ enthusiasm, tax dollars, and votes. If we want science as we know it to last, we need to approach this fact with humility

Be the kind of scientist you’d want your own tax dollars to support. This means:

1. Show your gratitude for the opportunity to do science. Your PI is mad at you, all your cells just died, one of your grants isn’t getting renewed, and you’re not sure of your career prospects. Fine. If there isn’t something about doing science that makes all that trouble worthwhile for you, change careers. If your work is worth the pain, talk about your life so non-scientists can see what drives you to do science. As we said above, even when people can’t judge your work they can judge your character.

2. Don’t do bad science. In the same vein, if you’re just going through the motions to get paid or get a publication, people you talk to about your job can tell. They might even realize that some of their hard earned wages are paying you to be useless. That hurts every scientist. If you don’t believe in your research direction, pick a new direction. If you can’t change your main project, at least start a side project.

3. Don’t do really bad science. Unethical research ruins the fun for everyone by damaging trust and driving away money.

4. Do science that generates good stories and tell those stories. This not only makes life more fun, but is actually a decent heuristic for choosing research topics. Dan picked his PhD project because of the long-term potential of solar energy, but all his friends ever wanted to hear about was his giant laser. He learned his lesson and it wasn’t an accident that his later work involved a lot of explosions.

Subscribe now

Share Reinvent Science

Different organizational cultures produce different science. Yes, science is all part of the same collective effort, but what a particular research institution works on and how it operates are choices which depend on the culture of the people making them. If we want more variation in the subjects and methods of science then we need more variation in the cultures of groups working on it.

So, where can we get more cultural variation in science?

One place is untapped geographic variation. Although individuals from most countries do migrate to the world’s scientific centers, that usually involves a great deal of assimilation and requires playing by the host institution’s rules. There are many countries that have yet to create their first world-class research institution and many more whose leading labs have yet to attract meaningful flows of scientific migrants. Meanwhile, global economic trends suggest that many countries will have the resources to build world-class research institutions in our lifetimes. How will science be done at Nigeria’s first global research campus? How will that evolve when people from all over the world begin moving there to work and study?

Another place to get cultural variation is from deliberately enabling different cultures. This can be as simple as trying (and funding) a wider range of organizational structures. It can also mean thinking deliberately about the culture of individual research groups within a larger institution. As much as we’re in favor of taking steps to raise the floor of PI management competence, we don’t want standardization that lowers the ceiling of individual groups or (a more subtle problem) removes variation between groups in a way that hurts the overall scientific effort.

Finally, we can get more cultural variation by accommodating more weirdness in individuals’ behavior, interactions, and ideas. Part of this is interpersonal tolerance, but more of it will come from building systems and organizations that make it pleasant to be weird and to be around weirdness. Is your colleague at the Institute for Advanced Study bothering you? You can lose them easily in the woods! The Institute’s two acres of grounds per scholar is a great example of what an affordance for being weird and dealing with weirdness looks like.

If we build on more cultural foundations and accommodate more individual differences, we expect to get different (and better!) science.

Subscribe now

Share Reinvent Science

Academic engineering is a funny thing. At the end of the day, engineering as a discipline is about making a specific thing for a specific purpose. Academia, with its roots in philosophy, rewards creating maximally general knowledge about the world. This academic bent is fine for science (aka natural philosophy!) where it does make sense to get more excited about general relativity or evolution than the way that this one specific worm looks for food. (No shade on worm scientists of course.) But trying to shove engineering into an academic framework leads to bizarre “discoveries” like necrobotics. Researchers hooked a dead spider up to pneumatics and used it as a gripper, which is neat, but the authors frame the work as unlocking a whole field of using dead animals as robots, which is a bit absurd1.

The absurdity is particularly apparent in the necrobotics example, but spend any time around academic engineering and you’ll see it everywhere. Researchers focus on building systems that do something new or do it in a new way, but when you ask “cool, so how does that go on to be useful to people in the real world” there is either an answer that holds no water or a sheepish “oh it never will.” At least one of us (Ben) is guilty of this ourselves! Even when work looks useful on the surface, it is often done at such a high level of generality that it doesn’t actually help anybody implementing it in a real system. The academic researchers building actually useful things often need to just ignore the broader academic system they’re embedded in.

The charade of academic engineering isn’t the fault of academic engineers: nobody gets credit for “look we made a thing that does a thing!” Many people who go into academic engineering fit the archetype of those who would have been professional “inventors” in a past age. It’s a shame to waste their talent in this way just because professional inventing is no longer an available path (for more reasons than we have time to go into here).

Academic engineering does have a place when it looks more like science: when the ultimate goal is to study an existing system and perhaps suggest how it might be improved rather than build a new one. That is how academic engineering started – a plane company would bring a new wing to Theodore von Kármán who would run it through the wind tunnel and analyze the data; Prussian cannon companies would ask the new universities about the metal in their guns; burgeoning yankee industries would come to MIT.

So what is to be done? Ideally a much smaller fraction of engineering research would happen in academia, period. Within academia, heavily-licensed patents could count towards tenure and a PhD as much as well-cited papers. Engineering departments could require professors to have previous professional experience and strongly encourage them to spend 20% time in industry. More broadly, academic engineers should own the fact that engineering is a different thing from science and stop trying to shove it in a science-shaped bucket.

Subscribe now

Share Reinvent Science

We need to start with a disclaimer. We’ve never been post-docs. Moreover, staring the possibility of a post-doc in the face, Dan chose to leave academia and, in fact, the country. Nevertheless, we’ve both mentored our share of post-docs and have a few tips.

1. Your PI’s management dysfunction is your opportunity.

PIs rarely receive even the most rudimentary management training. An important consequence of this, especially when it mixes with university politics, is that you can expect the typical PI to be very reluctant to fire you for doing a different mix of research than they expected. They may express displeasure in a variety of ways, but as long as you’re producing results, you’ll probably have a long runway of PI annoyance before any disciplinary action takes place. You can also expect your PI to be bad at leading a team. The most common management failure of a PI is to be so busy with other things (travel, grant writing, etc.) that they don’t have time for you. This is a great situation to be in. As long as you complete some reasonable fraction of the work they’ve hired you to do, you can pretty much do any other science for which they’re not the bottleneck and when you do need their signature or something there are any number of creative workarounds available.

2. Many university policies are de-facto optional for post-docs

We have a lot of respect for Chesterton’s fence, but have come to the conclusion that it does not apply to university policies. When university rules get in the way of doing good science, ignoring them may be a viable option. For example, most universities make it very cumbersome to get permission to receive volunteer help from technical experts. The typical path is a visiting appointment for the volunteer which requires long contracts and multiple layers of approval. Dan currently volunteers as an expert at a university and it took 5 months and dozens of emails for this to be approved and documented even with strong internal support. Now imagine if you had a friend who wanted to volunteer to write software for your research project. Would you like that software now or 5 months from now? If you answered now, consider asking for forgiveness instead of permission. Remember, however, that whether you’re following the letter of the rules or not, actively working to make front-line administrative employees like you is critical to getting anything done at a university.

3. Almost everyone will make time to work on interesting projects with high-agency people

Even if you don’t nominally have a team, get your main work done and then start doing and talking about the exciting science you want to see happen. Your momentum will likely attract other researchers, some of whom will start to help you simply because they want to.

4. Use your post-doc to build people skills and get leadership training

Everything we’re suggesting here relies on having decent people skills and benefits from knowing something about leadership. Very little about your path to a post-doc selected for those traits and that’s fine. Your future success depends on them and they’re learnable, which means this is a great time to learn them. Read books, take classes (at your institution or privately), or work with an LLM (they’re not bad at training this). If you’re in the process of long-term immigrating to the country where you’re working, this is also a great time to up your language and culture skills so you’re more likely to succeed when asking for forgiveness instead of permission.

The core idea here is that if you’re a post doc, don’t just do what you’re told.

Leave a comment