rand
Log inEstimate your claim

Integrations

Write the experiment into the pull request, while you are doing it

A pull request can be a contemporaneous experiment record if its description states the technical hurdle, the hypothesis, what was changed and measured, and the result, including failures. Commit messages that name each attempt show the progression. Write it at the time; a record reconstructed months later is weaker evidence.

Reviewed by George WalchFacts last verified

What Rand reads

  • The title and description you write on each pull request, which is where the hypothesis and result should go.
  • Commit messages with the date each was written, up to the first 100 commits in a pull request, which show each attempt in order.
  • The branch name, where a ticket key such as ENG-42 links the code to its plan.
  • The number of review comments, but not their text.

What it becomes in your claim

  • A dated record of each experiment, assessed against your R&D activities.
  • Work dates for the engineer, from the day the pull request opened and merged and the days of the commits read.
  • A revert kept and flagged as a possible failed experiment rather than screened out.

What Rand never does

  • Rand never writes to, edits or deletes pull requests, branches or commits.
  • No pull request is claimed without a person reviewing it.

What the regulator wants a software record to show

AusIndustry's software sector guide says the content of a record matters, not its format. For an experiment it asks for records showing the technical hurdle, the actions taken to overcome it, "including attempts that failed", and the data, analysis and conclusions. business.gov.au adds that records should be created at the time the activity is conducted. Section 355-25 of the Income Tax Assessment Act 1997 frames the same thing as a systematic progression from hypothesis to experiment, observation, evaluation and conclusion.

A pull request is created at the time, dated by GitHub, attributed to its author and tied to the exact change tested. That makes it a natural home for the experiment record, if someone writes the record into it. Two limits are worth knowing. The opened date shows when the pull request was created, not when each line of its description was written, so add a late conclusion as a dated update rather than rewriting the hypothesis; GitHub's edit history on the description shows what changed. And a good record supports a claim without settling it: whether the work is R&D depends on whether the outcome was genuinely unknown in advance and the work progressed systematically, not on how the pull request is worded.

A template for experiment pull requests

Use this only on pull requests that test something you could not predict. Routine work should stay routine.

  1. Hurdle. What stops progress, with the measurement that shows it.
  2. Why known approaches fall short. What you checked first and why it did not settle the question.
  3. Hypothesis. What you expect this change to do, and what result would disprove it.
  4. What was tested. The variant, what was held constant and how it was measured.
  5. Result. The numbers as text, and whether the hypothesis held.
  6. Next. What this result changes about the next attempt.

Good and weak examples (illustrative)

These examples are illustrative, not from any real company.

WeakStronger
Title "Improve search perf" with an empty descriptionTitle "Test: two-stage retrieval to hold p95 under 150 ms at 10M documents"
"Tried some things, faster now""Hypothesis: a coarse filter before re-ranking keeps recall above 0.92. Disproved if recall drops below 0.90"
Commits "wip", "fix", "more fixes"Commits "try bloom filter ahead of LRU lookup", "revert bloom filter: 3% false positives broke recall"
Benchmark screenshot only"p95 212 ms to 141 ms on the nightly 10M set, same hardware, 5 runs"
Failed branch deleted, nothing saidPull request closed with "Recall fell to 0.87; filter too lossy. Next: learned quantisation"
Conclusion buried in a 40-comment review threadReview outcome copied into the description under Result

The stronger column takes a few minutes per pull request, and only on the ones that matter.

Habits that keep the record intact

  • Name the ticket in the branch. A branch such as eng-42-two-stage-retrieval ties the code to the ticket that holds the plan.
  • Keep failures visible. Close the pull request with the reason, or merge a revert that says why. A failed attempt is evidence the outcome was uncertain.
  • Write numbers as text. Screenshots, CI artefacts and dashboards can expire or be hard to search, and business.gov.au asks you to keep R&D records for five years after claiming the expenditure.
  • Know how your tools treat drafts. A draft is still a record, but some importers, Rand included, hold drafts back. If you use Rand, mark an experiment ready for review before you close it so it is read.
  • Keep routine work honest. Do not dress up a dependency bump or a refactor as an experiment. Reviewers notice, and it weakens the records that are real.

What Rand does with these pull requests

If you connect GitHub, Rand reads each pull request's title, description, branch names and commit messages with their dates (up to the first 100 commits, and the first line of up to 20 of them in the record it assesses), so the hypothesis and the conclusion belong in the description, not only in commits. It reads the number of review comments but not the comments themselves, so the description is what carries the record. Bot pull requests are screened out, and drafts are held back until they are marked ready for review. Reverts are kept and flagged as possible failed experiments. These rules run when Rand first sees a pull request; a later edit sends it back for AI assessment without the rules running again.

Each pull request becomes a dated record that Rand's AI proposes to place on one of your R&D activities, for a person to confirm. For the ticket side of the same experiment, see Linear issues as hypotheses and results. The contemporaneous evidence glossary entry, the guide to failed experiments and the R&D Tax Incentive resources cover the rest.

Two longer guides go further: GitHub commits as R&D evidence and the record keeping requirements.

Frequently asked questions

Does every pull request need to read like an experiment?
No. Most pull requests are routine product work and should read that way. Write the hypothesis and result into the ones that test something whose outcome you could not know in advance, and link them to the ticket that holds the plan.
Should we keep the branches of failed experiments?
Keep the record. A closed pull request with a short note on what failed and why, or a revert that says the same, shows an outcome that was uncertain. Deleting a branch is fine once the pull request records it.
Is code review discussion evidence?
It can be, if it records why an approach was tried or rejected. Copy any conclusion that matters into the description, because long threads are hard to find later and some tools, Rand included, do not read their text.

Sources

  1. AusIndustry: Software development sector guide (record keeping for software-related activities)
  2. business.gov.au: Record keeping for the R&D Tax Incentive
  3. Income Tax Assessment Act 1997, section 355-25 (core R&D activities)

The R&D Tax Incentive is a self-assessment program. This page is general information, not tax, legal or financial advice; eligibility depends on your circumstances and you should seek independent advice for them.