Engineers often feel that making a system fast at scale is the most research-like thing they do. Sometimes it is. But the R&D Tax Incentive does not reward difficulty, and a large share of performance work is the skilled application of techniques that are written down.
Turn the target into a test
A performance requirement is a goal: p99 under 50 ms, 100,000 requests per second, half the cloud cost per transaction. Royal Wins [2020] AATA 4320 is a reminder that a hypothesis is a proposition tested by experiment, not an objective. To become one, the target needs three more things:
- A proposed mechanism. The specific design, data structure, partitioning scheme or concurrency model you expect to reach the target.
- A reason it is unknown. Published benchmarks, vendor limits and engineering write-ups either do not cover your scale and constraint or show others falling short. "Nobody here has done it" does not count.
- A disproof condition. The measured result that would show the mechanism does not work.
AusIndustry's software guide puts this as a specific technical hurdle that stops you progressing and that no existing method resolves.
Write the baseline down first. A measured starting point, taken on stated hardware and data before any change, is what lets each later run show whether the mechanism moved the number, and it is the record most often missing when a performance claim is reviewed.
What is routine, however hard
A principle from physical scale-up applies equally to software, and Rand applies it: bigger than before is not uncertainty, and complexity is not uncertainty. In practice the following are usually routine:
- Profiling and fixing hot paths, query tuning and adding indexes as the database documentation describes.
- Caching, CDNs, read replicas, connection pooling, horizontal scaling and autoscaling.
- Upgrading instance types or runtimes and measuring the gain.
- Load testing to confirm capacity. The ATO's alert TA 2017/5 lists system testing as ineligible unless it is an experiment that resolves a hypothesis about a specific technical issue.
Edge cases peculiar to performance work
- Experiments in production. Many performance hypotheses can only be tested under real traffic. That is allowed, but only the period the test actually ran counts, and the rest of the year's operation does not. Some experiments need weeks of continuous observation, for example to see behaviour across full weekly traffic cycles; then the whole planned window is the experiment, as the plan and records show.
- Always-on observability. Dashboards and tracing that run all year serve operations. Monitoring that is not part of an experiment's design is usually neither core nor supporting, even when it later informs one. Observation the plan calls for, such as a dedicated trace or metric captured to evaluate the hypothesis, belongs to the experiment.
- Incidents. Outage fixes are routine work under pressure. A genuine unknown exposed by an outage can be investigated later as its own activity.
- Rolling out the answer. Once a mechanism is proven, applying it to other services is ordinary work unless a new gap appears.
- Load-test rigs. Purpose-built rigs can be supporting for a named experiment.
When the bottleneck is coordination between nodes rather than a single component, see distributed systems; when it is the algorithm itself, see novel algorithms. The compute used for load tests is covered under cloud compute costs, and benchmark records count as contemporaneous evidence only when they were made at the time.