--- title: "Product Testing" description: "Product Testing lets you lock in expected calculation results as Test Cases and replay them automatically whenever a Product, ProductBrick, or DataPool changes — so regressions are caught the momen..." url: "https://support.pr-4.orbit.do/en/advanced-features/product-testing" locale: "en" lastReviewed: "2026-05-04" --- # Product Testing Product Testing lets you lock in expected calculation results as Test Cases and replay them automatically whenever a Product, ProductBrick, or DataPool changes — so regressions are caught the momen... > **Note:** Product Testing lets you lock in expected calculation results as `Test Cases` and replay them automatically whenever a `Product`, `ProductBrick`, or `DataPool` changes — so regressions are caught the moment they're introduced, not after they reach a customer. ## Overview Orbit Products are made up of `ProductBricks` and `DataPools`. Even a small change — a price increase, a new diesel surcharge, a tweak to feasibility logic — can ripple through every booking that uses the product. Product Testing turns each saved booking into a deterministic regression test, so changes are validated before they're persisted. A `Test Case` snapshots the inputs of a calculation (the full transport draft) together with the expected outputs (line items and feasibility) and the moment in time the calculation was performed. When you later edit the underlying `Product`, `ProductBrick`, or `DataPool`, Orbit replays every affected `Test Case` against the pending change and asks you to review the results before the save is committed. > **Key highlights:** **Save-blocking by design**: Every change to a `Product`, `ProductBrick`, or `DataPool` runs the full regression suite. There is no path to save without reviewing results. > > **Deterministic by simulated time**: Each `Test Case` runs against the system time at which it was recorded, so it pulls the versions of bricks and pools that were valid at that moment. > > **Cascading validation**: When a `DataPool` changes, every `Product` is tested automatically. > > **Two-phase save**: A soft-save runs the tests and produces a review summary; a hard-save commits the change and updates the expected results (the *ground truth*) for the test cases you keep. > > **API-first**: Every flow available in Orbit MissionControl is also available through the Orbit API, so agents and integrations can author and run tests programmatically. ## Concepts ### Test cases A `Test Case` is a frozen snapshot of one calculation: **Draft Snapshot**: The full transport input (properties, timestamps, shipment data, transport shape) used as the deterministic input. **Assertions**: The expected outputs of the calculation. Today this covers line items and feasibility. **Simulated time**: The moment in time at which the calculation was originally performed. When the `Test Case` runs, Orbit treats this as the current system time, so the calculation pulls the brick and pool versions that were valid then. A `Test Case` belongs to a `Product`. The `Product` is the natural aggregation point because it knows every `ProductBrick` and `DataPool` it depends on. ### Cohorts Test cases are organised into `Cohorts`. A `Cohort` groups the test cases that share the same simulated runtime, typically the test cases that were captured against the version of the `Product` that is currently live. > **Note:** The active group is the **current cohort**. When you publish a new version of a `Product`, `ProductBrick`, or `DataPool` with a future validity date, the current cohort is archived and a new current cohort is created with timestamps shifted forward to match the new validity window. Archived cohorts remain available for inspection. ### Soft-save and hard-save Saving a `Product`, `ProductBrick`, or `DataPool` is split into two steps: 1. **Soft-save**: Orbit runs every affected `Test Case` against the pending change and returns a result summary. Nothing is persisted yet. 2. **Hard-save**: Once you confirm the soft-save results, Orbit applies the change, updates the ground truth for the test cases you kept, and archives any superseded cohort. This split is what makes the feature both UI-friendly and agent-friendly: the same two steps drive the **Review Test Results** modal in Orbit MissionControl and the equivalent flow over the Orbit API. ## Creating a Test Case There are two entry points for capturing a `Test Case`. ### From the Transport Composer (recommended) This is the most natural way to build up a regression suite — capture realistic, present-day bookings as you encounter them. \1. In Orbit MissionControl, open the **Transport Composer**. \2. Enter a transport request and let the products calculate. \3. On the calculated product card, open the action menu and choose **Save as Test**. \4. Give the `Test Case` a descriptive name (for example, *Same-day Berlin courier, 2 stops, 30 kg*) and submit. Orbit captures the full draft, the moment of calculation, and the calculated line items and feasibility as the expected output. The button is intentionally tucked into the action menu so it does not disrupt the booking journey for everyday users. ### From the Testing tab Each `Product` has a **Testing** tab in **Settings → Products**. Use this surface to: \- Inspect the existing `Cohorts` for a `Product`. \- Open the draft snapshot of any `Test Case` to review its inputs. \- Run the current cohort manually with **Run current test cases**. \- Delete the current cohort if it has become irrelevant. If a `Product` has no `Test Cases` yet, the tab points you back to the Transport Composer, where most authoring happens. ## The save flow Saving a `Product`, `ProductBrick`, or `DataPool` always opens the **Review Test Results** modal. The modal walks through three phases. ### 1. Tests run Orbit resolves every `Product` affected by the change, including downstream products when a shared `DataPool` is edited, and runs the relevant test cases in parallel. While the run is in progress the modal shows the running state. ### 2. Results For each affected `Product`, the modal shows a **\{passed} / \{total} passed** count and a list of `Test Cases`, each marked as `Pass`, `Fail`, or `Error`: \- `Pass` The pending change produces output that matches the recorded expected output. \- `Fail` The pending change produces a different result than what was originally expected. \- `Error` The calculation could not complete (for example, missing data or a runtime exception in a brick). You can expand any `Test Case` to view its assertions and inspect the inputs. If a `Test Case` is no longer relevant, for example, because the change deliberately makes its scenario obsolete, hold to delete it before continuing. ### 3. Confirm save If every `Test Case` passes, the save proceeds straight through. If any `Test Case` fails, you have to make an explicit decision: **Update Ground Truth** acknowledges that the new outputs are correct and re-records them as the new expected results; **Keep Editing** cancels the save and returns you to the editor. > **Note:** In other words: a save with failing tests is allowed, but only after you've confirmed that every difference is intentional. The set of `Test Cases` you carry forward becomes the new baseline. There is no silent override. ### Cohort shifts on new versions When the change introduces a new validity window, for example, a `DataPool` valid from the first of next month, Orbit performs two operations on save: \1. The simulated time on every kept `Test Case` (and all timestamps inside its draft snapshot, such as pickup and delivery windows) is shifted forward by the delta between the old and new validity dates. The new `Cohort` continues to test the same scenarios, just under the new version. \2. The previous `Cohort` is archived. It remains visible in the **Testing** tab and is still used if you ever roll back to that earlier version. If the validity shift is not a whole number of weeks, weekday-sensitive logic (for example, a Sunday surcharge) may now resolve to a different day of the week. The simulated date is shown next to each `Test Case` so you can spot this and adjust the validity window if needed. ## Cascading changes A `DataPool` can be referenced by many `Products`. When you edit one, the soft-save resolves every `Product` that uses it and runs all of their current cohort `Test Cases`. The **Review Test Results** modal groups the results by `Product`, so you can scan the impact of a single change at a glance. `ProductBrick` and `Product` edits behave the same way, scoped to the `Products` that reference them. > **Note:** Real World Example: > > A logistics planner at Spaceport Shipping Co. in Paris is updating the diesel surcharge `DataPool` for the upcoming month. The current value is 7 %, the new value is 9 %, valid from the first of next month. > > The planner opens **Settings → Products → Pools**, edits the `DataPool`, and clicks save. The **Review Test Results** modal opens and runs the current cohort against the pending change. The same-day courier `Product` shows `9 / 12 passed`. Three test cases — all of them business-priority deliveries to ACME Ltd in Tokyo — are marked `Fail` because their total price has increased by 2 %. > > The planner expands the failing test cases, confirms the new totals are exactly what the surcharge increase should produce, and clicks **Update Ground Truth**. Orbit commits the new pool version, archives the previous `Cohort`, shifts the simulated timestamps of the kept `Test Cases` forward, and re-records the new totals as the expected results. The next time anyone touches this product, those updated totals are the baseline. ## API Access The full Product Testing flow is exposed over the Orbit API under `/v5/products/{productId}/test-cases`, including: \- Listing, fetching, creating, and deleting `Test Cases`. \- Running the test suite for a `Product`. \- Updating the ground truth after a soft-save. \- Removing individual assertions from a `Test Case`. This means an agent or integration can drive the same soft-save → review → hard-save loop programmatically: trigger a soft-save, inspect each `Test Case` result, decide which ones to keep, and commit the change. The Orbit MissionControl modal is one possible front-end for that flow — your own automation is another. For request and response schemas, see the Orbit API Reference. ## FAQ **Q: Where do test cases live?** Test cases are attached to the `Product`. The `Product` knows every `ProductBrick` and `DataPool` it depends on, so it is the natural place to attach the regression suite. **Q: Can I save a change without running the tests?** No. Every change to a `Product`, `ProductBrick`, or `DataPool` goes through the soft-save / hard-save flow. The tests always run, and you always confirm the result before the change is persisted. **Q: What happens to my old test cases when I publish a new version?** The previous `Cohort` is archived and remains attached to the `Product`. The `Test Cases` you keep are copied into a new current `Cohort` with timestamps shifted forward by the validity-window delta. If you later restore an earlier version, its archived cohort is still there. **Q: My test failed because the simulated date moved to a different weekday. What should I do?** The simulated date is shown next to each `Test Case`. If a weekday-sensitive rule (for example, a weekend surcharge) is the cause, adjust the validity window of your new version so the shift lands on the same weekday — typically by extending the delta to a multiple of seven days. **Q: What kinds of assertions are supported today?** Line items and feasibility are captured automatically when you save a `Test Case` from the Transport Composer. **Q: Can a test case call external APIs?** Tests are deterministic only as long as the `ProductBrick` code does not call out to live external services. Calls to external REST APIs cannot be replayed against a frozen point in time, so we recommend keeping that logic on the platform. But, of course, you may still call external API endpoints. Just make sure to notice that this might lead to flaky regression tests. **Q: Is Product Testing available through the Orbit API?** Yes. The full create / list / run / update-ground-truth flow is available under `/v5/products/{productId}/test-cases`. See the Orbit API Reference for details.