...
- [ SPH ]We need to make sure that all development is following our process as documented in the SMP and software development tickets in JIRA. We found significant issues during late stage development that would all be done if the checklists in those processes were used during development.
- [ MES ] This needs to be a no-exception thing – on the FOV stuff we had debugged code for TAT-C and assumed that it could simply be converted class-by-class into GmatBase derived classes. However that is where we missed things like variables in Hardware being used by Thruster in a completely different way from how the FOV computations would use them. The rework of Hardware/FieldOfView can be used to illustrate the process as documented in a step-by-step way.
- We need to test the samples earlier and more frequently on all 3 platforms (perhaps weekly starting right before the Release process and continuing with the RCs). NOTE that the samples should be run on a clean build without (or before) running preparegmat, to find missing input data files.
- After editing sample scripts and before committing and pushing, run the entire relevant folder so that interdependent issues are captured. (Example: some scripts were deleted that were #Include scripts. If the related folder had been run, that would have been caught.)
- Some code is missing elements that should be added before being pushed to the main repository. Examples are the block at the top of the file identifying the copyright information, and comments of class parameters and methods. We need to do do better in reviewing code before pushing it because the coder is much less likely to make those changes later in the process.
- Improve regression testing process
- [ SPH ]Mac tests need to be run regularly with results emailed out just like other platforms. We got lucky here but could have found issues late in release.
- [ SPH ]Need to update regression test process to include API tests. API tests cannot be run in the same RunDef as the rest of GMAT regression tests because they require a different mode. We need to create a new master test runner file that sets up multiple run defs
...
The idea here is to have a "super-smoke test" at the beginning of release testing to identify issues early. I think we would have caught the API issues, the OFI issues, and at least the impact of Hardware changes on the Thruster class.
- [ MES ] Acquire a new test Mac so we don't have to run on developer laptops. Even if we build GMAT on the test machine and unzip onto developer machines, we still have to split into two sets of tests due to VPN crapping out after 24 hours. We could explore if we can get this time increased to 48 hours, but I'm not optimistic on how easy it would be to waive IT's policies.
Stop Doing
- [ SPH ] The number of permutations of different packages is too large, zip, installer, public, internal, mac, linux, windows. I propose we consider only releasing a zip or installer on Windows to reduce the number of permutations of tests to run.
- [ MES ] An alternative to this is to run only smoke tests on either the zip or installer, with full tests on the other.
- [ MES ] Mac process currently runs full tests only on the internal version, on the theory that the public version is a subset of those capabilities. This approach may be reasonable on other platforms, with the addendum that all delivered versions should at least be smoke tested.