ASME B89.4.10-2021 pdf – Methods for Performance Evaluation of Coordinate Measuring System Software.
5.7 Process Data With Test Software
Special conversion software or a modified version of the CMS software may be required to allow for the introduction of data not acquired through the normal CMS data input channel. If the CMS is capable of executing a stored program, a program that performs data set evaluations must be written. This program may be subsequently used to evaluate new versions of CMS software. CMS systems without stored program capability maybe manually controlled to perform their evaluations, but it is recommended that automatic methods be made available if possible.
The order of the points in the data set may be changed to satisfy any special requirements of the software under test. If reordering of the data is required, it shall be noted on the test report.
The results of the algorithms should be output in a format compatible with the comparator function.
5.8 CaLcuLation and Interpretation of ResuLts
The guidelines for algorithm comparison in para. 4.1 shall be used to compare the results of the software under test to the reference results for each data set. For each geometric feature type, a statistical analysis shall be performed to evaluate the root-mean-square (RMS) and maximum magnitude of the observed evaluation parameter values.
Difference angles are to be expressed in microradians. Distance and radii differences shall be converted to the normal units of the CMS (see para. 5.4.1).
5.9 Reporting of Test ResuLts
A test report shall be produced at the conclusion of the comparison phase. The test report shall include the following information:
(a) the reference software used and its version identifier
(b) the characteristics of the software that was tested (including computing environment, software version, and any other necessary identifying characteristics)
(c) the geometric feature types tested
(d) any reordering of the data or seed values
(e) the range of conditions represented by the test data for each geometric feature type
(f) the RMS value of each evaluation parameter for each geometric feature type
(g) the maximum observed value of each evaluation parameter for each geometric feature type
(h) the criteria for identifying bad fits for exclusion from the statistical analysis
(I) the test results for bad fits excluded from the statistical analysis and the corresponding test data characteristics
If no fits were excluded from the analysis, the RMS statistic includes the effects of both systematic and random deviations between the software under test and the reference results. Thus, it can be interpreted as the expected deviation from true value for the software under test, over the range of conditions represented by the test data. To support this interpretation, the effects of uncertainty inherent in the reference results must be included in an uncertainty statement for the RMS statistic.
If any test results were excluded from the analysis, the above interpretation of the results does not hold. Rather, the software is unreliable for the conditions of the test. Although there is no consistent metrology interpretation of the test results in this case, the results have diagnostic value.
If the default test is used, the following minimum values shall be used where applicable in the test report when reporting RMS or maximum observed values for evaluation parameters:
Distances i05 pim
In the case the RMS or maximum value of an evaluation parameter is below the minimum, the reported value shall be reported as “less than iO” or “less than iO,” as appropriate, along with the corresponding units.
5.10 Periodic Reverification
CMS software should be evaluated when an upgraded version is released, when there is any change in the computing environment that might affect the results, or when results reported by the software appear to be abnormal.
6 SOFTWARE DOCUMENTATION
This section provides guidelines for minimum documentation for coordinate metrology software.