Automation State and Verdict Issues

In AutomationMeetings20120621 there was an extended discussion on the use of oslc_auto:state in the Automation Requests and Automation Results along with its relationship to oslc_auto:verdict in the Automation Result.

Automation Verdict

One conclusion was that the current values for oslc_auto:verdict in the [[AutoSpecificationV2#State_and_Verdict_properties][specification] are inadequate. The issues are:

1. The name of the oslc_auto:incomplete verdict is a poor one. The suggested replacement is oslc_auto:unavailable to indicate the final verdict cannot yet be given.
2. A new verdict is needed to indicate that an automation could not be run to a pass/fail verdict due to errors or a crash condition. The proposed oslc_auto:error verdict means the automation could not be run to completion due to some error condition. This is not to be confused with oslc_auto:fail which indicates that the automation completed, but the result of the automation is considered a failure (compile error, automated test failed, etc).

Proposed new wording for the oslc_auto:verdict property values defined in the specification:

The additional property values for oslc_auto:verdict are:

• http://open-services.net/ns/auto#unavailable - used to indicate an automation result is in a state where a final verdict such as oslc:auto_pass or oslc_auto:fail is not yet available. Usually used when the result is in a state other than oslc_auto:complete.
• http://open-services.net/ns/auto#pass - used to indicate an automation result represents a successful execution.
• http://open-services.net/ns/auto#warning - used to indicate an automation result represents an execution which encountered conditions which prevented successful execution but did not result in a failed execution.
• http://open-services.net/ns/auto#fail - used to indicate an automation result represents a failed execution.
• http://open-services.net/ns/auto#error - used to indicate an automation result has completed but did not run successfully due to some error. This could be a timeout, automation coding error, network problem or other error which prevented the automation from running successfully to a pass, warning or fail verdict as described above.

Automation State

The second issue centers around how to interpret oslc_auto:state on the Automation Request and Automation Result artifacts. The workgroup has recognized that requests and results will have a variety of potential lifecycles in different provider implementations. Some examples of what the workgroup has heard:

• Provider does not persist the Automation Request. The request is only in existence long enough to create the Automation Result and the result is the primary means of tracking the automation's state
• Provider does not create the Automation Result until late in the execution (or at the very end). The Automation Request is the primary means of tracking the automation state.
• Provider uses the Automation Request to queue or otherwise track the execution request until it starts execution. At that point the request is marked complete and the remainder of the execution is tracked through the result.

The flexibility allows a range of provider implementations, but causes some potential hardship for consumers:

• How does a generic consumer track an automation execution through its lifecycle?
  • Consumers only interested in end results can query for the result and wait for its state to go to oslc_auto:complete and then look at the oslc_auto:verdict
  • Consumers wanting to track the execution through all states must watch both the Request and the Result and make some inferences based on the combined states.
• How does a consumer know where to request cancellation of an execution?
  • Proposal: If Request is not complete, cancel the Request. If Request is complete and the Result is not complete, cancel the Result.

For discussion in the next Automation workgroup

• Do we need language in the spec to enforce state consistency across requests and results? Examples of potentially inconsistent state combinations:
  • Result is complete but associated Request is not
  • Result is oslc_auto:inProgress and Request is oslc_auto:queued or oslc_auto:canceled
• Canceling an automation execution - should there be normative language?