Based on most of our discussions in this forum, plus other discussions/presentations, it seems there is a consensus on four categories of metrics that are most relevant to this work:

Size, quality, schedule, and effort. (Please comment if you agree or disagree!!!)

But these categories aren't all that clear, and some issues are very relevant for the relationship between estimation tools and project management tools. So this topic starts that discussion. For now, we can keep it in one topic, but we can split it out if it that will organize the discussion better.

Two implication of this:

a) The four categories aren't just examples, they are the four categories THIS WORKGROUP should address---there are no doubt metrics that don't fit these categories, and there are other things we care about too (e.g. "development language") that might not be a metric, but I believe these are THE categories of metrics this group cares about.

b) There are different characteristics of the metrics in each of these categories, so we need to address each category separately.

Issues with Size:

[ajb] Please comment on all this:

Three related concepts get confused in common use, discussions, and written materials I've seen: Size, Scope, and Effort. The difference between size and effort "should" be clear, but if you read authors like Mike Cohn, he indicates that some teams use "ideal days" as a measure of size rather than effort: we estimate that this story is 8 ideal days, while this one is only 4 ideal days. Cohn says that really means "this story is twice as big as that story" rather than "this story will take 8 person/days to develop". He suggests using Story Points instead as a measure of relative size, which is clearer ("relative" is another discussion!!!) So let's put that confusion aside for just a moment, but not too long!

But I would like to comment on "size vs. scope", and how that affects the work of this group. First, I'll note that in the talk Larry Putnam (and I) gave at RSC 2009, he listed "Size/Scope" as the category. They're clearly related, and for the purpose of the talk I think it made sense to elide them as he did, but for our purposes, I think there's an important distinction.

Here's the summary of my current thought that I would appreciate if we can discuss: Size is a function of scope. Estimation tools care about size but not scope; project management tools care about scope but not size. So for the OSLC definition of information interchange between project management and estimation tools, if the project "knows" its scope, and the estimation tools "needs" its size, which tool computes the function? (Can the estimation tool "GET" the size from the project management tool, or can it only "GET" the scope?) (Note: remember I said put aside the confusion between size and schedule? While I'll try to explain why I think the project management tool doesn't care about size, it very definitely cares about schedule. But I think that many project planning tools I'm seeing then confuse size and schedule, thus the weird "ideal days" metric).

First: to justify the claim about which tool cares about which (really, which Actor cares about which): Scope isn't a "metric"...it's the list of "stuff that must be developed" (features, stories, enhancements, defects fixed). In project manager/product owner/portfolio manager/customer cares what that list is in detail. If the project management expected a system that lets a customer see what used books are available and also purchase used books, they won't be happy if the system developed lets them see what airline tickets are available and purchase airline tickets instead. But the estimation tool doesn't care about the unique specifics of the functionality: it cares whether the functionality to purchase used books is the same size (or comparable size) to the functionality to purchase airline tickets (to compare to historical estimates), or maybe it cares about the relative size of the purchasing and displaying stories (for some kind of bottom up totals). (There is MUCH more to say here, on all sides of this. Nothing in this area is this simplistic of course.)

Second: How does this relate to the "Project Initiation" use case? In the project initiation use case, to use the terms Larry Putnam injected into our conversations, we're interested in static metrics: the relationship between total size, total schedule, total effort and (total??? does this make sense???--need a separate discussion about quality) quality. Some are known to the project and inputs to the estimate and some are the outputs of the estimate of interest to the project manager. I don't think size is either--it's an input to the estimate, derived from scope which is known to the project. How it's derived is important to this workgroup. Note: While some of the categories, especially schedule, could end up being both inputs and outputs of the estimate (GET and POST to the project management OSLC interface) I'm coming to the conclusion that "total size" is not of interest to the project manager (so neither GET'ted norPOST'ed to the project); rather, the estimation tool computes it from some other input, and keeps it as part of its data.

One possible way: bottom up, from relative sizes of scope items. At least this seems to be a common way to try...if this makes sense, then it may point the way for how the estimation tool can get this as an input from the project management tool (MAYBE!!!), and thus a candidate for the OSLC interface we're defining. (Important note: that you could compute size bottom up does not automatically imply you can compute effort or schedule bottom up! There are lots of reasons it's different for the different categories. Definitely worth more discussion, maybe not relevant for this workgroup though)

So maybe I was hiding a subtlety when I said earlier "the project owner doesn't care about size." The project owner doesn't care about TOTAL size, but when you read authors like Mike Cohn, discussing agile project planning, he indicates the project owner DOES care about the size of each scope item (but I don't see anything that indicates "total size" is of interest). He claims size is used for prioritizing scope items for inclusion in an iteration because the project manager can say "that story is more important than this one" but this is tempered by "but this one is so small compared to the other one that it's worth doing now anyway." So many agile planning tools (including RTC) have a place to "estimate" each story/scope item. Cohn says very clearly that should be an estimate of relative size: that story is three times as big as this story (15 story points vs. 5 story points for example). (A whole other discussion--does this mean that that story will take three times the effort as this story: not necessarily: it depends on a bunch of other stuff--yet another discussion, but maybe not needed by this workgroup. We only need to define interfaces between two classes of tools, not answer all questions about estimation theory! Those get answered somewhere else.)

So, to say this another way: If a project has a collection of scope items with a measure of (relative) size for each one separately, can that be used to compute total size to be used as an input to the estimate? Two subquestions: is size additive, or is a more complicated computation needed than "add up the story points", and how to deal with "relative" size, i.e. one team says these two items are 5 and 10 story points, while another team says they're 3 and 6 story points)--I think that's the "data normalization issue", right?

Another question related to this workgroup: Does the project own the scope items with size, or does it have a reference to a list that's owned by another tool? Another way to say this: does the Project resource (in the "OSLC interfaces return a resource" sense not "a resource works on the project" sense) contain a list of scope items (each of which may be owned by another tool), or just a URL of the list owned by another tool? Or is there a query sent to another tool with the URL of the "project" as a parameter? In plainer English, from which tool does the estimation tool actually get the sized scope items and how? Maybe there are several ways.

Issues with Quality:

One issue was already raised by Skip Pletcher as a comment to Software Definition Metrics

Issues with Schedule:

Issues with Effort: