Estimation for Social Services Information Systems: Lessons from the Trenches

>>Coordinator: Welcome and thank you for
standing by. At this time, all participants are in a listen only mode until the question
and answer session of today’s conference. At that time, you may press Star, 1 on your
touchtone phone to ask a question. I would like to inform all parties that today’s conference
is being recorded. If you have any objections, you may disconnect at this time. I would now like to turn the conference over
to Elizabeth Mertinko. Thank you. You may begin.>>Elizabeth Mertinko: Thank you Jennifer.
Welcome to the Child Welfare Information Technology Systems Managers and Staff Webinar Series,
brought to you on behalf of the Health and Human Services Administration for Children
and Family’s Children’s Bureau and presented by ICF International. Today’s webinar is entitled
Estimation for Social Services Systems: Lessons from the Trenches. I’m Elizabeth Mertinko,
I’m your host today and your moderator. Attendees are encouraged to participate in
our webinar with questions and comments. All of the participant lines are muted now, but
we will open them for the question and answer session at the end of the presentation. However,
please be aware that you can submit questions at any time, using the Go To Webinar chat
feature. Those questions will be addressed during the Q & A session, or throughout the
presentation. Should we run out of time, we will respond
to your questions via email. And/Or should you have additional questions, you may submit
those to me at the email address listed on the slide. Also, if you have any topics that
you’d like to recommend as potential webinars in the future, please do not hesitate to contact
me at the email noted above. The Division of State Systems within the Children’s
Bureau continues to provide a series of monthly webinars supporting information sharing and
discussion. Understanding who is attending the webinars helps to identify content that
is applicable for everyone participating in an agency CCWIS effort. Please select one
of the five categories listed above so that we can get a better idea of who is participating
in our audience today. And, if you could give me just a minute. It
looks like our polling feature is running a little bit slow this morning. I’ll tell
you what. It’s not working. So, in the interest of keeping things moving forward and getting
right to the content, William, I’m going to go ahead and turn it over to you without doing
the poll today.>>William Roetzheim: All right. Hi. So, my
name is William Roetzheim. I’m the Chief Executive Officer and Founder of Level 4 Ventures. And,
I’ve got a 30 year history specializing in cost estimation. So, I’m very enthusiastic
about cost estimation. I really like it. It’s something I’ve spent a lot of my life working
on. So far, I’ve estimated more than 500 projects, including many, many projects in the social
services domain. So, I’ve done multiple social services estimates for states including California,
Florida, Texas, and Colorado — as well as looking at data and analyzing data from about
a dozen other projects. Also participating in this will be Les Fujitani.
He’s the Project Director for the Child Welfare Services Project in the State of California.
So, he’ll be talking about some experiences there. Also, Duc Truong, who is the IT Services
Manager and Architect for California. He’ll be talking about also some social services
related estimation activities as well. So, next slide. What we’ll be covering – we’ve
done the introductions. I’ll be talking about the power of estimation. Here, I really want
to lay the foundation for why should you care, why is estimation important? It’s a topic
that’s often neglected. Often, people don’t think about it much. I want to go into some
background on that. Then, many of you will not be expected to be the hands-on estimators,
but you will be expected to set leadership direction. So, I want to talk about leadership
and setting expectations as it pertains to expectation -what would I like to see you
bring back to your organization in terms of guidance and leadership in the area of estimation. Then, we’ll talk a little bit about the California
experience. California has been using formal estimation now on social services projects
for several years. So, we’ll talk about what kind of results have been obtained and how
it’s worked. Then, we’ll do the Q & A. I’ve also got a next steps slide for those of you
that are interested in pursuing additional information on this topic. Next slide. So, in terms of the power of estimation,
what we’re going to discover is the standard, what do we estimate, why do we estimate, who
does estimation, when do we estimate, and how do we estimate? So, the standard things
– everything you’d want to know about estimation. Just go on to the next slide. So, when people
talk about cost estimation, immediately what comes to mind is I’m going to go to somebody
or some organization or going to apply some process and I’m going to come up with the
cost. And, they think well, that’s what estimation is all about. But, that’s really only a very
small subset of the world of estimation. So certainly estimating the cost is one component.
But, there’s other components as well. For example, we want to be able to estimate the
schedule. How long should something take? Should this project be deployed in six months?
Should it be deployed in a year? Should it be deployed in two years? In many cases, schedule
can be as important as cost in terms of planning and in terms of setting expectations. We also want to look at effort. Effort is
important not just in terms of – as a driver of cost, because obviously with information
technology the labor is a big component of the cost. But, effort is also important because
we want to know what skills are we going to require and how many fulltime equivalents
or how many people will we need in order for this to be successful. So, if you’ve got a
project going forward and it happens that you’re going to need 75 JAVA developers at
some point in time in the project, well you certainly want to know that because you want
to make sure that you’ve got the project staffed to have that many developers available when
they’re needed. And then, another component that-s useful
to look at is benchmark data – estimating based on benchmark data the forecast for the
number of defects. So, for this project – this complexity project – this size of project
– how many defects would we expect to see? That’s useful because it helps set expectations
in terms of testing. It helps set expectations with your end users. And it also helps get
to know whether if a project does have quality problems – in other words, more defects than
expected – you want to know that as early as possible so that you can take corrective
action. When we’re doing estimation, we obviously
are going to be estimating for the design, develop, and install – which is basically
your build and deploy activities. That’s the DDI. But, there are other things that need
to be estimated as well. We want to estimate the change requests. When we’re doing a large
– long project, there’s going to be a high volume of change requests. It’s just unavoidable.
That’s the nature of information technology projects. So, we need to estimate the impact
of each one of those change requests. Besides the actual system that you’re deploying,
you want to estimate your supporting function. So, you want to estimate how much effort is
going to be required for oversight of the project. How much for independent verification
and validation of the project? How much for quality assurance? How much for organizational
change management? How much for training? How much for user testing? How much for maintenance
and operations once we deploy it? Now we’re going to have to support it for the next ten
years. How much – you know, what’s the cost to support this? So, there’s a lot of different
components there. We want to be able to look at resources broken
down by role – so, project manager, developer, tester, trainer, and so on. And, we also want
to be able to look at it by month. So, we want to know if we’ve got an 18 month project
or two year project, month-by-month we want to know when do we expect the trainers to
be wrapping up and come on board. What their peak requirement is going to be? When are
people going to start to drop off of the project because their role is no longer needed? So,
we want to be able to have that time component as well. When we’re looking at cost, we want to break
the cost down, not just in terms of the total cost for the project, but we typically want
to be able to break those down by fiscal year. We want to be able to know for this fiscal
year, how much money do we need? How much money do we need for the next fiscal year,
and how about the fiscal year after that? We want to be able to allocate those costs
to the proper fiscal year for budgetary reasons. We want to be able to break them down by activity.
So, for example, project management, testing, quality assurance, development – what’s the
cost for each one of those individual activities? It helps with your project management process.
Then, we also want to be able to break them down by high level requirement. If we’ve got
you know several dozen high level requirements that define what we want to build and deploy,
we’d like to know which of those costs are the most expensive? Which of those costs are pretty cheap to deploy,
because quite often what we’ll find is that we want to adjust those requirements a little
bit. If there are some requirements that are causing a huge bump in the cost — so they
are making the project much more expensive — perhaps that individual requirement is
not offering that much business value. So, in that case we want to be able to know what
is our cost by requirement so that we can go in there and adjust those requirements
if some individual requirements don’t give us a positive return. And then finally, we want to recognize that
estimation is not a point value. It’s not a one, single value. It’s a probability curve.
And because it’s a probability curve, we want to know what the shape of that curve is. How
much might this project cost if it’s a little bit more expensive than we expect? How much
might we be able to save if things go a little bit better than planned? Same thing for schedule
and so on. We also want to look at risk as part of our
estimate because when we’re setting our budget, we not only want to budget for the things
that we know we have to do, but we want to identify our risk mitigation activities that
need to be taken. We want to identify components of risk that we want to have contingency in
place for. And, we want to make sure that – some of those risks are going to become
a problem. There’s no such thing as a large project where none of the risks turn into
an issue or turn into a problem. We want to make sure that when things do become an issue
or do become a problem that there is some budget available to resolve that and to work
with it. Next slide please. So, you might ask the question
why do we estimate? Basically, why bother? The number one reason for doing good, solid,
defensible formal estimate is to reduce the likelihood of project failure. If we’re doing
information technology projects, those are always a high risk proposition. Most social
services projects are large in scope. They’re complex. That means they’re even higher risk.
And in fact, the rate of project failure, depending on the study that you look at is
anywhere between 25 and 35%. Information technology projects are complete failures. And, because of that, what we want to do is
we want to identify what do we have to do to make sure our project doesn’t fail – to
make sure our project is successful. And, one of the leading researchers, in terms of
project failures and somebody who is caught in quite a bit – so litigation – to ask as
an expert witness, is Capers Jones. He’s got dozens and dozens of books out there studying
information technology projects, the property of those projects, what makes projects successful,
what makes projects fail. He’s identified the six leading causes of
failure in projects that ended up in litigation. And what is – the six are shown here. But,
what I want to note is that number one – accurate estimates were either not prepared or were
rejected, number two – accurate estimates were not supported by objective benchmarks.
So right there, the number one and the number two cause of project failures are directly
estimation related. Then, if we look at the remaining factors that were down below – factor
number three – change control was not handled effectively. While estimation is a contributor
to effective change management, effective change control, it doesn’t – it doesn’t – automatically
ensure that it’s done right. But, it is certainly an important component. Quality control was inadequate – well estimation
gives you those defect benchmarks to tell whether you have good quality or not, so at
least it’s a contributor there as well. Progress tracking did not reveal the true status of
the project. In order to have good project tracking you have to have a baseline to track
against. It’s estimation that gives you that baseline to track against. And then the context
of many key topics, such as quality and out of scope changes. So three, four, and five
estimation is a key contributor to the success in those areas. The only one on that list
that estimation is really not involved in is number six. Next slide please. Now when we’re estimating
– why do we do it? In addition to avoiding project failure or turning it around to having
successful projects – because that’s what we want is successful projects. We estimate
because we have to budget. We have to – we have to have – go back to our legislatures.
We have to go to the federal funding authorities. We have to go to people and get money for
the project. In order to do that we have to have budgets that are defensible and that
are the right amount of money. So, we have to use it for budgeting. We need to use it for feasibility studies.
So for example, if we’re looking at costs and benefits about replacing a system, major
upgrades to a system – any kind of work that we’re doing, especially in the technology
world – at some point we have to say, here is the benefits of doing this work, here’s
the cost, and here’s why it’s a good idea. So, estimation helps with the cost component
of that and then to justify the funding. It’s not enough to just magically come up with
a number and say here’s how much budget I need. You have to sell that number to a lot
of people to convince them that it’s the right number. And a good, solid estimate is what
gives you the ammunition to sell that number and show and to demonstrate that it’s the
correct number and it’s the number that you should use going forward. You also need estimation for independent government
cost estimates. This is basically saying that we can’t rely on the vendors to do our estimates
for us. We have to have some independent cost estimates that we do, that are not – have
no conflict of interest or anything like that. So, this is the responsibility on the IGCE
or the Independent Government Cost Estimate. These are used for cost reasonableness and
cost realism analysis. The difference between those two is that cost reasonableness says,
are we paying a fair price? So, especially if you have a sole source procurement – you
know where only one bidder bids maybe. Or, everything through a change request, so it
is basically a mini sole source procurement, because really there’s only one vendor that
can implement that – that can deliver what’s required by that change request. So, every
change request is a little, mini sole source of procurement in some sense. So, you want to have some measure of saying
is the amount that’s bid reasonable? And, the way you do that is by comparing it to
an independent estimate prepared by someone else. Cost realism is kind of the flip side
of that. It’s saying can this vendor actually deliver what they promise for the amount that
they’re proposing? Because, a vendor – especially an inexperienced vendor, or someone who is
relatively new into an area of work – can come in there and they can make incorrect
estimates just like anyone else. They can come in there and have a project plan that’s
not achievable. So, what can end up happening is they can end up winning the bid because
their prices are unrealistically low. But, then once they’re in there somehow they have
to recover from that. The way that – basically what happens going
forward then is either there’s a bunch of change requests where they are making up the
difference that way, or the project is a massive failure, it ultimately ends up canceled, and
then you guys are in court litigating back and forth trying to see what you’re going
to do about it. None of those are good options. So, what we want to do is we want to make
sure that the amounts that the vendors bid are realistic and that they can actually deliver
what they promise. It’s typically required in the case of limited
competition. So, normally it’s required for things like change requests and it’s also
normally required if you have a situation where you go out to bid and only one or only
two people bid on your RFP, which can happen. Normally, it’s recommended for all high risk
projects – as basically a risk mitigation step. By high risk, what am I talking about
here? Well, any project that has a budget of over $25 million – any information technology
project that has a budget over $25 million is automatically high risk. So, you should
automatically consider it something where this is a candidate. But then, there’s also projects that maybe
have a smaller budget but they have a high or a very high mission impact if the schedule
slips by. And, I’m saying greater than 50% as a rule of thumb. So, you want to identify
– if that’s the case, then you want to do an independent estimate as well. You need
to do independent government estimates or you need to do estimates when you’re doing
alternative analysis. So for example, do we want go outsource – do we want to do this
with internal state resources or do we want to contract this out? Do we want to do this
with a transfer system or do we want to do it with commercial off the shelf software
system or a built from scratch system, or whatever. So, a lot of times there are alternative
approaches to a project. And, one of the ways you analyze those is you do an estimate for
each alternative approach and then you can compare the relative costs and benefits of
each approach. And then finally, we want to estimate the
amount of effort required for government work. So even if we’re going to go off and we’re
going to have a vendor come in and the vendor is going to do all of the development work
for us – so they’re obviously going to want to need to estimate their component to the
work as part of their proposal process. We still are going to have a responsibility.
We are going to need to be involved in the requirements phase. We need to be involved
in the oversight phase. We certainly are involved in the acquisition phase. We’re going to need
to be involved in the user acceptance testing phase. There’s a lot of deployment activities
that we’ll be involved in. So, the government will be involved in a lot of activities even
if we’re outsourcing much of the work. And, we want to know – for those government
activities – how many full-time equivalents will be needed? When will they be needed?
So, we need to do estimates of that type anyway. Let’s go to the next slide. So, when we’re
doing estimation, who does the estimates? Well, one of the kind of default approaches
that’s used by a lot of state government agencies is to rely on the vendors to do estimates.
And, I spend about half my time working with state government agencies and about half my
time working with the major vendors, helping each with estimation. So, I have a full insight
into how the vendors work here. What I’ll tell you is that the vendors, at the pre-proposal
stage – so this is when you’re doing a request for information, you’re basically trying to
– exploring alternatives, and so on. So, you ask the vendors to tell you about what would
something cost. I’ll tell you that the people that prepare
those estimates are the venders marketing people. And when they’re preparing those estimates,
they don’t really care very much what it’s going to cost to actually deploy it. What
they care about is what will it take to get you to go out to bid – to put out an RFP – because
those marketing people, what they want to do is they want to move the project forward.
They want to get it going. And, they’re going to tell you whatever they think you want to
hear in order to get that project moving forward. Then, the next level is we have the proposal
draft stage. So now the vendor is working on a proposal. At that point in time, the
technical staff will come in. They will analyze the project. And, they will internally – within
the vendor – determine what they think it will really cost. So there you get a true
estimate. But, that’s the first time that those technical staff have normally been involved
in doing that. Then the third step – the number that you
actually see though with the proposal final – is what the manager, the proposal managers
and so on, determine based on how much margin they think they can get. So, they’re going
to certainly factor in the numbers that their technical staff are telling them. But really,
then it becomes a question of you know what will the market bear? How much do they think
it will take to win? So really, the only one out of those three that’s a true estimate
is that one in the middle. The second place you might look is to state
technical or management staff – your project managers, your technical folks who are the
hands on folks. Unfortunately though, often these people have limited or not estimation
specific expertise. It’s just not something that’s taught as part of project management
curriculum very well. It’s not taught as part of software engineering curriculum very well.
So, it’s not something that’s taught very well. This is not limited to – it’s not throwing
a stone at state staff. This is true with federal agencies. This is true with commercial
Fortune 100 companies. In general, people who are responsible for doing the hands on
work, they understand how to estimate their own work – the things that they’re familiar
with. But in terms of formal estimation, they normally have very limited training. So then, your third option – independent government
cost estimators. Often these are contractors – not always though. But, it’s somebody that
takes it on as their role or one of their role descriptions to be an estimator. So,
they get training. They read some books on it. They practice the art and they develop
their skills. Where I see this used routinely is in aerospace and defense. So, aerospace
and defense, there’s a lot of companies out there that specialize in this kind of area
– Booz Allen, MCR Federal. There’s just a lot of companies – Tecolote – they built up
an entire large consulting practice around doing estimation. I also see it in specialized
estimation centers of expertise within individual organizations. Let’s go to the next slide. When do we estimate?
Well, we estimate at the feasibility study stage, at the budget definition stage. Then
we estimate again when we’re doing the acquisition. So, this is at the build and deploy type acquisition.
We estimate again as we’re doing change control – so all of our change requests during the
build and deploy work. We estimate whenever we’re in a recovery or turnaround situation.
So, if a project does get behind, if it does have slips or something you have to go back
and redo all your estimates. Then, we estimate when the project is transitioning.
So, this may be transitioning to a maintenance and operation steady state. So, the development
work is done. The project is deployed. Now we need to go into maintenance operations
– a year-by-year budget – to keep the system running. Or, we’re transitioning it from the
vender to the state. So, for example, IBM or CGI or Accenture or whoever is done with
their work, now they’re going to turn it over to the state and the state is going to take
responsibility for operating it and maintaining it. Let’s go to the next slide. Okay so, hopefully
by now I’ve convinced you that estimation is a good thing. It can help you have successful
projects. It can help you avoid projects that are failures. It can give all kinds of information
that’s useful to you as managers. So how do we estimate? Well, I start with not estimates.
Not estimates are things like what do we want it to cost? What will competitors bid? What’s
in the budget? The reality is that most estimates in information technology are not estimates.
Most estimates fit into these categories. The next level down we have expert judgement.
So, expert judgment is basically another word for saying guess. So, we want to break the
project into activities and guess for each activity. Then we can sum up our guesses.
So, this is basically an approach – you know one approach to doing it. Actually, this approach
works pretty well as long as what you’re estimating is something that is small in scope and something
that you’ve done a lot before. So, individual little maintenance order jobs – for example
– can often be estimated pretty accurately just using this approach. But, not really
big complex projects. It falls apart there. The third category is analogy. So, what did
another state pay? If there was another state out there that was exactly like your state
and they had just recently gone through exactly the same procurement that you’re going to
go through, this is probably a great way to come up with the number because that other
state’s experience – especially if you can get two or three states and kind of average
the experiences is probably going to be a very good indicator of what you can expect
to see. Unfortunately, that doesn’t always work because each state is a little different.
Populations are different. How your stakeholders are organized and set up is different. Your
approach – the state laws and regulations and so on are different. So, there’s often
enough differences that this becomes difficult. Then, the last approach and the approach that
I normally recommend and prefer is formal estimation. What formal estimation involves
is we break the project down into deliverable components. We define the size based on the
characteristics of each component. So you know, how many reports, how many pages, how
many interfaces and so on. Then we use historic benchmark data to convert that size into effort,
cost, schedule, and all the other components of an estimate that we’re interested in. Let’s go on to the next slide. So that’s kind
of in a nutshell why we estimate, how we estimate, and so on. But, your role – obviously you’re
not involved – I wouldn’t expect that most of the people on this call are going to go
off and bring up an Excel spreadsheet and start doing estimates. So what you’re going
to be doing though is setting expectations. What I would encourage you to do is to set
high expectations. There is this myth that’s been spread around that it’s impossible to
estimate information technology projects, or it’s impossible to accurately estimate
them. Or, that overruns are just a fact of life. You have to – of course it’s going to
double in size. That’s just normal. In fact, that’s not the case. In fact, people have
been sold a bill of goods by people that don’t know how to estimate – that estimation is
impossible – when it’s not true. So, I’m going to talk about what makes a good
estimate, pushing back against excuses and selling the benefits and encourage you to
– as I say – set high expectations in these areas. Let’s go to the next slide. So, what makes
the good estimate? One component that immediately is going to leap to your mind is accuracy.
In other words, a good estimate is one that is accurate and accurately forecasts what
the ultimate cost will be, what the ultimate schedule will be, and so on. That’s certainly
a critical component of a good estimate. But, there are other elements of a good estimate
that should be present as well. A good estimate is comprehensive. So, it has
to cover everything that has to be done and not leave everything out. So, all too often
we end up with a situation where the cause of problems with our estimation is not with
the parts that were estimated. The parts that were estimated perhaps were estimated accurately,
but that there were major components that were left out. For example, I’ve seen project
estimates where data conversion was just left out and not addressed anywhere. Then, it came
time to do data conversion and the people kind of looked at each other and was like
well who is supposed to do that and where is the money for that? Well, we don’t have
it in the budget anywhere because it was left out. So, a good estimate has to be comprehensive.
And, that’s partly the responsibility of the business unit to make sure that all the components
are there. But, it’s also the responsibility of the estimator to make sure that nothing
has been forgotten. A good estimate has to be credible. All too
often, people are standing in front of control agencies. They have a number in a spreadsheet
for what the cost of something is going to be, and they have no reasonable basis for
defending that number. And so, you’re basically saying trust me, but I have no idea how the
engineers came up with this number. This is just too important of a thing for that to
fly. We really – if we’re standing in front of the legislature and we’re trying to justify
funding for something, if we’re standing before a federal agency and we’re trying to justify
funding for something, every single component of our estimate, we should be able to point
to exactly how we got to it and why it’s the right number, and how we back that up and
so on. That makes that estimate credible, defensible, and so the funding will be approved. I should also be replicable and auditable.
This kind of comes from credible estimates. What I mean by this is if another estimator
came in and were to walk through the work of your estimate, they should be able to follow
step A to step B to step C to step D. Yes, I understand. Yes, I agree with each step.
And then they should be able to come up with more or less the same number that you came
up with. If you do two or three estimates using two or three different approaches they
should come up with the same number – more or less. I mean we’re on a probability tier,
so if they’re off by a percent or two that’s fine. But really, someone else should be able
to come in and end up with the same results that you ended up with. They need to be timely. What I’m saying here
is that even when a project is at the early stage and the requirements are at a very high
level, we should be able to do an estimate. We can’t say well, it’s impossible to estimate
until we have detailed technical requirements because you get those too late. You need to
be able to estimate with a higher level than detailed technical requirements. Finally,
they should be traceable. By traceable what I’m talking about is traceable back to the
driving business functionality that you’re going to deliver. So, if there’s a set of
business requirements, we should be able to trace the estimate back to the individual
business requirements that say which business requirements are driving our costs. Let’s go on to the next slide. When you’re
going forth and leading estimation improvements there’s going to be people that pushback.
And the kind of pushbacks that I see are, well you know, it’s beyond the state of the
art. It’s impossible. No one can do this. The fact is that I’ve done over 500 estimates
to date and I do keep track of my cumulative error. My cumulative error is less than 5%.
The standard deviation of my estimates on a cumulative basis is under 15%. That doesn’t
mean that every single estimate that I’ve done was within 5% as with a standard deviation
of 15%. That’s going to vary based on the level of the project and so on. But, I can tell you that statistically over
500 projects, that’s what has been achieved. And it’s not just me. I’ve deployed this with
these techniques. I’ve seen them deployed at multiple organizations with multiple estimation
teams doing this work and they’re all achieving similar results. The secondary of the – well, it won’t work
because of the project size. Here, they might be saying it’s too small. This is a change
request and it’s too small to estimate with formal methods. Or, it’s too large. This is
a really large project and the models don’t work. The historic data is not there and so
on. In fact, these formal techniques, formal estimation models will work with projects
anywhere between $10,000 up to – I’ve used in my projects, over $10 billion. And the
reason they work over such a wide range of project sizes is because they all are nonlinear
– meaning they have curves. And, these curves allow for economies and diseconomies of scale.
So, they all have built in adjustments where they will adjust these up or down automatically
based on the size of the work that you’re estimating. The third cat-area is the project requirements.
We can’t really estimate it until we have a detailed set of technical requirements.
The business requirements are too vague for us to do an estimate. In fact, formal techniques
will still be highly accurate but there will just be a wider probability range. There will
be a wider standard deviation because of the fact that there is more uncertainty in the
requirements. But, we should still be able to come up with highly accurate estimates.
And, the estimation process itself will typically help focus and pin down these business requirements.
What an estimator works with is high level requirements that are more detailed than general
business goals and objectives, but less detailed than detailed technical requirements. And
they’re in that middle ground -a which is really a very nice place to be, because what
it means is that they’re, they’re expressed in terms that are understandable to and useful
to the business unit because they can relate to-they can understand how the estimation
requirements tie back into delivered business functionality. But it means that it also can – those same
requirements will also be relevant to the developers because the developers can understand
how they need to develop detailed technical requirements to address those individual elements
of scope for the high level estimation requirements. So, the estimators will often find themselves
in a business analysis kind of a role where they’ve got one foot on the technical side,
one foot on the business side, and they’re making a bridge across that by defining these
requirements at that middle ground. And then finally, the last arguments that
you’ll hear is, that formal estimation is too expensive. That we can’t afford to spend
the money to do a thorough job of estimation. And what I would argue is that you want to
spend now to save later. You’re gonna be – ultimately, first of all, when you’re doing estimation,
you’re providing a good, solid, positive, foundation for project planning, you’re reducing
acquisition of project risk, you’re improving project quality, you’re helping to support
the management decision process, you help identify opportunities for efficiency improvement,
and most important of all, you’re radically reducing the probability of project failure.
And, for all of those reasons, spending money at the beginning of a project, to do formal
estimation is money well spent. Let’s go to the next slide. So, in terms of
selling the benefits of estimation, I think the one thing is we want to avoid death march
projects. Death march projects are projects that are started and from the very beginning,
they have an unrealistic sense of baseline conditions. So, the schedule is unrealistically
short, or the budget is unrealistically low, or the resources are not available that are
going to be required. So basically, we’re starting off this project, and we have a baseline
that can’t be achieved. So what will end up happening is that the project team will work
their butt off, they will give you 110%, they will do everything in their power to make
this project successful, because most people in information technology are driven for success,
they want successful projects no one wants to be part of a project failure it’s just-in
information technology, that will never happen. But, if you’ve got a project where the baseline
from the very beginning is completely unachievable, no matter what they do, the project will fail.
So what that means is, you’re gonna have good people working ridiculous hours and in the
end the project will fail anyway all these people will be burned out and it will be a
disaster for everybody involved. That’s called a death march project. You want to avoid those. We also want to promote a level playing field
for system acquisitions. And so to have a level playing field, what we’re looking at
is fair and reasonable pricing. We don’t want people coming in and lowballing projects and
then try to make it up later, or try to cut quality in order to deliver within what they
bid. We want people to – we want the proposals that are coming in to all be achievable. And
then we want to be able to – then we can look at the technical merits of each. And we can
look at things like labor rates or whatever. But really, what we want to make sure is that
all of the – we want to structure the acquisition so that all of the proposals that come in
are fair and reasonable given the vendor’s proposed solution approach. We also want to have, and I’m going to say,
low stress projects. That accept change as integral to the process. Coming into a project
– an information technology project – and expecting that there will not be change, or
that there will not be written problems, or that there will not be risks, is naive. It’s
not gonna happen. So, so let’s plan for it. Let’s budget for it. And let’s have a project
that, from the very beginning to the very end, deals with risk issues that come up in
a natural integral way using a budget that’s already established for that. Let’s deal with
changes that come up in a natural way using budgets that have been established for that.
Let’s make sure that we have an adequate budget so that people can do the work correctly so
that we’re gonna have good quality, because that they have enough time to deliver good
quality, and they have enough resources to deliver good quality, and so there’s nothing
to stop them from doing good quality work. And when we do all of that, what we end up
with is projects that, I’m not going to say, don’t have problems, every project has some
problems, but that the project overall is low stress and ultimately successful. And
then finally, let’s budget to maintain our systems so that they stay viable. What I find
all too often, is that even if people do a good job of estimating the development and
deployment activities, they give very little attention to what it will cost to maintain
and operate the system. And the vendor salespeople that are working together to put together
the proposal, they are going to be focused primarily on the development work, and the
maintenance and operations is going to kind of be a side show. It’s something that they
are not even thinking about that much. But the fact is that, over the life of the system,
the majority of the money is spent on maintenance and operation. It’s not spent on development.
The majority of the money is spent on maintenance and operations. And the reason for that is because you have
to spend a significant amount of money on an information technology system to keep it
working. Even if you deploy something, and at the moment you deploy it, it’s absolutely
perfect, it has no defects whatsoever, and it fully meets the end-user’s needs. So, even
if -obviously that’s not possible-but even if that were possible, from that moment forward,
a month later, a quarter later, a year later, that system is diverging from that absolute
perfection. Things are beginning to break. Forms have changed, so there’s no longer balance.
The laws have changed. Regulations have changed. Tax tables have been updated. People have
moved. There’s all kinds of change that happen in the world, and as the world changes, as
technology changes, as the entire environment around us changes, that software has to change
with it, or else it’s going to go backwards. It’s not going to stay the same, it’s going
to go backwards and get worse and worse. And so we have to spend money, just to stay in
the same place. Let’s go to the next slide. So, what I’m going to do now is I’m going
to turn it over to a couple of folks from California and they’re going to talk firsthand
about some of their experience with the application of formal estimation techniques. A couple
projects that you’ll hear referenced – Child Welfare System/New System. We’ll be talking
about the use of estimation to support budgeting and acquisition during the DDI phase. And
then the Child Support System -California Child Support Automation System – used estimation
to support change management and also to support the transition from vender to state maintenance
operations work. So at this point, I’ll turn it over to Les.>>Les Fujitani: Thank you William. Folks
on the phone, I’m Les Fujitani. I’m the Project Director of the Child Welfare Services/New
System Project in California. Just to give you some background, in July of 2013, the
all systems integration in collaboration with the California Department of Social Services
and California County launched a CWS/NS project to implement a new statewide child welfare
services system. And, so, to give you a little bit of context, our project decided to use
what we call buy build approach, where we acquire either a commercial office shelf solution
or a state transfer system and then build custom software extensions to fill any of
the functional gaps provided by the acquired solution. So, that’s the approach that we
selected. In addition to that, early on in the project,
our project actually embraced cost estimation. And so we believe cost estimation is a key
factor in successfully completing the planning procurement, DDI, and M&O phase of the effort,
you know, ranging from cost benefits feasibility study, to budgeting, cost realism, as well
as system changes that I’ll talk about in further detail in the upcoming slides. Next slide please. And so, for the-one more
slide-okay, so for the planning phase of the effort, we actually estimated cost-well let
me back up here, let me state that we are actually using William Roetzheim, as our contracted
cost-estimator. I forgot to mention that on an earlier slide, but we are utilizing his
services. And so, getting back to the planning phase, we estimated costs for the DDI phase,
by labor category, such as requirements analysis, software development, testing, training and
OCM. With regard to software development, since we’re utilizing a buy-build approach,
a lot of it’s going to be configuring existing functions in that particular solution. And
then the other emphasis is on OCM, since we’re purchasing or acquiring a system, the end-user
interaction with the new system will require more emphasis on OCM for them to get adept
to the change with regards to that new system. On the second bullet, we estimate the level
of effort and number of labor hours for system changes during both the DDI and M&O phases.
So what I mean by that is, based on, you know, projects of similar size and complexity, really
actually we estimated the labor hours, which we turned in to our system change budget within
our planning document. With the third bullet, estimated duration
of the DDI phase duration as William mentioned earlier is key, especially as it pertains
to your costs, because the longer your duration, that will absolutely mean, more than likely,
more money. And so, getting the duration right is also a critical factor. And in all the
estimation stuff that we have, went into our Implementation Advance Planning Document,
which we submitted to ACF. Next slide please. The procurement phase,
so since we’re utilizing a buy-build approach we’re acquiring either a COTS solution or
state transfer system, our actual requirements in our RFP are at a high level. It’s more
of what we’d consider business-based procurement rather than a custom build where you have
a lot of detailed requirements. So, as William mentioned before, again, our RFP has requirements
at a higher level; however, he was still able to give it an estimation, a pretty good estimate
although the standard deviation maybe wider than what you would see with a custom development
and detailed requirements. We also were thinking about using cost estimation
techniques or possibly a payment incentive model in our RFP. That’s still under consideration.
And so, we did use cost estimation with regard to that as well. And then, on the third bullet,
continually update estimates when RFP requirements change during the solicitation period. So,
in IT projects, we kind of call it progressive elaboration, but you start out planning, with
your feasibility study report, and you move towards the development of your initial RFP
draft, we actually did a cost estimate, based on our draft requirements. And we also planned
to get another estimate based on the RFP version that we planned to release to the vendors.
Again, that progressive elaboration that we’re getting closer and closer to a better estimate.
And then as we go through the solicitation process, with each RFP addendum where we have
added requirements to the RFP or changed requirements, we plan to get estimates redone again. And then on the fourth bullet, assess cost
reasonableness and cost realism of the bid during cost evaluation. So, once we receive
the bids from the vendors, as part of the evaluation, we want to utilize William’s services
to actually assess the cost reasonableness, as well as the cost realism. And for some
of that, we’ve actually built cost worksheet within our cost workbook to actually capture
some of the information that William needs to actually calculate those estimates. Some
of those items are things such as the number of technical environments, such as the production
environment, test environment, development environment, as well as other things, such
as number of published services already built into the new system, number of data exchange
interfaces, estimated number of unique pages in the new system, forms and reports and workflows
– all those types of factors go into an estimation model. He also is-we are collecting information
related to data conversion, so then we need to understand the number of unique databases,
number of database tables to convert, number of database fields to convert, as well, number
of amounts of records and the like. And so as a part of our procurement – our
RFP – we do have spreadsheets in the back to capture this information, about each of
the bidders so we get to actually do an assessment for cost realism and reasonableness. And then
with the last bullet, with the-using cost estimation techniques-you can establish a
baseline of the cost information to assist the contract negotiation. So with large projects,
it’s almost automatic that you go into cost negotiation. So from a state’s perspective,
by using cost estimation techniques, we believe that we’re on a better footing with the actual
bidders when it comes to the negotiation process. So, we think that’s very valuable to us. The next slide. The DDI phase. Cost estimates
for business rules analysis, you know, in our RFP, we did not include requirements for
our business rules. And therefore in regards to business rules analysis development, they
are not going to be part of our contract’s fixed cost. So what we’ve done-we had William
do an estimation of the number of labor hours that more than likely will be required for
our business rules analysis. We use that information to create a separate bucket of dollars, in
order to do that business rules analysis. And what we plan to do is, once the vendor
is on board, we plan to work with the vendor in getting the actual cost estimate associated
with business rules, and then bind them to a work authorization, in order to pay that
portion of that budget that’s specific to business rules analysis. That’s similar for
system changes as well. We have William figure out what our budget should be for system changes
during DDI as well as M&O. As most of you know, you know, systems need
to change all of the time. And so, due to legislation or just the business process requires
some added features enhancements that system change dollars allows you to make those changes.
Without estimating that budget amount, you’re stuck with postponing any of those enhancements
until a budget action could be done in order to get that money. So, we want to pre-allocate
dollars in that system change bucket that’s really close to what we expect in terms of
changes on the annual basis. The third bullet – duration estimate to measure
the time to delivery and benefit realization are under consideration by the state. So,
what that means is we want to deploy maybe 70% of the functional requirements in our
RFP as soon as possible. And, because of that, we’re calling it you know business value delivery
where we’re going to give incentive – we’re thinking about giving incentive points during
evaluation to encourage the vendors to come up with an approach based on a shortened schedule
with a lot of benefits from a business perspective to our end users. What I’m talking about is rather than a single-phased
effort that we’re looking into breaking up our project into multiple phases so that there
will be phased realized until a point in time when 100% of the functionality is implemented.
So, we’re looking into that. We’re also looking into quality estimates to measure defects.
We’re taking that – we’re actually looking into that as well to try to encourage vendors
to ensure that whatever they deliver is at a level quality that’s accessible to the state
based on benchmarks from other projects and we’ll invest in it. Next slide please. On the last slide, it has
to do with M&O phase. Our state is planning to implement what we call a sourcing strategy
during M&O where we are going to have all of our M&O services broken up into categories.
And, those categories William assisted us in estimating costs associated with those
service categories. So, down the road when we’re actually in the M&O phase, if the state
decides to insource it – that particular service category or categories – that we can actually
remove those costs out of the contract. So, it’s not only insourcing, our state made
also want to re-outsource to another vendor, certain service category, as well as another
category called co-sourcing, where there’s actually a split between the state and a vendor
in providing those services. So, again we have our sourcing strategy, the costs that
are associated with those service categories, so that from a contract perspective, we can
actually remove those costs from the SI vendor’s contract. Lastly, as I mentioned earlier, we’re also
going to use cost estimates for system changes during the M&O phase. That’s all I have William.>>William Roetzheim: Okay, so let’s go to
the next – now we’ll go onto Duc.>>Duc Truong: Good afternoon everyone. I
think it’s technically afternoon for everyone at this point. My name is Duc Truong. I’m
the Enterprise Architect with the California Franchise Tax Board. I’ve been validating
cost estimates an performed cost estimation for about ten years for the California Department
of Child Support and the Franchise Tax Board. Although, I said I’m an enterprise architect,
during the bulk of my time doing the cost estimation I’ve been a business analyst. So,
I don’t want to mislead anyone into thinking this is an architectural workload or skillset.
I’ll be going over in the next couple of slides of how we apply cost estimation at the two
departments and cover some of the lessons we’ve learned. I don’t want to go over a lot
of what has already been discussed. So, I will just touch on some of the high level
things. In 1999, the California Department of Child
Support Services as created. One of the mandates was to build a statewide child support system.
The Franchise Tax Board was brought in as the agent to help procure and implement the
system. At the time, the California Child Support Automation System – or CSAS – project
was the largest social services project from all of states. We saw a lot of change requests
come in, some for hundreds of thousands of dollars and some for millions. We used cost
estimation to validate the costs in the change requests which included a lot of what William
talked about earlier – such as covering cost of development testing, maintenance, and operations. Near the end of the project, there were less
funds and the larger change requests stopped coming in. During the M&O period, we found
that the same process and methods could still be applied to much smaller efforts. I think
William mentioned earlier that this is one of the excuses folks give for not doing it,
because of the project size being too small. But, we found that we can still do it and
get value out of it. When we transitioned the solution from the
vendor to the state, we had gotten used to the practice of estimating, and validating
the effort-the level of effort for work-that we continued to perform for much of our own
release planning processes. In 2011, the Franchise Tax Board kicked off
another large-scale effort called, the Enterprise Data Revenue, or EDR, project. It is our modernization
effort to bring common services to replace our legacy systems. So the technical environment
for EDR was very very different from CSAS in that we weren’t building a single system
for a brand new department. We have a lot of legacy systems that are impacted by the
new services, so to account for that total cost we had to bring a consistent method of
capturing the cost estimates with our legacy systems. So, what we’re doing now is we’re starting
to build up enterprise level cost estimation program to start tracking that total cost
of ownership. So on the next slide, I want to go over a few of the lessons that we’ve
learned over the years. The first thing was, getting over the need to make things perfect.
The general rule that we have is that-what we see is that-the better you are at estimating,
the better your estimates. But you can’t go from not doing it at all to doing it perfect.
So we have to set some baseline expectations. Even if we started with a WAG, if it was consistently
applied and done, we’re better off than not doing it at all. And, if we’re able to get,
start, folks start using expert judgment, that’s one of the things William mentioned
earlier, as long as it’s consistent, we’re continuously getting better. Eventually, we want everyone to follow a common
method, which is what we call function point analysis. This falls under the formal estimation
category that William covered earlier. So, that was the first lesson that we learned. The second less was that-dealing with the
cultural aspect of it. We found that there is an anxiety level that we have to deal with
because this is a new kind of work for a lot of people. But for others, it has been traditionally
fallen to the responsibility of the developers. Often, when it goes to the developers it’s
too late in terms of estimation because a lot of times they were already starting to
work. What we have to do is spend time to show that
this is an established repeatable process that can be performed upfront and that we
can provide a lot of help so that the analysts don’t feel like they’re doing this alone.
So, the need of getting our change management folks to trust this process was another big
lesson for us. The next thing was to get our management team
to understand the difference between what was reasonable versus what was expensive.
This is a lesson William covered just a little bit. When our contract folks see the dollar
amounts – just the straight dollar amounts – they always wonder why things cost so much.
Our job as estimators is not to talk about how expensive things are but rather is the
cost reasonable in terms of listed or agreed upon labor rate. It really was, we’re sitting
down with our contract team or the management teams to help them understand the estimation
models we use to show them how we come up with an assessment if something is reasonable.
It is really their job to reduce the cost. So once we got to that point, to the understanding
of the separation of those duties really allowed us to make this work. So, the last thing I wanted to bring up, especially
when we’re moving to the M&O phase that when we talk about cost estimation we usually think
about the dollar amounts. Our management team initially questioned that the ongoing value
of cost estimation after the project is complete because we no longer have to deal with the
contract. However we found a lot of value in – when we removed the dollar amounts, we
can still use the level of effort hours as a different kind of currency that we can use
for risk management and release planning. So we have a way to carve out hours for future
releases. From a process perspective, this was probably where we have gotten the most
benefit as it really helped us mature our release planning and management processes.
So those the-I think are the biggest lessons that we’ve learned over the last few years
working with the two departments. So, at this point, I’m tossing it back to
you William.>>William Roetzheim: I think we’re ready
for questions and answers.>>Coordinator: Thank you. Oh I’m sorry, oh
go ahead.>>Elizabeth Mertinko: Nope. I was going to
say Jennifer if you could tell us how folks can line up on the phone for questions that’d
be great.>>Coordinator: Absolutely. We’ll begin the
Q & A session now. If you would like to ask a question, please press Star 1, unmute your
phone, and record your name clearly. I do need your name to introduce your question.
If you need to withdraw that question, press Star 2. Again, to ask a question please press
Star 1 and it will take a few moments for those questions to come through. Please stand
by.>>Elizabeth Mertinko: Ok, and while we are
waiting for people to line up on the phone for those questions, I do have two that have
come in online. The first is what stage is the California project in now? And then the
second one-umm I’m not sure this guy-we might need a little bit more information on the
question, it says the total cost and the percentage of the estimate contract with William? So
I’m not sure that is a full question, so maybe if the questioner could-could sort of rephrase
that and send it in, but what stage is the California project in now?>>Les Fujitani: Okay. This is Les Fujitani,
umm right now, we are in the procurement phase, and so we’re in the process of completing
the RFP and that RFP will be released between now and Christmas.>>Elizabeth Mertinko: Okay.>>William Roetzheim: And Duc, you should
answer on-because Duc’s project is different from Les’s projects-are different from Les’s
projects.>>Duc Truong: So, for the EDR project-well
the child support project ended around, 20-I think, 2010. The EDR project, we’re going
into our last release this winter. So, from that point, we’re entering that transition
period where the state takes ownership.>>Elizabeth Mertinko: Alright. And, on our
open phone line right now, do we have any questions?>>Coordinator: There are no questions on
the phone at this time.>>Elizabeth Mertinko: Okay>>Joyce Rose: Elizabeth, this is Joyce Rose.
And->>Elizabeth Mertinko: Hi.>>Joyce Rose: Hi. I have a question to answer
our distinguished panel.>>Elizabeth Mertinko: Sure.>>Joyce Rose: Historically, completed projects
show that rework figures hover somewhere between 40 and 60%. And, that task is often not included
in project estimates. In your…>>Elizabeth Mertinko: Joyce, are you still
there?>>Joyce Rose: I’m sorry. In your methodology,
where do you include the potential for rework?>>William Roetzheim: Okay so this is William.
Rework will vary quite a bit. But, a good rule of thumb is one percent per month is
what Capers Jones says in terms of requirements change that necessitate rework during the
life of a project. So, if we have a 24 month project, Capers would say that there would
be 24% total effort spent on rework. What I found however is that if you’re doing procurement
in a government setting that would be across the board for commercial companies and so
on. If you limit it to strictly government projects or you’re under contracts for formal
change management, which is normally the case for large social services projects, the rework
is normally more close to 7% per year. So, in the models that we use, we have an
allowance that will be made for requirements related rework-requirement change related
rework. By default, those models are set up to allow for 7% per year in change.>>Elizabeth Mertinko: Okay. Joyce, does that
answer it? We might have lost her on the audio. I do have a couple questions online. Okay,
so the earlier question is what is the total project cost and the percent of the estimate
contract with William? That makes a little more sense than what I had before.>>Les Fujitani: Okay, I’m going to give you
the estimate from our latest project approval document. Our project estimated cost is around
400 million dollars. Then what was the second question?>>Elizabeth Mertinko: The second part of
it was what percentage of that is the contract with William?>>Les Fujitani: I don’t have that in front
of me. But, if that person wants to shoot me an email, I can get that information for
that person.>>Elizabeth Mertinko: Okay, and I will put
the slide that has the email addresses up…>>William Roetzheim: Les this is William-Les,
this is William. I’m not doing the math in my head, but the total contract over the life
of the acquisition is a couple hundred thousand. So, it’s in that ballpark.>>Les Fujitani: Okay that’s – Okay. Thank
you William.>>Elizabeth Mertinko: Ok. I just moved ahead
too this slide just so folks have email addresses in front of them. We do have some other questions
coming in online. How are you addressing the CCWIS NPRM in your functional requirements
in your overall RFP effort?>>Les Fujitani: Is that for me, Les?>>Elizabeth Mertinko: It’s not directed at
anyone in particular. So, I think whoever feels like they can answer that.>>Les Fujitani: Would you read it again because
I didn’t get it.>>Elizabeth Mertinko: Sure. How are you addressing
the CCWIS NPRM in your functional requirements and overall RFP effort?>>Les Fujitani: So, from the new system project,
we were well aware that NPRM was coming out soon. And, we also knew that you know RFP
was going to be released probably during the same time period. So, what we did from a California
perspective is we actually developed an RFP functional requirement. Again, as I mentioned
earlier, they’re at a higher level from a business perspective. So, we still plan to
release that RFP. But concurrently we’re also doing an analysis of the NPRM for two reasons:
one to see if there is any impact on the business functional requirement and then two where
California may have some problems with some of the rulings. But, again we are going to
do an impact analysis of the NPRM RFP if in fact we need to change some of the requirements
in our RFP that will more than likely occur via RFP agenda.>>Elizabeth Mertinko: Okay. Jennifer, do
we have any questions on the phone at this time?>>Coordinator: We do. We have one question
in the queue here. And, I wanted to let you know that Rose has joined us once more.>>Elizabeth Mertinko: Okay. Great. Thank
you.>>Coordinator: This question in the queue
comes from the Mississippi – I think it was – SACWIS project. Go ahead.>>Mississippi Caller: Hi. We’re finding this
information very useful and I appreciate it. We had a couple of questions though about
the availability of people with the right skillset to assist us with setting up an estimation
methodology and process. How common is it? How easy is it going to find people with the
adequate experience to perform these kinds of functions and how much time would it take
to bring someone in to start to develop those estimates at least for the DDI phase of the
project?>>Elizabeth Mertinko: Those are great questions
thank you.>>William Roetzheim: Okay, so this is William.
I’ll take that. First of all, the resources are out there if what we’re talking about
is formal estimation skillsets. There’s an international cost estimation association
that has certified cost estimating analyst certification. It’s kind of like the PMP for
project managers only it’s in the world of cost estimation. And a number of people in
the country that have been certified by this organization is – I want to say – 100 or 200.
It’s not just a handful. There’s a reasonably large number. There are organizations that have staff that
literally have hundreds of estimators available. So the resources are available. There’s probably
– I’ve mentioned before that there’s more of an emphasis on aerospace and defense for
this kind of stuff. So, in terms of domain expertise, many of those people are going
to be aerospace and defense kind of people, but not all of them. Then, in terms of how long does it take to
do this, to do an initial feasibility study type of estimate for a typical one of these
projects, you’re looking at a handful of months. So, it’s not something that’s done in a week
or two but it’s also not something that takes like six months. It would normally be more
like two or three months to do the estimate. Just to give an example, I was brought in
on the project for what was it – child welfare I believe it was – down in Florida. And they
wanted to have a ten year maintenance and operations kind of a forecast. That entire
project took six weeks. So, it’s those types of numbers.>>Mississippi Caller: All right. Thank you.>>Elizabeth Mertinko: So, we have another
question on line asking again for California. Is this a state based application or county
based?>>Les Fujitani: Actually, It’s actually not
county based. Its state administered, but the counties are the users of the system.>>Elizabeth Mertinko: Okay.>>Duc Truong: And for the,>>Elizabeth Mertinko: Jennifer?>>Duc Truong: EDR system, I’m sorry, it’s
completely state based.>>Elizabeth Mertinko: Okay, perfect, thank
you. Jennifer do we have other questions on the line?>>Coordinator: We have no further questions
in the queue. If you would like to ask a question over the phone, press Star 1 on your touchtone
phone.>>Elizabeth Mertinko: Okay, fantastic. And
a reminder, you can also enter questions on the chat box. Joyce, I know that you’ve rejoined
us. I don’t know if you had additional questions that you wanted to ask?>>Joyce Rose: Yes. I have an additional question
than that. The question is: does the developer’s methodology, such as Agile, change the estimation
approach?>>William Roetzheim: It changes-it changes
the outputs from the estimation. This is William again. It changes the outputs from the estimation
because obviously, the labor roles will be different, the activities that you’re splitting
the work into will be different, but in terms of the fundamental approach, it doesn’t. It
does have an impact on the models. So, in terms of the schedule and defects and costs
and so on, so it’s obviously one of the variables or one of the factors. But, it’s not a radical
shift. You can estimate, if you’re using a pure Agile methodology, I talked earlier about,
as your inputs, you can use things like reports and pages, and interfaces, and so on. So,
and those are normally things that are very understandable to the business unit, so you
can estimate using those components, and then turn those components into story points and
or into forecast numbers of stories, and then apply that during your Agile methodology. Later in the project, when you have a story
backlog, if you wanna-if you wanna look at your story runway, and if you wanna to look
at your stories, you can shift the input a little bit, and estimate based on the number
of stories. And that way, you can identify how much time is going to be required and
what’s going to be required per scrum or per iteration.>>Elizabeth Mertinko: Okay.>>Joyce Rose: So how, so how would you do
that as you’re’when you’re developing an RFP?>>William Roetzheim: You wouldn’t. I do it
for, for Agile methodology, what you would do, is you would basically do the beginning
of the estimate exactly the same as you would do for any other methodology, because, I’m
not trying to look at implementation approach, what I’m trying to look at is delivered business
functionality. So, I’m looking at: what is the business functionality that needs to be
delivered? How much of that is existing? How much of that is configuration work? How much
of that is new development? And I’m trying to define the project in terms of business
capabilities that are delivered, and how much of it needs to be built as a part of this
project. Using whatever methodology you wish. Then, I can apply standard metrics for things
like cost per delivered unit of functionality, effort per delivered unit of functionality,
and I can actually run two different scenarios: what if we did it Waterfall? What if we did
it Agile? What if we did it Spiral? But, it’s basically the same set of inputs, because
when all is said and done, if you’re a social worker in the field, you don’t care if the
software was built using an Agile methodology, and stories, and scrums, and so on, or if
it was built with Waterfall or if it was built with some other approach. All you care about
is how many screens I can bring up, what capability did they give me, what reports can I run.
And so, that’s really the thing that’s left behind when the project is done. And that’s
the core thing that’s used as the basis of estimate.>>Elizabeth Mertinko: Okay. Joyce, did you
have any other questions?>>Joyce Rose: No. I’m good. Thank you.>>Elizabeth Mertinko: Okay. Next question
for California. Did you have to customize for the larger counties?>>Les Fujitani: Again, we’re just in the
procurement phase right now. And, so, all we’re dealing with right now is the functional
requirements. We did do RFI earlier. As I mentioned earlier, we’re going to utilize
the buy/build approach. So again, it depends on what solution is going to be bid by the
vendors in terms of, you know, what the gap is going to be from our business practice
requirement. So, that is to be done during the DDI phase of our effort.>>Elizabeth Mertinko: Okay. Great, thank
you. Jennifer, do we have anybody else on the phone?>>Coordinator: There are no questions in
the queue at this time.>>Elizabeth Mertinko: Okay. William, since
we’re kind of running to the end of our time I’m going to turn it over to you for some
next steps.>>William Roetzheim: Okay. So, if any of
you are interested or if you have members of your staff that are interested in learning
more about this kind of stuff, I am kind of an advocate of estimation in general. So,
what I do is about once a quarter I do a three day free estimation training class. So, the
next one is scheduled for November 17, 18, and 19. Each day it runs between 9 a.m. and
1 p.m. Pacific Time. So, that’s noon till 4 o’clock in the afternoon Eastern Time. As
I pointed out, the registration is free, but I’ve only got a limited number of WebEx spots.
So, you do need to email me and let me know so I can reserve a slot if you have somebody
that’s interested.>>Elizabeth Mertinko: Okay. And then we have
it looks like email addresses for William and for Les and for Duc here, so, if you all
have questions for them, specifically, please feel free to go ahead and send them an email.
So, I wanted to take a moment and just thank all of our presenters today. I think this
was really a wonderful and informative session. And, I know you put a great deal of time into
putting your slides and your presentation together. So, I really want to thank you for
sharing your time and your expertise with us this afternoon. For our audience, this is our last webinar
for fiscal year 2015. Be watching for an invitation for our next webinar which will be on system
modernization to support forms automation, mobile apps, and data visualization. And,
that’s scheduled for October 2015. Again, if you have any questions regarding today’s
topic or you would like more information about any upcoming webinars or past webinars, and
in particular if you would like to suggest a topic or volunteer yourself or someone in
your state as a topic presenter, please don’t hesitate to contact me at the email address
above. Again, this webinar has been recorded and
it will be made available online at the address shown above. As soon as it is complete and
posted we will let everyone know. And, I hope to see you all back again in October. Thank
you all again and enjoy the rest of your Tuesday.>>Coordinator: That concludes today’s conference.
Thank you for your participation. You may disconnect at this time. END

Leave a Reply

Your email address will not be published. Required fields are marked *