Michael Wolfe on Quora posted an amazing analogy explaining why software development estimates are routinely way off, which results in developers padding their estimates to factor in the fudge of such inaccuracies.

Another thesis I’ve been pondering over lately are the terms “Software Engineering” and “Computer Science”, which imply that software development is purely scientific and engineering in nature. Most colleges run those programs out of their science or engineering departments, and the business surrounding technology (budgets, management, and processes) is premised on the assumption that coding is engineering in that it’s scientific, mathematical, calculate-able, and algorithmic.

All of that is a component for sure – but the process itself, the act of coding is much closer to art. You start off with an idea of what you need to do and how you’re going to do it, you start roughly putting it together, you realize later how things will really need to be put together so you make alterations/refactor, you get feedback from QA testers/analysts/focus groups, and continually refine until you get the final product.

So unless you’re solving the exact same problem every time, what exactly are we basing the effort on? If a carpet guy knows he can carpet a 2 bedroom house in a day, he can be very close with an estimate on a 4 bedroom house. And overtime, the variation of houses is fairly limited.

However, every software project is a new problem; why would the business ask to solve the same problem twice? So we’re estimating based off of as-similar projects done in the past, but unless it is purely a CRUD with zero rules, it’s unique.

Of course an estimate will always be an estimate. But plans & dates get created, promises are made around those estimates, and the estimates become commitments. Which then causes developers to factor in some fudge knowing that the estimate will become a commitment.

Would it help if we get rid of the word estimate and label that value for what it really is, the guess? Imagine updating all your change tracking, ticketing systems, and project management tools to replace the “Estimate” field with the term “Guess”? It would help manage some expectations as to what that number really represents.

The flip side

The problem on the flip side is that businesses are run by plans. They’re not designed to function off of guesses. For example Apple has to gets the contracts signed and dates sealed for the Apple WWDC, payments have to be made to Moscone, Moscone has to prep the building, Apple has to build their booths, sign up all the exhibitors, attendees, etc… so whatever software is planned to be presented has to be ready to roll by then.

It’s a fascinating challenge, and here are some thoughts.

1. Break down the problem into a collection of micro-small problems.
• Smaller problems are easier to size, internalize, and visualize. Many Agile based processes employ this in the form of small as possible user stories.
2. Get formulaic about it.
• If a team consistently applies a 2X b.s. factor, and is 80% accurate – you then have variables to build a formula & pattern around it where you project a target date -/+ x% based on the accuracy, and then build in risk mitigation strategies to make sure you’re managing the risk of not being accurate.
3. Prioritize & focus.
• A good practice at any level (from executive to individual contributor) at any thing (business, coding, personal life).
• Make sure you focus on achieving at least the top priority in it’s entirety by the target date instead of having advanced multiple priorities but never fully completing any. One thing done is better then 10 things almost-done, because almost-done is the same at not done.

I recently listened to the audiobook to Peter Drucker’s book entitled the Effective Executive. Here are my key take aways as to what makes an executive effective.

1. They ask what needs to be done?
• Do not ask “what do I want to do?”
• Concentrate on just one task, two at most, and never more than two at a time.
• Set priorities and stick to them.
• After completing tasks 1 and 2, instead of moving on to number 3 ask again what needs to be done as new opportunities may present themselves.
2. They ask what is right for the Enterprise?
• Not what is right for stock holders, employees, or themselves.
• A decision not right for the enterprise will eventually result in something that isn’t right for anyone.
• It won’t guarantee the right decision will be made, but failure to ask the question guarantees the wrong decision.
3. They develop action plans.
• Knowledge is useless unless you translate that into action.
• Before springing into action, plot a course of how you want to get there.
• Think about desired results, constraints, restraints, check-in points, and implications.
• The action plan is a statement of intention rather a commitment.
• It shouldn’t be ridged in that once sealed you’re locked in.
• It should be revised often as actions result in new opportunities.
• Time is the most scarce and valuable resource.
• Use check-ins to examine results vs. expectations, and revise the path.
• Without an Action Plan you are a victim of events.
4. They take responsibility for decisions.
• Decisions require an owner, deadline, those affected by it, those who need to approve it (or at least not oppose it), and those who need to be informed.
• People decisions (hiring and promotion) are one of the most important decisions.
• 1/3rd of people decisions have the desired positive result.
• The conclusion to make in cases where a people decision didn’t work out is that management made the wrong decision by not putting the right person in the right place (as opposed to viewing the person as “bad”).
• If someone is promoted into a job where they are underperforming, it may very well not be their fault as it wasn’t their decision to be placed in such a role. So if it’s not working out, offer them the previous job they held where they were successful. Odds are they won’t go for it, but it does send the message to the organization that it’s ok to take career growth risks.
• After making a people decision, set a check-in date to re-evaluate if it’s working.
5. They take responsibility for communicating.
• Effective executives make sure their action plans and informational needs are understood.
• Executives need to make sure that subordinates have access to the information they need.
6. They are focused on opportunities instead of problems.
• Problems can’t be swept under the rug, so don’t ignore them.
• However problem solving doesn’t produce results, it just prevents damage.
• Exploiting opportunity produces results.
• Changes present opportunities. For example changes in market conditions, economic conditions, consumer behavior, etc…
• Don’t let problems overwhelm opportunities. Find a way to promote opportunities over problems.
• Put your best people on opportunities instead of problems.
7. They run productive meetings.
• Make meetings work sessions (vs. a blab-a-thon).
• Decide in advance what kind of meeting it will be (press release, team meeting, etc…) as your preparations will differ, and what the desired result of the meeting should be.
• Terminate the meeting the moment the objective of the meeting  has been completed. Don’t introduce additional topics.
• At the beginning of a meeting, state its purpose.
• Follow up with meeting notes summarizing decisions, assignments, owners of each task, and deadlines.
8. They thought/said “we” instead of “I”.
• An executive only has authority because they have the trust of the organization.
• The needs of the organization come before the executive.
9. They listen first / speak last

As it happens, there is a CMS built on Grails called Weceem – it took some trial and error to get it working as the documentation could use a lot of work. Here are some tips if you’re trying to get it working on Mac OSX (as a quick and simple standalone development mode).

Tomcat

• Download the tar.gz distribution of Tomcat.
• Decompress Tomcat.
• Copy the folder somewhere as desired, I used /opt/apache-tomcat-7.023. Check out this really good tutorial here.
• Download the MySQL JDBC driver.
• Copy the MySQL JDBC driver (i.e. the mysql-connector-java-{ver}-bin.jar file_ to {tomcat home}/lib. I read postings that say you can copy to a common/lib folder, but that didn’t seem to work.
• Edit the {tomcat home}/conf/tomcat-users.xml file and add something along the following lines:

  <role rolename="admin"/>
  <user username="admin" password="admin" roles="manager-gui,admin"/>

Create a MySQL Database

• Create a DB in MySQL called whatever you’d like (I used cms).
• Create a username and password that has access to read/write/modify the schema.

Weceem config file

Create a weceem.properties file somewhere. /etc/weceem.properties might be a good choice. It’ll look like the following, update accordingly.

# Control whether or not connection pooling is enabled
dataSource.pooled=true
dataSource.driverClassName=com.mysql.jdbc.Driver
dataSource.username=dbusername
dataSource.password=dbpassword
dataSource.dbCreate=update
dataSource.url=jdbc:mysql://localhost/cms
searchable.index.path=/Websites/cms/search-indexes

Make sure that whatever you set as the searchable.index.path is a directory that that the user Tomcat will run under has read/write access to.

Update your environment variables

Edit your ~/.profile and make sure you have the following lines, update path accordingly:

export JAVA_HOME=/Library/Java/Home
export CATALINA_HOME=/opt/apache-tomcat-7.0.23
export JAVA_OPTS=-Dweceem.config.location=file:/Websites/cms/weceem.properties
export CATALINA_OPTS="-Xms756m -Xmx756m -XX:NewSize=256m
                      -XX:MaxNewSize=512m -XX:PermSize=256m -XX:MaxPermSize=512m"

Deploy Weceem

• Download the Weceem.war file.
• Copy the Weceem-{ver}.war file to {tomcat home}/webapps/weceem-{ver}.war.
• You can rename the Weceem file to just Weceem.war if you’d like.

See what happens

• Open a terminal window.
• cd to {tomcat home}/bin
• Type in ./startup.sh – you’ll see some stuff about environment variables, and the command line will return back to you.
• Open another terminal window.
• In the second terminal window, cd to {tomcat home}/logs, and type in tail -f catalina.out
• You should eventually see a line that says “INFO: Server startup in xyz ms”
• Open up a browser to http://localhost:8080/manager/html (use the username and password you set up when editing the tomcat-users.xml file above).
• You should see the Weceem application listed and started. If it’s not started, click on the start button.

• Once running, just click on the left column (the path column) and it should load up the default Weceem page.
• There’ll be a link on that page to edit content, the default admin user name and password is admin/admin.

Hope that helps save someone time who wants to tinker around with it.

Here are some take aways:

1. Keeping your burn rate extremely low buys the company the added probability of getting lucky.
1. The luck from a low burn rate comes from having the cash to experiment with various ideas until you find the right model/combination/technique/strategy. Whereas with a high burn rate, it’s all or nothing.
2. Once you find that right idea/model, you can then hone in on it and maximize it’s usage, which then drives more of what’s working (cash flow, sales leads, profits, etc…). With a now working business model, the profits come in, and you now have the ability to grow and take on more opportunities.
2. $1M should last you a year.
1. Starting with a team of 3 people, growing up to maybe 5 or 6 people.
2. Companies are the most productive when they’re less than 10 people. When you grow beyond 10, productivity goes down. So that first $1M is a team that’s lean and mean, and each individual contributes an enormous amount of productivity.
3. Have a Product Development Strategy and a Customer Development Strategy.
1. Most Silicon Valley start-ups describe an engineering project: Alpha, Beta, Limited Availability, General Availability, etc…
2. Author Steve Blank wrote a book called “Four Steps to the Epiphany“, the thesis of which is companies should have customer development milestones in parallel to the product development milestones such as customer discovery validation, creation, and scaling.
4. Discover the business before you scale the business
1. The companies that pursue the path of low burn experimentation dramatically add probabilities in their favor.
2. With off-shore labor, low/no cost open source technology stacks, and search engine marketing, companies easily have the opportunity to conduct low cost experimentation.
3. Use a business strategy of low cost experimentation done a lot to find out the winning answers, discard the losing answers, and don’t scale until you’ve figured out your business model.

class User
{
  String firstName
  String lastName
}

So I wanted to create a simple right side wildcard search that would effectively be the same as this in SQL Server:

where (firstName like '#searchTerm#%' or lastName like '#searchTerm#%')

In Grails you have a wrapper around Hibernate called GORM, which provides a mechanism called Dynamic Finders.

Imagine if you had to write a collection of functions for finding and retrieving one or many records based on each property (findByFirstName(), findAllByFirstName(), findFirstNameLike(), findByAgeGreaterThan(), etc…), it could take awhile.

But with GORM, it’s as if a little elf in the middle of the night coded all night long to create these functions for you. You never see them, but they’re magically there!

There’s a whole “with criteria” mechanism and the Hibernate Query Language (HQL), but I found the simplest way was just to do this:

def userList =

User.findAllByFirstNameLikeOrLastNameLike("${params.searchTerm}%",
"${params.searchTerm}%")

As a technology manager, one of my teams consists of a web development group building  both internal and external business applications. The core platform is built on ColdFusion, a JVM based technology, mixed in with various other frameworks and technologies (jQuey, Flex, ColdBox, SQL Server, etc…).

A couple of years ago we had a project that involved integrating JBoss Drools (a business rules workflow engine), and we needed a way to easily bridge ColdFusion and Drools together, and came across the works of Barney Boisvert and his CFGroovy project. using CFGroovy, we were able to successfully complete the project, while being able to assess the Groovy language itself.

Meanwhile as the years went by, whenever we had to recruit additional talent we found it increasingly difficult (you can find an article I wrote on the topic at RIARockStars). We don’t even bother looking for ColdFusion developers any more, we look for any talented individual with a web engineering background willing to learn.

With the success that we had on that one project, and the seemingly shrinking ecosystem we decided on a strategy of switching over to another JVM language. The reasoning and hypothesis include:

• Ability to progressively evolve our existing platform (we can do it feature by feature, vs. total rewrite)
• Opens up access to the larger Java talent pool
• Keeps the product on a more relevant platform
• Keeps the staff’s skills relevant

Why not Java itself?

The big assumption

Assessing the risk/potential

Disclaimer

This article is from the eyes of management, and not that of a developer. The findings are equally useful, but the conclusions a developer would make would be different than that of management. Most importantly as a developer, you should make it your mission to learn as many technologies as possible, it opens your eyes to new techniques and trends, and makes you an adaptive individual – and this is something a business values (Seven Languages in Seven Weeks is a particularly good book in this context).

Another thing to note is that I’m not evaluating the technologies themselves – there’s no shortage discussions and articles out there that cover this, so you can read up on those as part of evaluating technological fit (you’ll find some good ones on StackOverflow and Quora).

I’m by no means an expert on any of these platforms, and this was the result of a series of Googling for a week to gather various angles. If my perception is off on anything, I welcome the feedback. Thanks!

Observations

I just didn’t have the time to fully evaluate all of the JVM languages and their ecosystems, so I had to focus on (from my analysis) the biggest three: Groovy, Scala, and Clojure. Given more time, what I would need to do is focus the research on specifically web development as all these languages (including Java) encompass more than just web applications.

But I needed to start somewhere, and you can see where I try to rope in some web perspective.

Groovy

Groovy at this point in time would be the conservative/safest bet, business wise. It’s like the Ryan Seacrest of non-Java JVM languages; its conservative, clean, polished, and very active. Groovy is like the athletic younger brother of the lethargic and obese Java.

It appears to have the overall largest ecosystem of the three, and is backed by a huge well known entity (VMWare/EMC). Grails being the web framework of interest, they’re also about to release a big 2.0 update.

I am very disappointed that although SpringSource mentions they have a Groovy and Grails courses, they actually don’t conduct any. This could be used as clue that there’s not enough interest to warrant hosting such classes (well more than a clue, that is the case), but you figure just for strategic reasons they’d take a loss on the training (the classic Gillette move, sell the razor at a loss and make it up on blades).

However, I found the Scala and Clojure training availability just as disappointing.

Scala

Scala would be the other strong contender. Although its ecosystem is smaller than Groovy’s, it has a noticeably more passionate community. As well, having Twitter as the big success story is a massive notch on its belt.

The funding of Scala would be on my things to keep an eye on. Part of Scala is backed by a Swiss university (EPFL), and educational institutions tend to be extremely bureaucratic and their funding dependent on government entities. And then you have a business also involved (TypeSafe.com) who has only been able to generate $3M in venture capital, based on the size of their corporate team, that money won’t last long if they’re not generating revenue (since they are private there’s no way to know).

Trending wise it appears to be accelerating in popularity – it’ll be interesting to resample six months from now and evaluate the landscape. Although not as many books as Groovy, its books are more current.

Clojure

From a business perspective, I wouldn’t even put Clojure on the radar for now, I’d need to see if it gains more traction in order to justify investing in it, as well as a much more solid foundation behind it.

ColdFusion

Although I don’t fully analyse it, the ColdFusion ecosystem is much larger than any of these languages. Extremely passionate community, backed by Adobe who invests millions per year in it, vast array of physical and online user groups, etc… As of right now, going by numbers, ColdFusion wins.

But, we wouldn’t be looking to hire a Groovy/Scala developer, just a developer willing to learn. I know that Java/.NET/PHP folks have no interest in ColdFusion (beleive me, we tried on many occasion). So the question is, is that the same situation with Groovy/Scala/etc…, and trend wise is it a matter of time and we’re just at the infancy stages?

The Data

JavaRanch Posts

• Groovy: 2398
• Scala: 625
• Clojure: 532

Books on Amazon:

• Groovy: 12
• Grails; 7
• Scala: 7
• Lift: 2
• Clojure: 6
• Conjure: 0
• Noir: 0

Note: The current offerings of the Groovy & Grails books are relatively old (most recent Groovy one being from 2008, and the most recent Grails ones from 2009).

User Groups:

• Scala: 53
• Lift: 4?
• Clojure: 33
• Noir: 0?
• Conjure: 0?
• Groovy: 23
• Grails: 63

Email List Activity:

• Groovy: 50/day
• Grails: 83/day
• Scala: 38/day
• Lift: 46/day
• Clojure: 33/day
• Noir: 2/day (Google Groups)
• Conjure: 1/day (Google Groups)

Tiobe Index:

• Scala: 50
• ColdFusion: 59
• Groovy: 69
• Clojure: not on the list

eWeek Article 09/12/11 (http://bit.ly/nbbtbx):

• “Groovy, JavaScript, Ruby among the fastest growing programming languages”
• Note: the early relative percentages are interesting, but as impressive as a 100% increase is, going from 1 job to 2 jobs isn’t.
• The more relevant thing here is the industry perception an article like this generates.

StackOverFlow Search on terms:

• “groovy” : 4390
• “grails” : 3391
• “scala” : 3222
• “lift” : 1608
• “clojure” : 3059
• “conjure” : 89
• “noir” : 34

Source of funding/corporate support:

• Groovy
• SpringSource a VMWare company, subsidiary of EMC Corporation
• VMWare: Publicly traded on the NYSE (VMW)
• Employees: 9000 employees
• Market Cap: $42B
• Revenue: $3.54B
• Gross Profit: $2.36B
• EMC Corporation: Publicly traded on the NYSE (EMC)
• Employees: 48,500
• Market Cap: $51B
• Revenue: $19B
• Gross Profit: $10B
• Team Size: 68? (http://bit.ly/ryL8fb)
• Scala:
• Ecole Polytechnique Federale De Lausanne (EPFL), a Swiss Federal Institute of Technology organization.
• Scala Solutions, acquired by TypeSafe.
• Privately held
• Founded in 2011 by the creators of Scala.
• Received $3M (euro) in Series A funding on 5/12/2011 by Greylock Partners.
• Team size: 12? (http://bit.ly/u3uPmI)
• Clojure
• Primarily via the personal “commercial endeavors” of the creator of Clojure (Rich Hickey) and private donations.
• Team size: 8? (http://bit.ly/tIEaQ6)

Who’s using it:

• Scala
• Twitter, LinkedIn, EDFT, Novell, The Guardian, Xebia, FourSquare, Sony, Siemens, Thatcham, OPower, GridGain, AppJet, Reaktor
• Groovy
• Wired.com, LinkedIn.com, Sky.com, Aegeon, eHarmony, EverBank, ExpertPlan, NetJay, NimBuzz, XWiki, Vodafone Music Store,
• Clojure
• BackType, Sonian, Fightcaster, Akamai, BankSimple, Relevance, KamaGames, Stere, Infinitely Beta, Wusoup, Factual, The Deadline, holodb, Prismatic, Amazon

Job Searches on Dice.com:

• ColdFusion: 343
• Groovy: 247
• Scala: 126
• Clojure: 20

Job Searches at Monster.com:

• ColdFusion: 196
• Groovy: 120
• Scala: 64
• Clojure: 9

Indeed/SimplyHired trends

Survey Responses

Hosted by Peter Ledbrook, Grails Advocate

With the imminent release of Grails 2, it’s time to find out just why we think you should upgrade. Or if you’re not using it, why you should give it a second look.

This webinar will introduce you to the new usability features that are targeted at improving your productivity, such as the new interactive command line, improved class reloading, and much better unit testing support. You’ll also discover powerful new features such as ‘where’ queries, database migrations, and static resource (CSS, JS, etc.) handling.

North America: November 03, 1:00pm EST / 10:00am PST – Register Here
Europe: November 03, 3:00pm UK / 4:00pm Europe – Register Here

I’m in the process of learning Groovy and Grails, and Grails has this neat feature called scaffolding where you just create a domain class like a User and all its properties, and it’s auto-create a full CRUD to be able to create/read/update/and delete.

class User
{
	String userId
 	String password
	Date dateCreated
	Date coolStuff
	Profile profile
}

Which when creating, will generate a form like this:

But I was wondering what’s the deal with dateCreated. Turns out that if you define dateCreated and lastUpdated properties, Grails will automatically take care of storing in the database the record creation date and when it was last updated (which is all facilitated via Grails’ ORM Hibernate wrapper).

It’s organized by the Scrum Master, and each team member briefly recaps:

1. What they worked on yesterday
2. What they’re working on today
3. Any blocking issues