What Makes Managing Component Based Development Hard?
By Ed Taekema in Jython, Python, Software on December 8th, 2008
I’m looking for input on where the management problems are for building large scale applications with reusable components.
Reusable software components are a pretty established development pattern and are used across the software development language landscape. Lately, this area has experienced a boon of growth as component development became married with service oriented architecture and web services. At the same time, components seem to be getting larger and are used more widely … think Pinax as a great Python pattern. What I don’t think has grown at a sufficient pace is the management thinking and tools to control changes to components across a large scale of interdependent components, particularly from the point of view of the “component consumer“.
I`ve noticed a couple of related patterns.
- Its hard for an individual consumer developer to bring a new version of a component into their application stack, test it and make whatever changes are necessary to the consuming side, and then promote their changes along with the new version of the component to the rest of their consuming side team.
- Its hard for a technical lead or build engineer to manage the component revision level that a large consuming application team is going to use. Often different software engineers use different versions of components resulting in code integration problems. This gets especially tricky if the components have lots of dependencies between them, or if they are assembled in a layered architecture.
I`m looking for additional feedback. Please let me know what you think of these problem areas and point me at any difficulties I`ve missed.
12/08/2008 at 12:42 pm
I’ve been the build/integration engineer on a couple of ambitious component-based projects.
One problem I saw was what I’d call “shopping spree.” The CTO/architect’s attitude was “throw it all in, we’ll synthesize something of value from it eventually.” I had my hands full vetting and integrating components, as engineers were adding several new ones every week.
As we exceeded 110+ components, we started to get version conflicts. Sometimes we had multiple copies of the same component due to this. Also, engineers were simply not aware when a component (or one with similar functionality) was already in the project, so they’d introduce a new one.
Another problem was trying to integrate components that are themselves large assemblies of components. One Java portal server was comprised of hundreds of megabytes of useless frameworks, apps, and games that we didn’t need. The vendor couldn’t help us whittle it down to core features. We didn’t have the manpower it would have taken to do that analysis ourselves.
BTW, I’ve been taking a look at Pinax 0.5.0. It’s 33+ MB. This reminds me of that bloated Java portal server. Especially compared to Django-CMS which is less than one-half MB.
Another issue is integrating numerous different software licenses. I tracked components using LGPL, MPL, MIT, BSD, etc. I had to constantly tell engineers (and even the CTO) that they couldn’t bring in GPL components, unless they wanted to oblige our own project to use GPL or a compatible license.
12/08/2008 at 12:48 pm
I think there’s a “sweet spot” for the footprint size of a component you can manage. Too small or too large and the cost of the overhead is too much. (This also depends on how big your application is; the bigger your application, the bigger the sweet spot).
12/08/2008 at 2:23 pm
@Bill Karwin -
So how did you keep track of the dependences and manage to keep it consistent across the stack?
12/08/2008 at 2:24 pm
@Cory -
The point about overhead is interesting. How would you describe the overhead? What sorts of activities and roles are involved in your experience? Thanks for the comment!
12/09/2008 at 12:13 pm
Re tracking dependencies: I was the build engineer too, so as I integrated each new component into the build I placed it at the start of the build first, with no dependencies in the Ant scripts.
As I ran a test build from a clean tree, I’d get unresolved class errors. So I’d declare dependencies in the Ant script and re-run a clean build. Then I repeated as needed until all dependencies of a component have been built prior to its build.
Sometimes I could short-cut this process by reading the release notes for a component.
Fortunately the company had a policy of building every component from source, not checking jars into the tree. This made the build longer but it was easier to analyze dependencies in the way I described.
I also maintained a spreadsheet, where I also tracked the software license, size, vendor’s URL, etc. for each component. If you’re using a non-compiled language like PHP or Python, this may be the best/only way to track dependencies.
12/10/2008 at 1:48 pm
@Bill Karwin -
Hmmm sounds like something that a CM system could help with …
12/11/2008 at 5:49 pm
I’m wondering what rules teams doing component development would want to apply. For example:
* Everyone on the team must use the same version of the component?
* Anyone on the team can modify the subcomponent?
* No one on the team can modify the subcomponent?
Also, what process do teams use to bring in a new version of the component? Does the build engineer usually put it in, or might a developer do the task of testing and integrating the new component?
I have no answers, only questions.
12/12/2008 at 10:13 am
@Darcy Wiborg Weber -
I’m really struggling to find a good answer to how this should play out. I think that a lot tools constrain things so that everyone ends up using whatever version of the component gets into the version control system. Its hard to tell if this is a choice for projects or not. I think that there is some traction out there about where components are modified … design by contract ideas would say a consumer doesn’t modify the component but the producer does. Where I have seen this rule broken is in new systems where components are immature and evolving rapidly. I’m hoping for more input on this here … stay tuned
12/12/2008 at 3:03 pm
I’ve found that by getting the gnu32 utilities and putting them in my path, the cmd shell works just fine for me. I can’t fork processes with xargs but I don’t need to do that often.
12/16/2008 at 11:59 pm
@James Watson -
Hmmm I think you mean to comment on my shell replacement question … but regardless, great answer. Nice having tail again …