Nov 9, 2005

Data Warehousing for the Rest of Us

If you are part of a small to mid-sized company trying to implement a Data Warehouse or research whether Data Warehousing is right for your company, it probably wont take you long to discover that Data Warehousing is still for the BIG guys and not for the rest of us. Data Warehouse tools vendors cater to the big companies because they are the one's that bring in the big revenues. These vendors so proud of their "rocket science" that they can charge exorbitant amounts for their solutions and then wrap industry heavy weights such as Ralph Kimball and Bill Inmon around their products declaring best practices that are unreasonable and cost prohibitive.

I am sick of reading about "the ideal" data warehousing solution. I have been reading data warehousing and data management articles for about two years now and I firmly believe that these authors live in a dream world that only exists where there is an infinite amount of money and resources to go around.

I do give them credit, however, for promoting technology and practices that can, at least, get you off in the right direction; but in the same breath I blame them for the unbelievably high failure rates of data warehousing projects.

What I want is a Data Warehousing Methodology that doesn't fall on its face in the real world and doesn't cost my company a fortune to implement. Of course, I want my complex analytics, ad-hoc engines, and consolidated reporting tools too! And I also want it to have impeccable performance. Can I have my cake and eat it too?

I am currently working on a methodology that I hope will do just that. I don't have a name for it yet but I have some underlying principles laid out. My goal is to develop a methodology for building data warehouses for the rest of us. The core of my new methodology is centered on agility and speed. My hope is to have some time over the next couple of months to develop my ideas a little more and then post them here for comment. Any feedback is appreciated.


Mike Farmer said...

Dr. Windley was kind enough to comment on some of what I've said here at ZDNet. Thanks for the feedback!

Tom Foydel said...

You have an interesting idea. But if I am a small or medium sized business then I probably don't have the volume of transactions that make a Data Warehouse necessary. With my volume of transactions I ought to be able to use a business analytics tool that's part of my ERP/CRM system. I think that you're right about the costs being prohibitive.

My other comment is that large business that have data warehouses have not fared very well lately. Some smart people like Peter Drucker have noted that small and medium sized business are growing today, but large businesses are mostly stuck in neutral. Are data warehouses necessary? Do they really propel growth? I don't know the answer but it seems like a worthwile question to ask.

Andy Hayler said...

I don't think the size of the company is so much of an issue. If you have a business with more than one set of data split across different systems then you have the inherent problem: how do I take a single view across the enterprise? For example, "how profitable is customer X?" is not trivial to answer if you have more than one system with customer information in, and non-uniform finance allocation rules. This can happen even in quite small companies. For example at Kalido we have a CRM package (Salesforce) a support package and a finance system (Sun Systems) all of which have information about customers. We actually have an internal data warehouse to help us, though of course the volumes of data are small. I do agree that most BI products are priced at the high end, since naturally vendors try and get as much as they can for their products. There are some nascent efforts at open source BI, which you may want to look into, though I have no direct experience here e.g. you could use a MySQL database and examine some free software:

or just look ar MS Access, which is a highly productive and very cheap environemnt.