I wasn’t going to do it. The whole recap of the year just retired. The goal-setting of the year just arrived. But I did find myself having to plan a few things-plan for customer visits and activities, plan travel for upcoming events, plan time to prepare content for those events, and so on. And as I thought about those plans, I couldn’t help but ponder where I’ve been and where I’m going, and thus this post was born.

What’s Been Happening?

This post also breaks my long silence. I have spent a significant amount of time the past year writing in several forms that consumed my time:

  • Microsoft SQL Server 2012 Reporting Services. That’s book #12 for me, co-authored with my daughter Erika Bakse who has since moved onto another job (in the hopes that she’ll never have to write another book, no doubt!). I’m very excited about this book because it goes into more detail than I ever could in the Step By Step series and consequently becomes a great reference not only for beginners, but for people like Erika and me who have used Reporting Services for years but who have might have used certain features only sporadically. For reasons unknown to me, this book has had a very long production cycle. I finished writing in July, but the book won’t be available until April 2013. And things have changed since then with the release of SQL Server 2012 SP1 with regard to Power View. It’s too late to add onto the relevant chapters now. And of course during the review process, I discovered all sorts of things I wanted to add in other areas, but I have to draw the line somewhere. I can see a whole series of posts coming up to supplement the book!
  • Business Intelligence in Microsoft SharePoint 2013. Which makes book #13. It’s a second edition with lots of updates thanks to the changes in Office 2013 – both in Excel and in SharePoint. My contributions to this book were chapters on Planning for BI Adoption and Power View in Excel 2013. There are some very cool things in Power View now that I wish could have been added to book #12, but c’est la vie. Technology moves at a faster pace than book production! Look for the release of this book in March 2013.
  • SQL Pro Magazine articles. I’ve written for SQL Pro for years and am now a contributing editor as well. In 2012, I contributed more than usual:
  • Books #14 and #15 are in the wings. They’ll be e-books so I’m not sure I can really count them as books, but it’s my list, so I can count the way I want to! I officially started the draft of #14 last night. And I have a meeting scheduled for #16 next month. Call me crazy… or is that maybe? I get so confused. (Need a laugh? Check out the Call Me Maybe parody that I and the Pluralsight gang produced.)

Accidental Business Intelligence? Not Really

But back to the musing…I always think of my career in BI as accidental, although taking a longer view I realize that it really wasn’t. In February 2011, I told my story to Andy Leonard (blog | twitter) as part of his SQLPeople series of interviews. There are some karmic aspects to that story that I would never put in print, but I’d be happy to tell you in person some day if you ask nicely.

In that story, I mention working with Lotus Notes. In the late 1990s, I had this feeling that using technology as a knowledge management tool was an attainable goal and put a lot of energy into learning how that should happen. But then I got deflected into business intelligence which turned into a fascinating and rewarding career. But all along, I had this nagging feeling that BI was just part of the story. I wasn’t satisfied with just delivering on reporting and analysis. As important as that is, I believed additional transformation in the way we work with data and with each other was necessary in order for BI to fulfill its promise.

Collaborative BI

I’ll admit that in the beginning I wasn’t very impressed with SharePoint – I believe it was SharePoint 2003 when I was first introduced to it. I had been working with Lotus Notes long before that time, and felt that it could run circles around that release of SharePoint. Full disclosure – I haven’t looked back at Lotus Notes since I left it, so I have no idea of its capabilities today. But starting with SharePoint 2007, I started thinking beyond the traditional dashboard compilation of scorecards and reports. I was thinking about unstructured data to support the structured, and would mention it in my presentations and classes. Then with SharePoint 2010, I started thinking about the collaborative and social aspects and started putting these pieces together with ideas that I had been nurturing since the late 1990s. And so, a presentation was born for a webinar, a few SQLSaturdays, and continues to evolve as I gear up for the PASS Business Analytics Conference in April 2013. Because an hour presentation only sets the stage for some of my ideas, I have set up a Collaborative BI resource page that will grow as I commit these ideas to writing.

Big Data

Meanwhile, the buzz around Big Data became louder in 2012. Now I’ve been around a few years, and I’ve seen buzz come and go. I had plenty to keep myself busy meanwhile during 2012, and just watched and waited to see what would happen. And then things started to get interesting. So much so that it’s time to start talking about it. Consequently, I am working on presentations on this topic throughout the year (keep an eye on Upcoming Events for online and in-person events), including a session at the PASS Business Analytics Conference on Power View and Hadoop in collaboration with Joey D’Antoni (blog | twitter).

And that’s just the beginning. I plan to add another resource page for my thoughts on BI and Big Data. Watch for more blog posts and presentations.

When I think back to my “accidental” discovery of BI and the ideas we were throwing around at the time, I realize we were a bit ahead of our time. What we needed at that company to achieve those big ideas was big data – we just didn’t call it that then. We started on a much smaller scale and focused on data warehousing and reporting and analysis tools, and we were barely ready for that then. I work with customers today who are still barely ready for that.

Data Science

But now in the era of big data and data science, I start thinking about those big ideas again and how much more attainable they are today 14 years after I started down this path. The BI world is poised for the biggest change I’ve seen in my career. While I cannot share the specifics of what we were thinking about in my R&D days, I can try to explain how I see the difference between BI as we traditionally think of it and where data science can take us. BI helps us understand what happened or what is happening now, using established processes and tools. Although BI can scale quite dramatically, scale introduces some complexities that in some ways limits the types of reporting and analysis that we can do. Data mining is often included in a discussion of BI technologies, but its use has not been very prevalent in my client base. Data mining can not only be used to explore data to help us understand what happened, but can also be used to predict what might happen. And this is where we see data science come into play now. Data science can help us look forward and to predict an outcome or a correlation. It incorporates many techniques that are common to data mining, but it can go beyond those techniques as well. We can work with larger data sets than ever before because we can store data more cheaply than ever before and we have better tools for dealing with these larger data sets using commodity hardware.

Is the Data Warehouse Dead?

No, I don’t think so. At least not completely. There’s still a place for operational and mission-critical information that’s been consolidated, cleansed, and corporately-sanctioned as truth. I don’t really care what we call that information source – a data warehouse, a data mart, whatever. We need access to that type of information because that’s how we decide what to do today to achieve our goals, respond to specific problems, or show the board (or the world) how we’re doing as a business. The new potential with big data and data science is the opportunity to explore data in ways never before possible. We don’t know what the opportunity or business value in that data might be until we examine it in new ways or combine it with other types of data–data that was captured by others and shared publicly is just one example. Creativity is key. But like traditional BI, in my mind, it’s all useless unless we can DO something with that information. And of course, we need to share and collaborate!

What do you think? Is BI as we know it going to to die or thrive in this brave new world of big data?

I look forward to delving more into these topics more in future posts. I hope you do, too!