Musings on a BI Career: Past and Future

January 3, 2013

I wasn’t going to do it. The whole recap of the year just retired. The goal-setting of the year just arrived. But I did find myself having to plan a few things-plan for customer visits and activities, plan travel for upcoming events, plan time to prepare content for those events, and so on. And as I thought about those plans, I couldn’t help but ponder where I’ve been and where I’m going, and thus this post was born.

What’s Been Happening?

This post also breaks my long silence. I have spent a significant amount of time the past year writing in several forms that consumed my time:

  • Microsoft SQL Server 2012 Reporting Services. That’s book #12 for me, co-authored with my daughter Erika Bakse who has since moved onto another job (in the hopes that she’ll never have to write another book, no doubt!). I’m very excited about this book because it goes into more detail than I ever could in the Step By Step series and consequently becomes a great reference not only for beginners, but for people like Erika and me who have used Reporting Services for years but who have might have used certain features only sporadically. For reasons unknown to me, this book has had a very long production cycle. I finished writing in July, but the book won’t be available until April 2013. And things have changed since then with the release of SQL Server 2012 SP1 with regard to Power View. It’s too late to add onto the relevant chapters now. And of course during the review process, I discovered all sorts of things I wanted to add in other areas, but I have to draw the line somewhere. I can see a whole series of posts coming up to supplement the book!
  • Business Intelligence in Microsoft SharePoint 2013. Which makes book #13. It’s a second edition with lots of updates thanks to the changes in Office 2013 – both in Excel and in SharePoint. My contributions to this book were chapters on Planning for BI Adoption and Power View in Excel 2013. There are some very cool things in Power View now that I wish could have been added to book #12, but c’est la vie. Technology moves at a faster pace than book production! Look for the release of this book in March 2013.
  • SQL Pro Magazine articles. I’ve written for SQL Pro for years and am now a contributing editor as well. In 2012, I contributed more than usual:
  • Books #14 and #15 are in the wings. They’ll be e-books so I’m not sure I can really count them as books, but it’s my list, so I can count the way I want to! I officially started the draft of #14 last night. And I have a meeting scheduled for #16 next month. Call me crazy… or is that maybe? I get so confused. (Need a laugh? Check out the Call Me Maybe parody that I and the Pluralsight gang produced.)

Accidental Business Intelligence? Not Really

But back to the musing…I always think of my career in BI as accidental, although taking a longer view I realize that it really wasn’t. In February 2011, I told my story to Andy Leonard (blog | twitter) as part of his SQLPeople series of interviews. There are some karmic aspects to that story that I would never put in print, but I’d be happy to tell you in person some day if you ask nicely.

In that story, I mention working with Lotus Notes. In the late 1990s, I had this feeling that using technology as a knowledge management tool was an attainable goal and put a lot of energy into learning how that should happen. But then I got deflected into business intelligence which turned into a fascinating and rewarding career. But all along, I had this nagging feeling that BI was just part of the story. I wasn’t satisfied with just delivering on reporting and analysis. As important as that is, I believed additional transformation in the way we work with data and with each other was necessary in order for BI to fulfill its promise.

Collaborative BI

I’ll admit that in the beginning I wasn’t very impressed with SharePoint – I believe it was SharePoint 2003 when I was first introduced to it. I had been working with Lotus Notes long before that time, and felt that it could run circles around that release of SharePoint. Full disclosure – I haven’t looked back at Lotus Notes since I left it, so I have no idea of its capabilities today. But starting with SharePoint 2007, I started thinking beyond the traditional dashboard compilation of scorecards and reports. I was thinking about unstructured data to support the structured, and would mention it in my presentations and classes. Then with SharePoint 2010, I started thinking about the collaborative and social aspects and started putting these pieces together with ideas that I had been nurturing since the late 1990s. And so, a presentation was born for a webinar, a few SQLSaturdays, and continues to evolve as I gear up for the PASS Business Analytics Conference in April 2013. Because an hour presentation only sets the stage for some of my ideas, I have set up a Collaborative BI resource page that will grow as I commit these ideas to writing.

Big Data

Meanwhile, the buzz around Big Data became louder in 2012. Now I’ve been around a few years, and I’ve seen buzz come and go. I had plenty to keep myself busy meanwhile during 2012, and just watched and waited to see what would happen. And then things started to get interesting. So much so that it’s time to start talking about it. Consequently, I am working on presentations on this topic throughout the year (keep an eye on Upcoming Events for online and in-person events), including a session at the PASS Business Analytics Conference on Power View and Hadoop in collaboration with Joey D’Antoni (blog | twitter).

And that’s just the beginning. I plan to add another resource page for my thoughts on BI and Big Data. Watch for more blog posts and presentations.

When I think back to my “accidental” discovery of BI and the ideas we were throwing around at the time, I realize we were a bit ahead of our time. What we needed at that company to achieve those big ideas was big data – we just didn’t call it that then. We started on a much smaller scale and focused on data warehousing and reporting and analysis tools, and we were barely ready for that then. I work with customers today who are still barely ready for that.

Data Science

But now in the era of big data and data science, I start thinking about those big ideas again and how much more attainable they are today 14 years after I started down this path. The BI world is poised for the biggest change I’ve seen in my career. While I cannot share the specifics of what we were thinking about in my R&D days, I can try to explain how I see the difference between BI as we traditionally think of it and where data science can take us. BI helps us understand what happened or what is happening now, using established processes and tools. Although BI can scale quite dramatically, scale introduces some complexities that in some ways limits the types of reporting and analysis that we can do. Data mining is often included in a discussion of BI technologies, but its use has not been very prevalent in my client base. Data mining can not only be used to explore data to help us understand what happened, but can also be used to predict what might happen. And this is where we see data science come into play now. Data science can help us look forward and to predict an outcome or a correlation. It incorporates many techniques that are common to data mining, but it can go beyond those techniques as well. We can work with larger data sets than ever before because we can store data more cheaply than ever before and we have better tools for dealing with these larger data sets using commodity hardware.

Is the Data Warehouse Dead?

No, I don’t think so. At least not completely. There’s still a place for operational and mission-critical information that’s been consolidated, cleansed, and corporately-sanctioned as truth. I don’t really care what we call that information source – a data warehouse, a data mart, whatever. We need access to that type of information because that’s how we decide what to do today to achieve our goals, respond to specific problems, or show the board (or the world) how we’re doing as a business. The new potential with big data and data science is the opportunity to explore data in ways never before possible. We don’t know what the opportunity or business value in that data might be until we examine it in new ways or combine it with other types of data–data that was captured by others and shared publicly is just one example. Creativity is key. But like traditional BI, in my mind, it’s all useless unless we can DO something with that information. And of course, we need to share and collaborate!

What do you think? Is BI as we know it going to to die or thrive in this brave new world of big data?

I look forward to delving more into these topics more in future posts. I hope you do, too!

10 Responses to “Musings on a BI Career: Past and Future”

  1. “For reasons unknown to me, this book has had a very long production cycle. I finished writing in July, but the book won’t be available until April 2013.”

    Yeah, I got a dose of that with SSIS Design Patterns last year. I knew there was a lot that went into publishing a book, but was unpleasantly surprised at the amount of lag between when the last edited words hitting the page and the actual publication date.

    Sounds like you’ve had quite a busy year! Hope to see you at a few events in 2013.

  2. My experience with my previous books was a shorter cycle. But I don’t know that I could have properly handled some of the changes that needed to be made earlier in the year (a new requirement is adding descriptive text to each figure-it’s like writing half a book again!)

    And I have no doubt we’ll cross paths somewhere this year!

  3. Ahah! I always suspected that even those that write the books can’t keep track of all the features. :-)

    I am going to be attempting a proof-of-concept deployment of SharePoint 2013 BI for my organization, a small non-profit medical association. It’s too hard to figure out exactly what it can and can’t do for us just from reading and watching demos. I am trying to convince our IT contractor to support it for us, like they do Exchange now. It would be nice to be able to offer them a book containing guidance for deploying SharePoint 2013 for a small organization. Is there any chance of finding that in your book?

    We have a small data warehouse (currently running on SQL Server 2013 Analysis Services multidimensional mode), and I don’t see that going away. There are potential projects on the horizon, though, that might involve Big Data, and their developer (me) would have to be prepared for that. More books presentations (and more books) sounds like a good idea! I am hoping, though, that “big” in our world will still be relatively small and that we can continue with what we already know.

    By the way, if we do become involved with Big Data then it must be secure as well, since at least some of it is likely to fall under HIPPA regulations. I don’t even have a sense of the size of that issue at this point.

  4. Oops — make that SQL Server 2012 Analysis Services. Where did 2013 come from?

  5. Nice post Stacia and looking forward to see what’s in store in 2013.

  6. Happy new year Stacia!

    I thought Megan’s comment on features was interesting and is a common perception.

    I think that the main reason for writing a book nowadays is to consolidate your own knowledge on a topic and to make you explore it deeply. It certainly wouldn’t be for the income. The ability to explain something to others is the best indication that you understand it yourself.

    Never feel bad about discovering things that you don’t understand while you are writing about them.

    In terms of data warehouses, I think the line that’s often pushed by Microsoft now, that you can just layer a tabular model over an existing transactional or source system, is seriously misguided advice. Most transactional or source systems are horrid to work with, have loose integrity (at best), don’t deal with versioning of data, etc. etc. I still find it useful to collect all the required data, in a form that’s suitable for analysis, in a single place whenever possible. Reporting and analytics are then trivial by comparison.

  7. Hi Megan – There are a LOT of features in Reporting Services. I’ve been working with it since before SSRS 2000 was released and every implementation I’ve done has been completely different. It’s an amazingly flexible product and it’s fascinating to see how many different ways it can be used. But there are also things I never or rarely use, too. :)

    It’s possible that the BI in SharePoint 2013 will provide some guidance. It’s not completely finished so I can’t say with certainty at this point, but I do know that it’s the goal to help organizations like yours succeed.

    “Big” means a lot of different things right now. I’ve also seen where it can be small volumes of data. Here’s an interesting article about big data for small business: http://www.forbes.com/sites/ciocentral/2012/12/28/big-data-analytics-not-just-for-big-business-anymore/. They do focus on social media which I’m not convinced is always relevant, but setting that aside, there are some good tips here.

    Please stay in touch. I’d love to keep a dialog going and keep in mind how this big data – and the inherent security issues with HIPPA – affect organizations like yours!

  8. Happy New Year, Greg! Agreed on multiple points there. Especially with regard to data warehousing. I don’t see it going away. I call the new architecture a compromise solution.

  9. Thanks, Zach – so glad to hear from you! I think it’s going to be an interesting and instructional year!

  10. [...] I mentioned in my New Year’s musings, I felt this was the year to start paying more attention to Big Data. Accordingly, I have begun [...]

Leave a Reply