It can be a bit daunting to get started with these other languages, especially when your T-SQL mindset keeps getting in the way. Never fear. I’m here to help! There’s still time to register for my SQLSaturday #220 Preconference Workshop, Languages of BI, this week in Atlanta on Friday, May 17.

In my workshop, you’ll learn which languages each presentation layer tool supports and the basic syntax used by each language. You’ll also learn how take advantage of query builder tools to help you produce queries until you’ve developed the proficiency to create more complex queries on your own. In addition, you’ll learn how to work with expressions to add calculations to cubes and PowerPivot workbook, and how to use expressions to customize the appearance of reports, scorecards, and dashboards.

And if you come out to the workshop, you might as well stay for SQLSaturday, too. This will be my second year for this event in Atlanta, and I’m looking forward to seeing longtime friends in the SQL community and making some new ones. There are a whopping 11 tracks on the schedule, with 3 devoted to BI topics. I’ll be presenting Troubleshooting MDX Query Performance, which is a nice complement to the Friday workshop. Hope to see you there!

Although I think there are certainly some claims made out there about Big Data that fall into these categories, I lean towards describing the current rising interest in the topic as hyperopia, a form of farsightedness that our data management industry is experiencing. In this sense, hyperopia refers to temporal rather than spatial distance. I think we can see something happening with Big Data now but it’s rather fuzzy still, whereas the future implications are more clear.

Put another way (if the vision analogy doesn’t work for you), Big Data is no longer in its infancy. But it’s not fully mature yet either. Toddler? Preteen? Teen? I’m not sure where it is in this cycle, except certainly not at the teen stage. Of course, we can expect each stage in this maturity cycle to have its ups and downs. It’s a normal part of the process of technological change. Let’s not expect too much where it’s not appropriate and give it full support where and when it’s ready.

As I mentioned in my New Year’s musings, I felt this was the year to start paying more attention to Big Data. Accordingly, I have begun working with some of the technologies in the Big Data ecosystem and started speaking about it. Of course, I’ve been reading as much as I can, too. As a regular conference speaker, I highly value the educational and networking opportunities unique to conference attendance, so was thrilled to have the opportunity to attend GigaOM’s Structure:Data 2013 in New York City this week.

The Structure:Data conference used a different format than I’ve ever seen at a conference. Each session lasted 15 minutes with the attendees primarily positioned in one spot for the day. The speakers were rotated on and off the stage throughout the day with an emcee making introductions to the next session. The session styles varied also, from moderated panel to a speaker at a podium with slides to a speaker center stage without slides. I recall only one demo and even that was pre recorded.

The frequency with which speakers and formats changed meant that I didn’t get too antsy sitting in one spot for an extended period. There were also longer 45-60 minute workshops that allowed us a) to get up and move to a different place for a while, and b) to get exposed to a topic in a bit more depth than the main stage topics. Overall, I liked the format for its ability to introduce me to a lot of ideas in a compressed timeframe. The irony did not escape me though….Big Data in small bites.

Hands down, my favorite session was The CIA’s “grand challenge” with big data delivered by Ira “Gus” Hunt, CTO for the Central Intelligence Agency. His talk (online here) was fascinating and even a bit scary. Did you know your smartphone can identify you by your gait? That’s just one of many tidbits Hunt shared. Others that struck me most in this session:

• Big Data allows us to grow the haystack and magnify the needles.
• information no longer flows from the few to the many, but from the many to the many, thereby generating more and more data. Sure, there’s a lot of noise there, but there’s also a signal to be discovered lying within.
• Sensors in everything results in explosive growth in data volumes. It’s not just Big, but Really Big. Sensors monitor location, health and even identity (through gait, for example).
• Through analysis of sensor data, the inanimate can become sentient. (Oh, dear…)

• We don’t know the future value of data.
• We cannot connect the dots that we don’t have.
• Traditional requirements analysis don’t work in the world of Big Data.

Of particular interest to me, and a theme I heard many times over at the conference, was Hunt’s contention that the power of Big Data can be achieved only when the end user can interact it. We cannot expect users to be data scientists, nor can we expect them to be dependent on data scientists either. Yet, he says, analytical tools for Big Data are hard to use. (I stopped at some vendor booths to see what they had to show, but in fairness to all need more time to digest their offerings before I can comment.) He cited Wikipedia’s definition of data science, and in particular the following quote, “There is probably no living person who is an expert in all of these disciplines – if so they would be extremely rare.”

So are our academic institutions going to miraculously churn out hordes of data scientists to place in every business? I don’t think so. I don’t have anything against data scientists in general, but I think the mission of accomplishing Big Things with Big Data is better served if their skills are used to develop the next generation(s) of software that empower the business user, rather than having them become yet another barrier between regular people and data just because the data gets “bigger.” Or is this just wishful thinking on my part?

The goal of this presentation was to provide an introduction to terms and technologies. The term “Big Data” dominates the headlines in technology news these days, but what is the fuss really all about? I’ve seen hype come and I’ve seen hype go in my career, and waited to see what would come of it. In my presentation, I try to paint a picture for you that gives you a better idea of the many ways that the Big Data movement is impacting the use of data. Then I spend some time discussing some of the technologies that you could use to manage Big Data. And last I delve into approaches you can take to manage a Big Data project. Along the way, I compare and contrast Big Data with traditional data warehousing and BI.

You can find a copy of my presentation here.

Allow me to set the stage properly for this property. You have a table in a multiple page report. Of course, you set the RepeatOnNewPage property to display the table header row(s) on each page. But did you know you can also repeat other items alongside the table when using the HTML or PDF format?

To do this, set the RepeatWith property of the item to repeat. The item will repeat alongside the table on every page that contains the table. You can use the RepeatWith property of three types of report items: text box, rectangle, and line. To set the property value, you select the name of a data region—table, matrix, list, or chart—having the same parent as the item to be repeated. In my example above, both the table and the text box have the same parent: Body.

In SQL Server 2005 and earlier, you don’t see the text box repeated with the table in Preview mode within BIDS, but you can confirm the results by deploying the report to the report server. I haven’t tested in SQL Server 2008 or SQL Server 2008 R2, but it definitely is visible in preview mode in SQL  Server 2012 in SSDT as you can see in the images above.

Incidentally, if you specify the RepeatWith property for a rectangle that contains a data region, you cannot render the report.

The problem is that this data in this format doesn’t help me know whether those values are good or bad. And of course I need to know what defines good or bad. Do I have a sales target to compare to? Do I want to know whether the quarter-to-date sales are better than the same time last quarter or the same time last year?

Let’s assume that there is an established sales target. Without the information in the dashboard, the viewer is forced to go look up a report that lists the forecasts by territory and then do calculations to figure out how close or far away each territory is from that goal. That defeats the purpose of the dashboard which should provide information at a glance. Consider the following chart as an alternative to showing a table.

Here the only two countries for which a quarter-to-date sales target exists for the time frame represented are Canada and the United States. Notice that the column chart reflects the variance. You can see that Canada has sales that are almost 100% of the goal. Wow – not only did Canada meet its goal but has almost doubled the goal. Great news! On the other hand, the US is not meeting the goal. Whether that amount by which its not meeting the goal is a little or a lot depends on management expectations. The chart could be enhanced by adding a tooltip that displays the variance percentage value, the QTD sales, and QTD target to provide even more context without cluttering up the visualization.

This is just one way to present a visualization that provides context for sales. At a glance, it’s easy to tell where things are good and where things are not so good.

Want to learn more tips like these? Join me and a great lineup of speakers at the PASS Business Analytics Conference in Chicago on April 10-12. Here are some related sessions to help you build your skills:

• Advanced Dashboards Using Excel, Excel Services, and PerformancePoint
• Advanced Data Visualization with SQL Server Reporting Services
• Excel Charting Tips

Hope to see you there!

]]> http://blog.datainspirations.com/2013/02/14/dashboard-design-tip-3-provide-context/feed/ 1 http://blog.datainspirations.com/2013/02/12/let-go-of-my-ispac/ http://blog.datainspirations.com/2013/02/12/let-go-of-my-ispac/#comments Tue, 12 Feb 2013 12:36:06 +0000 Stacia Misner http://blog.datainspirations.com/?p=883 Sometimes it happens. SQL Server Data Tools crashes and burns. (It’s also known as Visual Studio or SSDT but I don’t know how to pronounce that…) Was it something I said? I’m currently building a small data warehouse and building out a lot of packages in a fairly short period of time, so maybe I was wearing out good ole’ SSDT. After setting up a new package, I tried to execute it and the application hung. I waited patiently. Nothing. Waited some more, and then decided I had no choice. I had to put SSDT out of its misery and (hide the children) kill the process. Now this was something that I encountered often enough when working with earlier versions of SQL Server using Business Intelligence Development Studio (or the more pronounceable BIDS), but SSDT has been much more stable for me. Until this moment.

After reopening SSDT, I tried to run my package, but got this error:

I’m good at following directions. This is what I found in the Output window:

------ Build started: Project: DW_ETL, Configuration: Development ------

Build started: SQL Server Integration Services project: Incremental ...

Starting project consistency check ...

Project consistency check completed. The project is consistent.

Error : System.IO.IOException: The process cannot access the file

'C:\Users\<snip>\<solution>\<project>\bin\Development\DW_ETL.ispac'

because it is being used by another process.

   at System.IO.__Error.WinIOError(Int32 errorCode, String maybeFullPath)

   at System.IO.File.Delete(String path)

   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectBuilder.

IncrementalBuildThroughObj(IOutputWindow outputWindow)

   at Microsoft.DataTransformationServices.Project.DataTransformationsProjectBuilder.

BuildIncremental(IOutputWindow outputWindow)

Build complete -- 1 errors, 0 warnings

========== Build: 0 succeeded or up-to-date, 1 failed, 0 skipped ==========

And for good measure, I also looked in the Error window which contained essentially the same message.

The ISPAC file is necessary for deploying a project to the SSISDB catalog. It gets built automatically when you execute a package in SSDT, or you can manually build it by using the Build command in SSDT. Its purpose is to consolidate all the packages and parameters for your project (assuming you are using the default Project Deployment mode), so that you can deploy the entire project to the SSISDB database as a single unit. In other words, you can hand off one file to an administrator of the Integration Services server and they can deploy it onto the server.

During the development process, I don’t need the ISPAC file. And if it’s being used by another process, ostensibly the one I just killed, but blocking my current process, I should just kill it, too. The problem I found is that it was good at self-defense. When I tried to delete the ISPAC file, I got the following message:

Okay, fine, I closed SSDT and tried to delete the ISPAC file again. Same message.

Next I went to Task Manager and found two processes. DTSDebugHost.exe and DTSDebugHost.exe *32. Can you guess what I did? I had to be pretty ruthless yesterday. I don’t feel good about it, but it had to be done. After the deed was done, I never looked back. I reopened SSDT and ran my package successfully. All’s well that ends well.

In a recent Software Advice interview, Rob hits the highlights of the new Excel by answering questions about BI functionality in Excel, the message about self-service BI that it sends, the motivation for users to upgrade, the impact of Office Apps, and his vision for Excel users. I agree with Rob’s contention that having PowerPivot (and of course Power View) available directly in the product is going to help users discover this functionality faster than they would if they had to rely on finding, downloading, and installing an add-in. However, I do have a concern that this functionality still might not be obvious. Although PowerPivot is built into Excel now, it’s still considered an add-in, and furthermore it’s disabled by default. To use it, you have to enable the PowerPivot add-in, which isn’t difficult but isn’t necessarily obvious.

So I’m not sure that people are going to discover PowerPivot on their own without the PowerPivot tab of the ribbon enticing them to explore. But they might happen to notice changes to the Import Data dialog box that mentions the Data Model.

The Data Model gives you an entry-level path to PowerPivot. You can bring together data that you can use to build Power View sheets in Excel, the other great new feature in Excel 2013. Although there is also some set up required before you can use Power View, the process is a bit smoother. When you first attempt to insert a new Power View sheet, Excel prompts you to enable the add-in and also to install Silverlight. As long as you have data in the Data Model, you’re off and running.

So how does that relate to PowerPivot? Well, the Data Model is simple, based on the data you feed it. Power View allows you to build great visualizations, but what if you want to refine the model by renaming a field or adding a calculation? When users decide the basic Data Model isn’t enough, that’s when they’ll want to learn more about the Data Model and discover that PowerPivot takes them deeper into data analysis.

As Rob mentions, I don’t think this capability makes every user a data analyst, but Excel 2013 provides something for everyone. Those users who just need simple spreadsheet calculations can do just that, while others who would like to explore and enhance their data with visualizations have some amazing capabilities without having to take a week’s worth of training, and those who are already comfortable with embedding advanced formulas into a worksheet will quickly learn and take advantage of the features in PowerPivot that support richer modeling of the data. It’s great to see the evolution of Excel, and especially to see people get excited about their data when they try out this latest version.

The purpose of the scale break is to prevent a category with a small value, like France in the chart above, from virtually disappearing from view because other categories like Canada and United States have significantly higher values. The problem is that your brain doesn’t easily interpret those scale breaks. A glance at the chart might lead you to think that sales in France are roughly half of sales in the UK which in tern are roughly half of the sales in the US – which is not the case.To interpret the chart correctly, you have to carefully review the values on the scale and then mentally calculate the difference between two categories rather than rely on the size of the column. And that violates the rules of dashboard design. If the viewer must start doing math to understand the message, you’re not doing it right!

When the scale break is removed, you can more easily discern the correct relative sizes between categories. You can see that Canada sales are under half of US sales and that the difference between France sales and US sales is significant. Sure, you can’t see the actual value here but remember that the purpose of the dashboard is to communicate information visually. You should understand what you need to know immediately, and in this case, the information we glean from the revised chart is where sales are high and where they are low.

As a side note, I also cleaned up the chart a bit by removing the gridlines and reducing the emphasis of the non-data points. That is, I used a softer color (gray) for the labels and the axis lines because they are merely supporting features and not the data points themselves. The data points here are represented by the columns and pop out from the chart. Your eye focuses on the column sizes and is not distracted by extra lines that contribute no meaning.

To communicate sales values, the new chart is better than the old chart. But from a dashboard perspective, are these values what we expect to see? Are these good values or bad values? For use in a dashboard, there is more improvement that should be made to enhance the usefulness of this information, but I’ll save that discussion for a future post.

P.S. You can introduce a similar problem with skewing the message when you start the scale for a column or bar chart at a value other than 0. The relative size of the columns and bars are not reflected accurately when you do this.

Continue on with Dashboard Design Tip #3: Provide Context.

Because 60 minutes is a very short period of time to fully explore the Do’s and Don’ts, this post and others to follow will explore the various tips that I discussed (in no particular order) and show the “Before” and “After” shots of dashboard elements used in the demonstrations. The focus of today’s post is on the type of measurement you use in a chart.

My dashboard examples in the presentation use sales data for the fictional AdventureWorks company. Furthermore, I used SharePoint 2013 and SQL Server 2012 Reporting Services to produce the examples, although it’s possible to use earlier versions (or even other software) to produce similar results in most cases. Most of the elements in my first dashboard are Reporting Services web parts because I can manage the exact look and feel for a chart by using Reporting Services. Plus I have many more options for data sources.

Because a dashboard is often used to measure progress towards goals, one of the items in my dashboard compares actual sales (blue line) to the sales quota (orange line) over a period of time on a scale of absolue dollars:

At a glance, you can see there are trends for both sales and quota values that track together generally in the same direction. Much of the time, sales are below the quota, while only twice do we see sales above quota. However, it’s difficult to gauge the relative difference. Is the difference a lot or a little? Of course, a lot or a little can be subjective values, but we can probably agree that a 60% variance would be a lot while a 5% variance would be a little. Regardless, we cannot describe those relative differences by looking at this chart. At least not very easily, which defeats the purpose of a dashboard.

As I mentioned in my presentation yesterday, a dashboard should present information in such a way that I can glance at it and understand what I need to know immediately. In the chart above, I immediately understand that in general sales are not meeting quotas. But if I want undertand by how much (i.e. the relative difference), my eye has to travel to the left to find the scale value for two different points and do the math. Of course, if I’m going to go to that much trouble, I really care about the value, right?

To make it easier to arrive at the value of interest – the variance, we need a chart that shows that value like this:

Now although this chart is a better chart for seeing the swings in variance over time, it didn’t make it into the final dashboard. At least not the one that I showed yesterday. I reserve the right to fine-tune the dashboard as I progress through this blog post series! After all, a dashboard is allowed to evolve based on user feedback.

In fact, to set the stage for future posts leading to the final dashboard, let’s review the makeover plan:

1. Identify the dashboard type and audience
2. Identify the metrics
3. Select the visualization
4. Add support and interaction
5. Keep it simple

For steps 1 and 2, I interviewed the fictional AdventureWorks sales managers. (An easy interview, by the way…I get to make up my own requirements for a change, so just roll with me here…) As a result, I determined the goal of the dashboard makeover is to produce an analytic dashboard for the sales management team. Together, we also identified a limited number of metrics as the focus of the dashboard: sales variance by product mix and margins by territory.

That means many of the items in the original dashboard will get eliminated, at least from this first dashboard. However, they instead become candidates for other dashboards having different requirements, so stay tuned as we explore in future posts how to fix common mistakes.

Bottom line for today–be clear about what you’re measuring and what the user really needs to know. If there is a relationship between two measurements that you’re tracking, such as an actual value and a target value, it’s sometimes better to display that relationship as a percentage rather than use separate bars or separate lines to compare the values side by side. Common scenarios to which this would apply would be showing sales versus quota, expenses versus budget, or sales versus costs.

Continue to follow this series with Dashboard Design Tip #2: Use Scales Correctly.

What’s Been Happening?

This post also breaks my long silence. I have spent a significant amount of time the past year writing in several forms that consumed my time:

• Microsoft SQL Server 2012 Reporting Services. That’s book #12 for me, co-authored with my daughter Erika Bakse who has since moved onto another job (in the hopes that she’ll never have to write another book, no doubt!). I’m very excited about this book because it goes into more detail than I ever could in the Step By Step series and consequently becomes a great reference not only for beginners, but for people like Erika and me who have used Reporting Services for years but who have might have used certain features only sporadically. For reasons unknown to me, this book has had a very long production cycle. I finished writing in July, but the book won’t be available until April 2013. And things have changed since then with the release of SQL Server 2012 SP1 with regard to Power View. It’s too late to add onto the relevant chapters now. And of course during the review process, I discovered all sorts of things I wanted to add in other areas, but I have to draw the line somewhere. I can see a whole series of posts coming up to supplement the book!
• Business Intelligence in Microsoft SharePoint 2013. Which makes book #13. It’s a second edition with lots of updates thanks to the changes in Office 2013 – both in Excel and in SharePoint. My contributions to this book were chapters on Planning for BI Adoption and Power View in Excel 2013. There are some very cool things in Power View now that I wish could have been added to book #12, but c’est la vie. Technology moves at a faster pace than book production! Look for the release of this book in March 2013.
• SQL Pro Magazine articles. I’ve written for SQL Pro for years and am now a contributing editor as well. In 2012, I contributed more than usual:
  • Understanding SQL Server Reporting Services Authentication
  • Touring the Tablix in SQL Server Reporting Services
  • Implement Kerberos Delegation with SSRS
• Books #14 and #15 are in the wings. They’ll be e-books so I’m not sure I can really count them as books, but it’s my list, so I can count the way I want to! I officially started the draft of #14 last night. And I have a meeting scheduled for #16 next month. Call me crazy… or is that maybe? I get so confused. (Need a laugh? Check out the Call Me Maybe parody that I and the Pluralsight gang produced.)

Accidental Business Intelligence? Not Really

But back to the musing…I always think of my career in BI as accidental, although taking a longer view I realize that it really wasn’t. In February 2011, I told my story to Andy Leonard (blog | twitter) as part of his SQLPeople series of interviews. There are some karmic aspects to that story that I would never put in print, but I’d be happy to tell you in person some day if you ask nicely.

In that story, I mention working with Lotus Notes. In the late 1990s, I had this feeling that using technology as a knowledge management tool was an attainable goal and put a lot of energy into learning how that should happen. But then I got deflected into business intelligence which turned into a fascinating and rewarding career. But all along, I had this nagging feeling that BI was just part of the story. I wasn’t satisfied with just delivering on reporting and analysis. As important as that is, I believed additional transformation in the way we work with data and with each other was necessary in order for BI to fulfill its promise.

Collaborative BI

I’ll admit that in the beginning I wasn’t very impressed with SharePoint – I believe it was SharePoint 2003 when I was first introduced to it. I had been working with Lotus Notes long before that time, and felt that it could run circles around that release of SharePoint. Full disclosure – I haven’t looked back at Lotus Notes since I left it, so I have no idea of its capabilities today. But starting with SharePoint 2007, I started thinking beyond the traditional dashboard compilation of scorecards and reports. I was thinking about unstructured data to support the structured, and would mention it in my presentations and classes. Then with SharePoint 2010, I started thinking about the collaborative and social aspects and started putting these pieces together with ideas that I had been nurturing since the late 1990s. And so, a presentation was born for a webinar, a few SQLSaturdays, and continues to evolve as I gear up for the PASS Business Analytics Conference in April 2013. Because an hour presentation only sets the stage for some of my ideas, I have set up a Collaborative BI resource page that will grow as I commit these ideas to writing.

Big Data

Meanwhile, the buzz around Big Data became louder in 2012. Now I’ve been around a few years, and I’ve seen buzz come and go. I had plenty to keep myself busy meanwhile during 2012, and just watched and waited to see what would happen. And then things started to get interesting. So much so that it’s time to start talking about it. Consequently, I am working on presentations on this topic throughout the year (keep an eye on Upcoming Events for online and in-person events), including a session at the PASS Business Analytics Conference on Power View and Hadoop in collaboration with Joey D’Antoni (blog | twitter).

And that’s just the beginning. I plan to add another resource page for my thoughts on BI and Big Data. Watch for more blog posts and presentations.

When I think back to my “accidental” discovery of BI and the ideas we were throwing around at the time, I realize we were a bit ahead of our time. What we needed at that company to achieve those big ideas was big data – we just didn’t call it that then. We started on a much smaller scale and focused on data warehousing and reporting and analysis tools, and we were barely ready for that then. I work with customers today who are still barely ready for that.

Data Science

But now in the era of big data and data science, I start thinking about those big ideas again and how much more attainable they are today 14 years after I started down this path. The BI world is poised for the biggest change I’ve seen in my career. While I cannot share the specifics of what we were thinking about in my R&D days, I can try to explain how I see the difference between BI as we traditionally think of it and where data science can take us. BI helps us understand what happened or what is happening now, using established processes and tools. Although BI can scale quite dramatically, scale introduces some complexities that in some ways limits the types of reporting and analysis that we can do. Data mining is often included in a discussion of BI technologies, but its use has not been very prevalent in my client base. Data mining can not only be used to explore data to help us understand what happened, but can also be used to predict what might happen. And this is where we see data science come into play now. Data science can help us look forward and to predict an outcome or a correlation. It incorporates many techniques that are common to data mining, but it can go beyond those techniques as well. We can work with larger data sets than ever before because we can store data more cheaply than ever before and we have better tools for dealing with these larger data sets using commodity hardware.

Is the Data Warehouse Dead?

No, I don’t think so. At least not completely. There’s still a place for operational and mission-critical information that’s been consolidated, cleansed, and corporately-sanctioned as truth. I don’t really care what we call that information source – a data warehouse, a data mart, whatever. We need access to that type of information because that’s how we decide what to do today to achieve our goals, respond to specific problems, or show the board (or the world) how we’re doing as a business. The new potential with big data and data science is the opportunity to explore data in ways never before possible. We don’t know what the opportunity or business value in that data might be until we examine it in new ways or combine it with other types of data–data that was captured by others and shared publicly is just one example. Creativity is key. But like traditional BI, in my mind, it’s all useless unless we can DO something with that information. And of course, we need to share and collaborate!

What do you think? Is BI as we know it going to to die or thrive in this brave new world of big data?

I look forward to delving more into these topics more in future posts. I hope you do, too!