Click here to subscribe to our RSS feed

FYI Solutions Blog

Nov 19, 2014

Make the Most of the New IBM Cognos Information Distribution License

Authors: Jeff Busch and Joe Rodriguez

As part of their Cognos licensing evolution, IBM has created a new license entitlement called Information Distribution. Information Distribution users inherit the capabilities of the former Remote Recipient users, but also gain some additional capabilities that add significant benefits for Cognos customers. In this blog we will explore some of these benefits as well as briefly describe the best method for implementing this new role.

One of the major challenges Cognos managers face is how to get the greatest benefit from their licensing and development investment. Maintaining license compliance, while providing the features that users need, requires constant monitoring of usage and coordination with the business. The old entitlements often required managers to purchase higher level licenses or add-ons to provide the access needed. Consumers could run pre-built reports but not Active Reports; this required an upgrade to an Enhanced Consumer license. Many users just need to run simple reports.  Remote Recipient licenses were cheaper and could receive scheduled report output and burst reports, but couldn’t access Cognos directly to run these reports for themselves when they needed them nor could they interact with an Active Report. Alternatively, a Recipient license could be used, although that entitlement did not include rights to receive or view Active Reports. In addition, keeping track of which Consumer, Recipient and Remote Recipient users actually made use of the access granted them, and thus being able to maintain only the number of licenses needed, is difficult at best albeit necessary to be in compliance with license entitlements. Also, some companies may have security restrictions preventing pre-run reports from being sent to users requiring Recipient or Consumer licenses when Remote Recipients could otherwise be used.

For all of these reasons and more, the new Information Distribution license is ideal. Information Distribution users can receive pre-run reports through email, including Active Reports and burst reports;  they can also access Cognos directly to view the saved content for themselves. Now you have extended delivery options. Further, you can setup an area in Cognos where light-duty users can go to download pre-run reports and Active Reports and you can send links to users instead of the actual report. Access to the reports using the links would utilize Cognos security measures already in place.

Another key feature of the Information Distribution user licenses is that they are PVU only. This means that they are not purchased on a user by user basis (i.e. named user), but are applied to a specific Cognos server based on the number of processor cores in the server. For those who are new to the concept of PVUs, a Processor Value Unit (PVU) is a unit of measure by which the Program can be licensed. The number of PVU entitlements required is based on the processor technology (defined within the PVU Table by Processor Vendor, Brand, Type and Model Number (click here to go to the IBM PVU calculator page). This is a benefit to the customer because they no longer need to keep track of individual users when those users simply consume pre-run content. The PVU-based licensing will require customers to take a second look at their environment infrastructure. If, for example, your environment has two redundant dispatchers with 12 CPU cores each you would need to purchase enough Information Distribution licenses for 24 cores. This is because both servers will by default be handling requests from all users. This may not appear to be cost effective for your organization. Fortunately, this is actually an easy challenge to solve.

Cognos has features built in that will allow you to add a new light-duty server to the environment and dedicate that server to Information Distribution user activity. Continuing our example above, you might choose to add a new dispatcher server with 2 cores. The server can be configured with a fewer number of cores because delivering saved content is much less processing intensive than actually running the reports. As long as the Information Distribution users’ activity is restricted to the new server, you will need to maintain only a two-core license (this equates to approximately 200 PVUs by IBM guidelines). This will also have the additional benefit of transferring some of the activity load off of the primary server. You could even designate the new server for all batch processing or other similar tasks taking even more of the processing load off of the primary server(s). If you are designing an entire environment from scratch, or are designing an upgrade to your existing environment, you may be able to decrease the hardware requirements of your primary servers and thus decrease the cost of those servers.

The ability to provide direct, secure access to a company’s report content without having to purchase more advanced licenses is a clear advantage of the new Information Distribution license. As an added benefit, administrators will not need to spend time keeping track of these new users’ usage to maintain compliance because the licenses are PVU based and the users will have a server dedicated to their needs. Explaining the details of implementing this strategy of dedicating a server to Information Distribution users is outside of the scope of this blog.

As an IBM Premier Business Partner for Business Analytics, FYI Solutions has expertise in implementing these solutions and is available to help you with your implementation.  Please contact us for more information.

Nov 13, 2014

IBM Unleashes Watson Analytics Tool to Help Businesses Make Sense of Big Data

Author: Barbara Schiffman

As Jeopardy is broadcasting its annual Tournament of Champions this week and next, it brings back the memories of IBM’s Watson taking on former winners Brad Rutter and Ken Jennings. Watson is an artificially intelligent computer system capable of answering questions posed in natural language.   For Jeopardy, Watson had access to 200 million pages of structured and unstructured content consuming four terabytes of disk storage including the full text of Wikipedia.

Watson Analytics has moved on to take on more pressing business priorities. IBM wants to unlock big data secrets for businesses with a new free Watson Analytics tool that allows clients to leverage the “Jeopardy”-winning supercomputer to explore and unearth valuable data from structured data sets.

Tom Spring, Senior Editor of CRN Magazine, offered the following summary of the new capabilities in his recent blog, found at

The technology now allows businesses to upload data to IBM’s Watson Analytics cloud service and then query and explore results to spot trends, patterns, and conduct predictive analysis. The Watson Analytics tool is free; however, IBM executives said the goal is to demonstrate the power of the technology so users will become paying customers with access to advanced features such as slicing and dicing larger data sets, correlation with live data feeds, and advanced analysis.

“We are helping companies make better fact-based decisions through visual analysis,” said Eric Sall, IBM vice president, business analytics. “It’s still way too hard for most companies to make sense of the wealth of data they are sitting on.”

Watson Analytics is aimed at enterprises but also at ISVs and mid-tier companies that lack a big data budget, according to IBM, Armonk, N.Y. Watson Analytics also will later be available through its Bluemix developer’s cloud services offering. This link will take you to the beta offering.

Watson Analytics parses data, cleans it up, preps it for analysis, identifies important trends, and makes it easily searchable via natural language queries, according to Sall. The tool, for example, could help companies better understand customer behavior or connect the dots between sales, weather, time of day and customer demographic data, he said.

“For mid-tier companies that can’t afford a dedicated business analyst, Watson Analytics delivers a powerful self-service tool for understanding complex pools of data,” Sall said.

IBM partner Avnet said it will use the Watson Analytics tool in conjunction with its business analytics service, VenueEdge. John Lucas, director of solutions delivery for Avnet, said tools such as Watson Analytics are helping the Tempe, Ariz.-based distributor win business.

Avnet said it is already leveraging Watson Analytics as an IBM launch partner. “We are using it to help Major League baseball teams stitch together connections between ticket sales, concessions, CRM data, and merchandise sales so our customers can get a 365-degree view of their business with deep insight into who their customers are and what services they buy and when,” Lucas said. “We can help a baseball franchise figure out how much more merchandise they can expect to sell if one of their players hits eight more home runs.”

FYI Solutions is a leader in Business Analytics.  To find out more about Watson for your business, contact us.


Nov 07, 2014

Cognos Report Studio: Performance in Multidimensional Reporting

Author: Jason Apwah

Reports based on cubes can perform really well or really, really poorly. If the performance is somewhere in between, then chances are it has failed because it has neither fully met performance goals nor fully met reporting requirements. It is important to understand that although the report is the end product of a lot of previous labor and good management of the entire project in its early stages is invaluable, the report developer has the last opportunity to influence the outcome.

Ask Them Why

When it comes to reporting requirements, many clients that are new to real business analytics do not know what they want, so they understandably ask for more than they need. However, the purpose of business analytics is to provide reliable, clear, and timely information that allow users to make data-driven decisions. This is why ‘extract’ type reports are usually only useful as a source for some other application to be further processed. Ask them repeatedly ‘why?’ they want what they say they want, while pushing for best practices. By virtue of condensing information and making it possible to gain insight, in a lot of cases you have limited the data set. The smaller the number of elements on each crosstab edge, the better performance will be. Allow for the user to dynamically select the context of the report at run time. This way, the data available remains constant, but the number of elements on each crosstab edge at any given time is limited.

Stay MDX Friendly

A lot depends on the functions you use in the report. For example, although detail filters may work, they will hurt performance. Detail filters are applied to the lowest level element intersections, therefore even result sets that are not necessarily in the report still need to be created. Filter functions and slicers should be used instead. Filters and slicers in most cases only affect the rows or columns of a crosstab, where detail filters affect both.

Keep from Automatic Zero-Suppression

Zero suppression is always done locally. The built-in zero suppression is performed on the MDX result set. Unless the cube is very small, performance will suffer. Instead, create a calculated measure and then use the measure in a filter function to suppress zeros. For example, let’s say we want to suppress all products where the units and revenue are both zero for 2014. We create [Calc], a calculated measure as the calculation: if(tuple([Units], [2014]) = 0 and tuple([Revenue], [2014]) = 0) then (0) else (1). In the calculated [Filtered Products] set, we put the calculation: filter([Product].[Product].[Product], [Calc] <>0). If [Units] and [Revenue] are on the crosstab columns, and [Filtered Products] are on the crosstab rows, Products where the Units and Revenue are zero for 2014 will be suppressed while sparing local processing for zero suppression.

Minimize Nested Dimensions

Nested dimensions require Cognos to use the cross join function, which can significantly reduce performance especially in a sparse cube. Cross joins combine two sets of dimension members into one set. Essentially, the members of each set are combined by matching every member from the first set with every member of the second set. In a sparse cube (which most cubes are), a large number of combinations would yield a zero or null tuple which is inefficient. When possible, use tuples instead of nesting to reduce the sparsity of the measures in scope.

FYI Solutions received the prestigious North America Best Industry Solution Award from Cognos. This honor recognizes Cognos partners who excel at providing outstanding solutions tailored for the specific business challenges of their clients.  For more information on Cognos, feel free to contact us.

Oct 28, 2014

Five Tips to Start Your New Job Off Right

Author: Dan Scovill

We all want to make sure that when we find a good job that we keep it, especially in today’s job market.  We have all heard it before, but we know “you never get a second chance to make a first impression.”  Here are a few straight-forward tips to keep in mind at your new job.

Learn as much as you can about your new employer – This process should have already started before your first interview, as you want to understand the business of your new potential employer, but it should not stop there.  Most businesses nowadays are looking for employees with an entrepreneurial spirit who can bring new ideas, new potential revenue streams, ways to increase efficiency, process improvement, etc.  Just because it may not be in your job description, employers are always looking for ideas to improve business.  The better you understand the ins and outs of your employer, the more easily these ideas may come to you.

Minimize surprises upfront – If you have a vacation planned, please let your potential employer know during the interview process.  If you are right for the job then it shouldn’t affect their hiring decision, but taking a week off or taking a long weekend in your first month or so could very well leave management with a bad taste in their mouths.  Your manager may also have an on-boarding/training schedule for you and time off could interfere.  They may not even let you take the vacation as you likely haven’t accrued enough vacation time that soon.

Dress to impress (or at least appropriately) – Though it may seem unnecessary or superficial, your attire is something you must take into account in the work place.  If you are working with a recruiter or someone in Human Resources who has been helping you through the interview process, ask them about the proper attire.  If you don’t have that information available, dress in a business suit for your first day and then you can judge how to go from there, once you get a feel for the work environment.  It may sound silly, but make sure clothes are well-ironed and/or dry cleaned.  Though a wrinkled outfit seems harmless, it projects an “I don’t care” attitude.  Please also make sure to have the right dress socks — nothing stands out more (in a bad way) than white socks with a business suit.

Be (and project the attitude of) a committed team member – Employers more and more are allowing employees flexible work hours and time working from home.  This is a good thing but it is not something you want to press for in the first few months (unless it was explicitly discussed during the interview process).  Take the time to establish yourself (at least 6 months to a year) before thinking about asking for time working from home or flexible hours.  Management wants to see a solid track record of success in order to build trust with you as an employee.  Remember it also depends on your job; not all jobs allow for offsite work.

Use your past experience to bring fresh ideas to your new place of work – A fresh perspective can be helpful to a stagnant environment.  However, be careful not to make everything about “well, this is how we used to do it.” That can get old quickly and you should still respect the procedures in place. However, a well-placed solution from your past experience can be very helpful.

FYI Solutions has been a leader in specialized staffing and solutions for 30 years. Please check our website at to see some great opportunities available through us!  

Oct 21, 2014

Seven (7) overlooked benefits of temp IT staffing vs. direct permanent hiring.

Author:  Ralph Cetrulo

Here are seven commonly overlooked benefits of IT Staffing versus direct permanent hiring of IT talent.

  1. Access Top Talent – IT staffing firms attract, employ and recruit highly skill IT talent. Often, the talent is highly specialized and for that reason, many IT consultants are not even interested in full-time employment.  Top talent can also come in the form of retired baby boomers that possess subject matter expertise and can return as consultants.


  1. Try Before You Buy – Hiring the right employee is a challenging process. Hiring the wrong employee is expensive, costly to your work environment, and time consuming. In some cases, converting a consultant or temp allows you to thoroughly assess the talent skill level and cultural fit before making an employee. The working interview is best way to hire!


  1. Flexibility – Firms that don’t use consultants are overstaffed. Consultants offer much faster hiring than direct hires, the ability to hire for specific projects or specific skills, and the option to terminate the contract is simple and risk free.


  1. Eliminate Payroll Tax – Some more often overlooked benefits include eliminating federal and state payroll filing and no year-end w-2 administration costs. Guarantee IRS compliance by converting 1099 independent consultants into staffing company consultants.


  1. No Unemployment and Workers Comp Exposure – With consultants, there are no obligations to pay unemployment insurance premiums or worker’s comp administration expenses for the consultants. This is all handled by the staffing firm.


  1. No Benefit Administrations – This is because the consultant’s employee benefits are managed by staffing firm.  Also, contract employees are not eligible for the same benefits as employees.


  1. Eliminate HR Issues – With consultant hiring, there is no formal (singular) corporate hiring process, which reduces the need for HR support. And most importantly, all issues are handled by the staffing firm immediately.


The Bottom Line is Reduced Cost!!!

  • Consultants are paid out of an Operating budget. That means it’s an expense that can be written off.
  • No administrative cost for hiring and paperwork
  • No payroll expense
  • No workers comp or employee benefits administration
  • No added expense of paid time off, Vacation, or Holiday pay
  • Less supervisory cost
  • No training costs

The next time you have a staffing need, consider the pluses that come with a temporary IT staffing arrangement.  FYI Solutions has been a leader in Information Technology staffing for over 30 years.  Please contact us for more information on how we can help you fulfill your staffing requirements.

Oct 15, 2014

Top Facebook Tips & Tricks you should know


Author: Patty Ploykrachang

As Facebook becomes as much a corporate tool as a location for connecting with friends and family, it is good to know things that can be done to get to the information you need to understand your customers and prospects without sacrificing your privacy, or the privacy of others. Here are a few hints that we have found useful at FYI Solutions.

Facebook makes its money from knowing who their users are – and what they are interested in.  They do this by finding out what the user is searching for, both on and off their website. The company basically tracks almost everything that you do.  To prevent this, you can use a browser extension such as “DISCONNECT” or “GHOSTERY” . They both are available for Firefox, Chrome, Opera, and Safari. Once you install it in your browser, it blocks Facebook from retrieving information about what websites you’re visiting. “DISCONNECT” is available for your mobile device as well.

Facebook will never notify you about messages from people who are not in your network.  You may be missing out on hidden messages from others! If you receive a message from someone and you DO NOT have mutual friends with this person, your message automatically gets filtered into the “Other” message folder (located right next to “Inbox”). So next time you are checking your messages, keep in mind the “other” folder. Please see the picture below.

message other

Stop auto play videos on your mobile and desktop. While scrolling down the news feed, it can get irritating having videos play by themselves.  Click here to find out how to disable auto video play on your mobile. From your desktop, select the down arrow in the top right “Settings”, then “Video” from the bottom left side menu and then “Auto Play Off.” Please see the picture below.

fb video play

Most Recent vs. Top Stories.  You can change what you see on your news feed. By selecting the down arrow next to “News Feed” on top left of your news feed, you have the option to view either “Most Recent” or “Top Stories.”

Your customized list isn’t private.  Be aware whenever you make a custom list and post to that list — you will can expose whoever is added to that list. Please see the picture below.

fb pic

Log out remotely.  Don’t worry if you forgot to log out of someone else’s computer.  You can sign onto facebook and log out remotely by clicking edit next to “Where You’re Logged In.” This will show you where your account is logged in and you can choose “end activity” to log out right away. OR on the top right choose “Settings” then “Security” tab.  You can choose to be notified via SMS or email if a new computer or mobile device logs into your account.

FYI Solutions has been a leader in business analytics for over 30 years. Tools like Facebook and Twitter have opened up a new area within Analytics:  Social Media Analytics. For more information about Social Media Analytics, and how to use social data to understand your customer and improve ROI, contact us.

Oct 01, 2014

How You Can Identify Consulting Firms That Value You Most

Author: Gregg Ruoti

By last count, there are literally hundreds of IT search/consulting firms in the Greater NY metropolitan area.  Adding this number to out of state and sometimes offshore firms that service the IT needs of this area, this makes for an insufferable number of firms competing for the right to employ your services.  These firms range from multi-billion dollar publicly traded firms, to individuals working out of their homes. The dilemma is how to best narrow down this huge list to a manageable amount of quality firms that are the most likely to result in you finding your next project.

Obviously a common place to start is using your network of trusted associates to recommend search/consulting firms for you.  They will have had first-hand experience with these firms and should be able to tell you their preferred point-of–contact at these firms.

But your next step involves reaching out to companies that neither you nor your associates have experience with.  This is where you need to do some research to identify companies and individuals at these firms who truly have credibility in your technology domain.  If you want to reach businesses that truly specialize in your field, you need to seek out the firms that have dedicated the time, money and effort to establish themselves as subject matter experts dedicated to providing solutions to their clients in specific areas of technology.

Using an example of a Business Intelligence Architect with Cognos, this individual should be seeking out search/consulting firms following this formula:

1. Look for companies that have either a Business Intelligence or Business Analytics Practice.

Also browse to see if this firm Attends or Sponsors Conferences or provides training in Business Intelligence or Business Analytics

This translates that this business has dedicated substantial financial resources to this vertical and have full time employees dedicated to finding and creating solutions to their clients’ needs in this arena.

2. Seek out search/consulting firms that are Certified IBM Business Analytics Business Partners.

This Certification signifies this firm is very serious about providing Cognos based solutions to their clients with people just like you.

3. Go to LinkedIn and review their employees to see if any of them carry Certifications such as (but not limited to):

  • IBM Certified Solution Expert
  • IBM Certified Developer
  • IBM Certified Designer
  • SPSS
  • IBM Business Analytics Certification
  • Certified Business Intelligence Professional (CBIP)

If after reviewing some of their backgrounds you see the firm has employees or consultants with relevant certifications/qualifications, that is a strong indicator this company attracts good people and invests in their training to keep them up to date and marketable.

Clearly depending on your area of IT specialty, you would fine tune the parameters of your search for type of Practice, Partnership and Certifications.  This method will minimize the distractions of the multitude of ill-suited firms and help you identify the quality search/consulting firms that place the most value on your particular skillset and thus will better integrate you into their culture and their clients.

FYI Solutions has been a leader in specialized staffing and solutions for 30 years. Please check our website at to see some great opportunities available through us!



Sep 25, 2014

Tips for your next job interview!

Author:  Michele D’Aries

I recently read an article by Cecile Peterkin on “15 ways to win a job interview”.  These are great tips and a reminder on what we can do to win the job.  When candidates go for an interview, they should be prepared.  Research the company by going to their website and be sure to read the CEO’s message which you will find on the web site or by reading the Annual Financial report for the company.  Make sure, if you are on LinkedIn, that your profile is suitable for the position.  Know how to greet your interviewer and how to match their style. After the interview is done, ask if the interviewer has any questions to ask of you.  One question I ask my candidates to consider asking the interviewer:

Is there any reason that you think I would not be a good candidate for this position?

This is not an easy question to ask but if the interviewer has this thought it will give you a chance to present a rebuttal.  At the end of the interview, thank the interviewer for their time and let them know you are very interested in the position and you would like them to consider you to be part of the team.

I would suggest taking time to read this article “15 ways to win a Job Interview” by Cecile Peterkin . You just may pick up a few tips to beat out the competition!

FYI Solutions is a leader in specialized staffing and business analytics.  We look forward to placing on that next position! Contact us for more information.

Sep 18, 2014

Big Business is Watching!

Author: Marianela Peraza

Big business is watching and they will continue to watch more carefully in the next few years!  From what your favorite TV shows are to what your favorite candy bar is, businesses that will continue to grow successfully will need to know this.   In a new report by Markets and Markets, Social Media Analytics is expected to grow from $620.3 Million in 2014 to $2.73 Billion by 2019! It is no surprise as the benefits for businesses are so apparent that not considering the significance of social media monitoring and intelligence can cause serious damage to a business’s brand if they are not meeting their customers’ expectations.  Understanding what your customers like and what they purchase can be very powerful for the company’s marketing. Even knowing when a customer is not happy with your brand can provide very valuable information to help improve your product, customer service, etc. If leveraged well, social insights can reveal consumer opinion and trends, and can be useful to make future predictions.

When Social media analytics began, it was all about tracking the number of Fans/Followers, the number of “shares”, and website visitors. Today, those metrics have changed as better tools arose to reduce big data into a more manageable group of metrics.  Does your strategy translate into meaningful insights that guide your business tactics? If the answer is no, you are in good company. Many businesses still struggle to produce actionable insights from these metrics. In an article written on, they offer some six key reasons as to why there is a disconnect between real-time social media analytics and ROI.

1. Speed

Because social media is always “on”, 24/ 7, and literally MILLIONS of pieces of social media appear every second. WHO can keep up with all that?
So, while the massive AMOUNT of data slows down the social media analytics process, other factors account for why insights are so slow to emerge.

2. Getting social media metrics to the right people.

Often, social media is treated like the ugly stepchild within the marketing department and real-time social media analytics are either absent or ignored.

Real-time social media analytics create serious challenges for many organizations. Often, organizations are married to an old paradigm — a vestige of by-gone days when data was hard to get, taking months of data gathering and analysis. These organizations didn’t integrate data gathering into tactical and strategic decision-making because they couldn’t. Incorporating real-time analytics just isn’t possible within their existing environment.

For one thing, real-time analytics requires moving analysts closer to decision-makers and enabling decision-makers with analytic skills for ad hoc data analysis. But, that’s not what most businesses look like. Many decision-makers lack the analytics skills necessary for ad hoc analysis.

New arrival, Uber, which runs a ride-sharing program that competes with taxi companies and car services, uses real-time analytics to show how people move around a city at any given time, allowing Uber to optimize their customer service. Placing cars nearby reduces competition with local cab companies and real-time analytics provide insights necessary to do that. To do this, Uber uses real-time data to incentivize more drivers to provide services by raising the price of a ride.

Others such as Samsung and NASCAR do a great job of providing real-time social media analytics to guide decision-makers. NASCAR uses a control center to monitor chatter surrounding their events.

3. Visualization

Visualizing real-time social media analytics is another key element involved in developing insights that matter.

Face it: human beings don’t do a great job of processing long tables of numbers. Notice on NASCAR’s command center, much of the date is displayed visually.

Simply displaying values graphically helps in making the kinds of fast interpretations necessary for making decisions with real-time data, but adding more complex algorithms and using models provides deeper insights, especially when visualized.

4. Unstructured data is challenging.

Unlike the survey data firms are used to dealing with, most (IBM estimates 80%) is unstructured — meaning it consists of words rather than numbers. And, text analytics lags seriously behind numeric analysis.

While unstructured data tends to muck-up any kind of analysis, it’s especially challenging in the context of real-time analytics, because you want interpretations IN REAL TIME. Handling text in real time often means using computer-generated translations of the written word. However, no computer can effectively categorize much of what’s written in social media where “bad” might mean bad or it might mean good, depending on context, relationship, and other variables.

5. Increasing signal to noise.

Social media data is inherently noisy. Reducing noise to even detect signal is challenging — especially in real time. Sure, with enough time, new analytics tools can ferret out the few meaningful comments across various social networks, but few can handle this in real-time.

6. A “wait and see” attitude.

Again, businesses are used to a certain operational model that makes real-time social media analytics challenging. For instance, we listed to a presentation by an analyst from NPR. He showed complex A/B testing used to determine the effectiveness of headlines, even whole articles online. As a statistician, he’s concerned about achieving statistical significance in his testing before making decisions.

And that’s great if you’re talking about putting $100 million into building and marketing a product, but doesn’t make much sense in the fast-paced world of social media. Real-time analytics require real-time decisions. Period.

If it’ll take you several days to gather enough data for statistical significance, forget it. Especially if you’re only trying to determine which headline does better, by the time you have a statistically significant answer, no one cares anymore. The news trend has moved to another topic.
FYI Solutions is a leader in business analytics and has partnerships with IBM, Tableau, and Microsoft. Give us a call to understand how to incorporate social media analytics into your environment and reap the benefits of increased understanding of your clients and prospects.


Sep 10, 2014

Data munging or data wrangling, a latest process in the data world!!!

Written by: Joe Wasiuk

I think this is an interesting article from the NY Times, written by Steve Lohr. The term “data science” has been around for a longer period of time than I thought. I would imagine it is now widely known that companies have a demand for “Data Scientists”. It was interesting to read that data scientists spend from 50 percent to 80 percent of their time mired in this more mundane labor (process) of collecting and preparing unruly digital data, before it can be explored for useful insight.

Please see a few of the article highlights below and here is a link to the full article:

Technology revolutions come in measured, sometimes foot-dragging steps. The field known as “big data” offers a contemporary case study. The catchphrase stands for the modern abundance of digital data from many sources — the web, sensors, smartphones and corporate databases — that can be mined with clever software for discoveries and insights.
Yet far too much handcrafted work — what data scientists call “data wrangling,” “data munging” and “data janitor work” — is still required.

Several start-ups are trying to break through these big data bottlenecks by developing software to automate the gathering, cleaning and organizing of disparate data, which is plentiful but messy. The modern Wild West of data needs to be tamed somewhat so it can be recognized and exploited by a computer program.

Timothy Weaver, the chief information officer of Del Monte Foods, calls the predicament of data wrangling big data’s “iceberg” issue, meaning attention is focused on the result that is seen rather than all the unseen toil beneath. In the food industry, he explained, the data available today could include production volumes, location data on shipments, weather reports, retailers’ daily sales and social network comments, parsed for signals of shifts in sentiment and demand.

But if the value comes from combining different data sets, so does the headache. Data from sensors, documents, the web and conventional databases all come in different formats. Before a software algorithm can go looking for answers, the data must be cleaned up and converted into a unified form that the algorithm can understand. Data experts try to automate as many steps in the process as possible. “But practically, because of the diversity of data, you spend a lot of your time being a data janitor, before you can get to the cool, sexy things that got you into the field in the first place,” said Matt Mohebbi, a data scientist and co-founder of Iodine.

The big data challenge today fits a familiar pattern in computing. A new technology emerges and initially it is mastered by an elite few. But with time, ingenuity and investment, the tools get better, the economics improve, business practices adapt and the technology eventually gets diffused and democratized into the mainstream.

FYI Solutions is a leader in Business Analytics, and can help you get started on the “data wrangling” required for Big Data success. Contact us for more information.