The intelligence of machines and the branch of computer science which aims to create it

Artificial Intelligence Journal

Subscribe to Artificial Intelligence Journal: eMailAlertsEmail Alerts newslettersWeekly Newsletters
Get Artificial Intelligence Journal: homepageHomepage mobileMobile rssRSS facebookFacebook twitterTwitter linkedinLinkedIn

Artificial Intelligence Authors: Corey Roth, Pat Romanski, Liz McMillan, Yeshim Deniz, Kevin Benedict

Related Topics: Artificial Intelligence Journal, CIO, SEO Journal, Cloud Data Analytics, CIO/CTO Update, Big Data on Ulitzer

Blog Feed Post

Yes, you need more than just R for Big Data Analytics

Douglas Merrill, former CIO/VP of Engineering at Google, writes in Forbes about using the R language for data analysis: Most folks with math-oriented graduate degrees will have written something in R, a non-commercial option for your big data analysis.  So, great graduates from great graduate schools know great tools. His post is titled 'R Is Not Enough For "Big Data"', and you might be surprised to learn that I agree that title, although for a different reason. Douglas's point — and it's a valid one — is that simply pumping data through any software tool, without an understanding of the problem you're trying to solve and how statistical models apply to it, can lead to getting the wrong answers to the wrong questions: If you ask the wrong question, you will be able to find statistics that give answers that are simply wrong (or, at best, misleading). On net, having a degree in math, economics, AI, etc., isn’t enough. Tool expertise isn’t enough.  You need experience in solving real world problems, because there are a lot of importat limitations to the statistics that you learned in school.  Big data isn’t about bits, it’s about talent. This is a great illustration of why the data science process is a valuable one for extracting information from Big Data, because it combines tool expertise with statistical expertise and the domain expertise required to understand the problem and the data applicable to it. He's right that you need data science talent and software to solve problems with Big Data ... and having software like R that supports the exploratory nature of the data science process is also critical. But I also agree with the title for a different, technical reason: the R software is just one piece of software ecosystem — an analytics stack, if you will — of tools used to analyze Big Data. For one thing R isn't a data store in its own right: you also need a data layer where R can access structured and unstructured data for analysis. (For example, see how you can use R to extract data from Hadoop in the slides from today's webinar by Antonio Piccolboni.) At the analytics layer, you need statistical algorithms that work with Big Data, like those in Revolution R Enterprise. And at the presentation layer, you need the ability to embed the results of the analysis in reports, BI tools, or data apps. So yes, Douglas is right: you need more than just R for Big Data. You also need a data layer, an analytics layer, and a presentation layer (all of which supports Big Data) ... and you need Data Science skills to make sure you're asking the right questions and getting appropriate answers. Forbes: R Is Not Enough For "Big Data"

Read the original blog entry...

More Stories By David Smith

David Smith is Vice President of Marketing and Community at Revolution Analytics. He has a long history with the R and statistics communities. After graduating with a degree in Statistics from the University of Adelaide, South Australia, he spent four years researching statistical methodology at Lancaster University in the United Kingdom, where he also developed a number of packages for the S-PLUS statistical modeling environment. He continued his association with S-PLUS at Insightful (now TIBCO Spotfire) overseeing the product management of S-PLUS and other statistical and data mining products.<

David smith is the co-author (with Bill Venables) of the popular tutorial manual, An Introduction to R, and one of the originating developers of the ESS: Emacs Speaks Statistics project. Today, he leads marketing for REvolution R, supports R communities worldwide, and is responsible for the Revolutions blog. Prior to joining Revolution Analytics, he served as vice president of product management at Zynchros, Inc. Follow him on twitter at @RevoDavid