The cliché holds that we are drowning in data, yet starving for knowledge. Wikipedia characterises data science as the generalizable extraction of knowledge from data, and it defines statistics as the study of the collection, organization, analysis, interpretation and presentation of data. These two seem very similar to me. What characterises the difference between data science and statistics? Is one a subset of the other?
4
-
$\begingroup$ This is also important for meta (not sure where it belongs). $\endgroup$– SklivvzCommented May 13, 2014 at 23:51
-
$\begingroup$ I also was thinking this probably belonged on meta. $\endgroup$– Ben CollinsCommented May 14, 2014 at 15:52
-
2$\begingroup$ I wasn't sure whether to post in on the main site or on meta, but decided to post it on main because the aim of the question is not to determine what is on-topic on Data Science SE, but to determine what Data Science is generally. $\endgroup$– gerritCommented May 14, 2014 at 15:55
-
2$\begingroup$ It doesn't belong anywhere, being as the answers will just be either too long or opinion-based (unless there's a scientific method to discriminate between DS and stats). $\endgroup$– SpacedmanCommented Jun 12, 2014 at 13:29
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
8
Although including statistics, data science lies at the intersection of programming, statistical modeling, and social science (or other substantive) knowledge.
Special problems for data scientists include:
- Data collection: web scraping and online surveys
- Data manipulation: recoding messy data and extracting meaning from linguistic and social network data
- Data scale: working with extremely large data sets
- Data mining: finding patterns in large, complex data sets, with an emphasis on algorithmic techniques
- Data communication: helping turn "machine-readable" data into "human-readable" information via visualization
Thus, although incorporating many of the techniques of statistics (such as regression modeling), data science can be viewed as a more general field, with an emphasis on practical applications, communicating results to a wide audience, and using algorithmic models.
Edit: Also note that statistics has traditionally focused on inference from a sample to a population, while data scientists today typically have a full population or a sample so large that the traditional tools of statistical inference break down (e.g., small p-values are plentiful in a data of several million individuals!). Furthermore, statistics has traditionally under-emphasized algorithmic models (e.g., k-means clustering, neural nets, random forests), data visualization (e.g., Tufte is actually a Ph.D. in political science!), natural language processing (cf. computational linguists), and the difficulties of cleaning messy data.
-
2$\begingroup$ I think the last paragraph really hits it. Stats is a tool in the toolbox of a data science while data science encompasses a broader range of tasks, skills, and scenarios. $\endgroup$ Commented May 14, 2014 at 5:05
-
-
$\begingroup$ Fortunately I've heard some start-ups distinguishing between "data engineers" and "data scientists." This is really needed, I think, so that data engineers would focus more data storage and relational databases, while data scientists would work on translating these data to information. $\endgroup$– statsRusCommented May 14, 2014 at 17:20
-
$\begingroup$ Very thoughtful categorization of data science special problems! $\endgroup$ Commented May 21, 2014 at 8:03
-
3$\begingroup$ Regarding your edit, I have read similar things before but it seems based at least in part on a misconception of what (modern) statistics are about (possibly fueled by the way statistics are taught in sociology or psychology departments). Statistics are by no means limited to p-values! And Tufte might be better known to the CS crowd but Tukey or Cleveland are card-carrying statisticians and have worked on visualization for decades. $\endgroup$– GalaCommented Jun 12, 2014 at 16:45
-
$\begingroup$ Regarding Data manipulation: I think it is also a part of practical statistics. We had rather good "practical statistics" course after the pure mathematics one and I really liked it because it confronted you with problems like merging, cleaning etc. People who had only the mathematic background are REALLY struggling in statistics jobs because cleaning is needed almost 10 times out of ten. $\endgroup$ Commented Jun 13, 2014 at 7:32
-
$\begingroup$ To this fine list I'll add two more components: 1) Continuous learning, discovery, and validation (including experimentation); and 2) Support effective decision-making and actions (often automated action in business processes). E.g. choosing the "best" data analysis method depends critically on how it works in learning/experiment settings and also in production decision/action settings. $\endgroup$ Commented Jun 20, 2014 at 18:36
-
1$\begingroup$ NEURAL NETS AND RANDOM FORESTS ARE NOT ALGORITHMIC METHODS! THEY ARE STATISTICAL CLASSIFIERS! random forests are part of the greater family of ensemble methods $\endgroup$– VassCommented Mar 14, 2016 at 11:31
$\begingroup$
$\endgroup$
2
My personal impression is that a pure (old-school) statistician would usually live in an R (or SAS) world; they are rather unlikely to be running Hadoop jobs on EMR. The scale of the data is one difference, the type of tools is another, and a third difference is close ties to various complex/irregular/rich/high volume/high velocity sources of data like IoT sensors or machine vision or web clickstreams.
Some of the very early sources on data science talk about this:
William S. Cleveland, Data Science: An Action Plan for Expanding the Technical Areas of the Field of Statistics
C.F. Jeff Wu, Statistics = Data Science? (7th P. C. Mahalanobis Memorial Lecture, not available online, any source?)
Visual answer by Drew Conway:
-
1$\begingroup$ The author of The Data Science Venn Diagram is Drew Conway. Her is a link to his blog post describing the history of its creation. drewconway.com/zia/2013/3/26/the-data-science-venn-diagram $\endgroup$ Commented Jun 27, 2014 at 10:57
-
1$\begingroup$ Thanks! I was not aware of the original source. Added :) $\endgroup$– Alex ICommented Jun 27, 2014 at 11:39
$\begingroup$
$\endgroup$
2
Statistics is very powerful tool in the field of data science, but as the question put it, we are drowning in data. The one aspect of datascience not included in statistics is initial data exploration. The methods used to initially inspect very large datasets, which the investigator may have no idea of, are not part of statistics literature. An example of this could be the selection of an algorithm to plot network related data; force-directed or spring, or a mixed approach. If someone is dealing with large scale community detection you need to choose a proper tool such as the Louvain algorithm or walk trap. These are not covered in conventional statistics.
As well a datascientist must be aware of computational challenges with large datasets where even addition can be a difficult task.
-
$\begingroup$ Statistics is a vital part of Data Science, but not all of Data Science is part of statistics, so judging statistics for not doing data science is missing the point; Data Science is a relative new discipline that draws from multiple existing disciplines; the intersections between Statistics, Computer Science, domain knowledge and EDA should reside squarely in the definition of Data Science. What does Statistics add to the discipline? The importance of asking the right question, whether it comes from the researchers interest or from exploitation of big data. $\endgroup$ Commented Sep 21, 2017 at 19:47
-
$\begingroup$ "The one aspect of datascience not included in statistics is initial data exploration". This is plainly wrong -e.g. I wonder in what kind of circumstances someone would blindly try to apply some statistical method, without at the very least checking if there is some problem with the data, missing data, etc. That person would be a terrible statistician or analyst. That's for initial data analysis. As for exploratory data analysis, reading the very first paragraph of the Wikipedia article shows it's directly related to statistics. $\endgroup$– J-J-JCommented May 14 at 14:09
$\begingroup$
$\endgroup$
1
Similar to those applied-statistics cases that are doing factor analysis, I think that as data scientists, we are frequently engaged in making epistemological judgments. That is, what sorts of things can our data sets tell us (what inferences can be made)? And which types of correlational relationships warrant further investigation as putative causal factors.
Statistics, taken solely as a mathematical endeavor, is quite clearly deductive and a priori in its search for knowledge.
Data science can't really ever be divorced from culture and perspective, and is a meaningful pursuit within certain kinds of contexts to answer certain kinds of questions (or to ask new ones?). Data science, as I see it, has its value because of the practitioners' empirical-embededness.
With the burgeoning issues created by big data, a data scientist separates the cream from the spoiled milk (by analogy). The result is that we not only assist in answering questions, but we have a sort of rootedness in our experiences that enables us to undertake informed data-collection. We find and create meaning with the specifically chosen tools of our respective trades.
-
$\begingroup$ What do you mean by Statistics (...) is a priori in its search for knowledge? I'm aware of the role of a priori knowledge in some statistical techniques (in particular in Bayesian interpretations), but isn't some statistics without any a priori? $\endgroup$– gerritCommented May 14, 2014 at 14:51