The cliché holds that we are drowning in data, yet starving for knowledge. Wikipedia characterises data science as the generalizable extraction of knowledge from data, and it defines statistics as the study of the collection, organization, analysis, interpretation and presentation of data. These two seem very similar to me. What characterises the difference between data science and statistics? Is one a subset of the other?
Although including statistics, data science lies at the intersection of programming, statistical modeling, and social science (or other substantive) knowledge.
Special problems for data scientists include:
- Data collection: web scraping and online surveys
- Data manipulation: recoding messy data and extracting meaning from linguistic and social network data
- Data scale: working with extremely large data sets
- Data mining: finding patterns in large, complex data sets, with an emphasis on algorithmic techniques
- Data communication: helping turn "machine-readable" data into "human-readable" information via visualization
Thus, although incorporating many of the techniques of statistics (such as regression modeling), data science can be viewed as a more general field, with an emphasis on practical applications, communicating results to a wide audience, and using algorithmic models.
Edit: Also note that statistics has traditionally focused on inference from a sample to a population, while data scientists today typically have a full population or a sample so large that the traditional tools of statistical inference break down (e.g., small p-values are plentiful in a data of several million individuals!). Furthermore, statistics has traditionally under-emphasized algorithmic models (e.g., k-means clustering, neural nets, random forests), data visualization (e.g., Tufte is actually a Ph.D. in political science!), natural language processing (cf. computational linguists), and the difficulties of cleaning messy data.
My personal impression is that a pure (old-school) statistician would usually live in an R (or SAS) world; they are rather unlikely to be running Hadoop jobs on EMR. The scale of the data is one difference, the type of tools is another, and a third difference is close ties to various complex/irregular/rich/high volume/high velocity sources of data like IoT sensors or machine vision or web clickstreams.
Some of the very early sources on data science talk about this:
C.F. Jeff Wu, Statistics = Data Science? (7th P. C. Mahalanobis Memorial Lecture, not available online, any source?)
Visual answer by Drew Conway:
Statistics is very powerful tool in the field of data science, but as the question put it, we are drowning in data. The one aspect of datascience not included in statistics is initial data exploration. The methods used to initially inspect very large datasets, which the investigator may have no idea of, are not part of statistics literature. An example of this could be the selection of an algorithm to plot network related data; force-directed or spring, or a mixed approach. If someone is dealing with large scale community detection you need to choose a proper tool such as the Louvain algorithm or walk trap. These are not covered in conventional statistics.
As well a datascientist must be aware of computational challenges with large datasets where even addition can be a difficult task.
Similar to those applied-statistics cases that are doing factor analysis, I think that as data scientists, we are frequently engaged in making epistemological judgments. That is, what sorts of things can our data sets tell us (what inferences can be made)? And which types of correlational relationships warrant further investigation as putative causal factors.
Statistics, taken solely as a mathematical endeavor, is quite clearly deductive and a priori in its search for knowledge.
Data science can't really ever be divorced from culture and perspective, and is a meaningful pursuit within certain kinds of contexts to answer certain kinds of questions (or to ask new ones?). Data science, as I see it, has its value because of the practitioners' empirical-embededness.
With the burgeoning issues created by big data, a data scientist separates the cream from the spoiled milk (by analogy). The result is that we not only assist in answering questions, but we have a sort of rootedness in our experiences that enables us to undertake informed data-collection. We find and create meaning with the specifically chosen tools of our respective trades.