8
$\begingroup$

One of the areas I am particularly interested in, is using systems like Lucene (ElasticSearch/Solr, etc.) to store data that is used for other computations.

Are questions about how to best index the data, structure the storage schemas, etc on topic?

For example, A question like:

I use ElasticSearch, and I have a need to index multiple versions of data that are sourced from different places. Should I add a source field to the data and store it all in one large index, then filter by the source for most queries, or should I have multiple indexes, and then, only when I have queries that require multiple sources for the data should I aggregate the results over the plural indexes?

$\endgroup$
5
$\begingroup$

If it's extremely specific to a single platform and the question is more about implementation or a technology than a general data science question, I would be inclined toward saying it is off-topic. If however it can be a general question, not tied to a very specific technological product and more about reasoning than implementation, then it could be on topic.

$\endgroup$
3
  • 3
    $\begingroup$ I think in general topics related to "tidy data" are appropriate (albeit if they are platform/language agnostic to some extent like you mention). The task of manipulating raw data into a usable form is so ubiquitous in data science that a resource for solutions to common problems in this space would be very useful. $\endgroup$ – cwharland May 14 '14 at 5:09
  • $\begingroup$ @cwharland Yes, and in those cases, the bulk of the discussion falls under general reasoning or schema design or data manipulation, and not under "How do I do X in R?" $\endgroup$ – Ansari May 14 '14 at 5:23
  • $\begingroup$ Sounds like a mostly off-topic discussion. SO and DB sites, maybe more, are a good place to discuss schemas, indexing, data retention design. $\endgroup$ – Dave Mar 3 '16 at 23:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .