# Are questions about structuring data for storage in databases/repositories/xxxx systems on topic?

One of the areas I am particularly interested in, is using systems like Lucene (ElasticSearch/Solr, etc.) to store data that is used for other computations.

Are questions about how to best index the data, structure the storage schemas, etc on topic?

For example, A question like:

I use ElasticSearch, and I have a need to index multiple versions of data that are sourced from different places. Should I add a source field to the data and store it all in one large index, then filter by the source for most queries, or should I have multiple indexes, and then, only when I have queries that require multiple sources for the data should I aggregate the results over the plural indexes?

## 1 Answer

If it's extremely specific to a single platform and the question is more about implementation or a technology than a general data science question, I would be inclined toward saying it is off-topic. If however it can be a general question, not tied to a very specific technological product and more about reasoning than implementation, then it could be on topic.

• I think in general topics related to "tidy data" are appropriate (albeit if they are platform/language agnostic to some extent like you mention). The task of manipulating raw data into a usable form is so ubiquitous in data science that a resource for solutions to common problems in this space would be very useful. May 14, 2014 at 5:09
• @cwharland Yes, and in those cases, the bulk of the discussion falls under general reasoning or schema design or data manipulation, and not under "How do I do X in R?" May 14, 2014 at 5:23
• Sounds like a mostly off-topic discussion. SO and DB sites, maybe more, are a good place to discuss schemas, indexing, data retention design.
– Dave
Mar 3, 2016 at 23:56