What to do about "Are my model ideas for this problem good?" or "What is best model for this problem?" questions

Question

There is a common question type that gets down voted or closed here. Rightly so in my opinion. However, I think the steady stream of similar questions needs somewhere we can point them.

Typical scenario is:

• A beginner to machine learning has a data set, imagined or real, and a problem to solve.

• As a beginner, they are confused about selecting the "best" modelling approach. There are loads of choices to make after all, and most involve learning about the model class and an API for it in some framework.

• The question is presented better than "Here's some data, what do I do?" or "Give me the model?" questions in that the poster is aware of some options in front of them and has a real problem that they are trying to solve themselves. However, despite trying, they have still managed to post a question that is badly framed or too broad.

• The question is at a fork in the road - either the poster could share more concrete details about what they are doing, or they could simply be encouraged to try and test their own ideas. Both could be acceptable outcomes, it is not 100% clear whether the problem needs more guidance than "What I'd do is try all 3 of your ideas and test them - so why don't you do the same?".

The reality is that even with a lot of experience, second-guessing the best model for a problem is not how things work. Instead an expert will typically explore the data and problem, select a metric to measure success, then try a range of possible models and feature engineering, testing each.

I think the asker's desire is to get some help through the model selection maze. However, they tend to forget how unique each data problem often is, and/or seem to think that more experienced ML experts will somehow know what to do, sometimes only with the problem domain, a hand-waving description of the data and a rough goal.

I'd like either this question, or even a canonical duplicate, to serve as a pointer to how the posters could improve their questions so that they can be answered and/or to be a frame challenge - i.e. the experts don't just read literature looking for "best solution to problem X", in fact they mostly just try and test stuff; the best thing is to learn the skills that enable the asker to do the same.

---

Some recent example questions:

• I want to maximise a sum by selecting best examples based on predictions, how to model?
• Anomoly detection method selection
• Are deep learning algorithms good at predicting the outcomes of sports matches?

Answer 1

I think every "model selection" question must provide the following information:

Data:

• What's the size of the Data?
• How is the data structured?
• Can you provide a sample? (Even if it is a fake one)

Whats the nature of the problem?

• What's more critical, avoid false positives or false negatives?
• What's more critical, avoid over estimating or underestimating?
• What's the minimum efficiency your system must have so it become viable?

What are your results so far?

• What models have you training and with what parameters?
• Why aren't you satisfied with current results?

Also, it must provide:

• Training, Validation and Test curve results
• Details about validation procedure (k-fold, leave-of-out, simple split)

Data Scientists:

People asking here are suppose to be data scientist, they should at least have the ability to explain their problem and approach to it in order to get help, any question about a particular problem should be long and detailed. Broad questions should de downvoted and closed at first sight.

Beginners:

If they are not stuck with a particular concept, they should be directed to basic tutorials. If they seem to simply want to skip the learning process I would just downvote and close the question as too broad or in need of clarity or details.

Answer 2

When I first read your post, I agreed with it; but the more I think about it, the more I don't. In your post, you say:

Instead an expert will typically explore the data and problem, select a metric to measure success, then try a range of possible models and feature engineering, testing each.

This is certainly true. But can't more be said on each of these points? What does explore mean (and do different problems have data exploration)? What are typical metrics of success is each problem? What kind of features might be good to engineer for the problem at hand? With models, are there models to try as a baseline first?

I think there is a lot of process to unpack here. Certainly there are a lot of questions posed which have too few details; your complaint, however is about all such questions, not just vague the ones. If you remove all such questions, it looks to me like you're a step closer to making the questions here a subset of the questions of Cross-Validated. So in fact it seems that these questions are the kind that distinguish the board from Cross-Validated.

Answer 3

What about having a page with examples of tasks and their categories to help people to categorize their tasks, and hopefully help them to model their problem?

Today I encountered a question, which of course is being flagged as too broad, but the problem is that the person didn't even know the term entity extraction.

Often I see this problem, where the person ask, "How do I find the trend?" or "How do I predict?" when often it is a "simple"classification, regression or clustering problem. However, the person cannot classify their problem (either because they don't know the terms or because they don't understand their own task).

And if they still cannot classify their problem, there is a list of information they need to provide to be pointed to the right direction. A list like Paul wrote in the comments of the other answer:

(1) goal of a predictive model;

(2) description of the data, including (a) number of rows, (b) type of target, (c) type of possible inputs already in the model, and

(3) description of what they have tried so far