A useful model is one that is simple enough to be analyzed easily, while nevertheless being similar enough to reality that this analysis can be used as a basis for predictions about the actual world. Unfortunately, it can be difficult to judge whether a given model is in fact similar enough. Furthermore, even if some predictions based on a model come true, this does not necessarily mean that next prediction based on the model will also come true.
A classic illustration of the importance of using appropriate models, as well as the difficulty of noticing when a model is inappropriate, is the 2007 financial crisis. In the years leading up to the crisis, many financial actors made investment decisions on the basis of models that assumed economic stability. Once this simplifying assumption ceased to hold, it became clear that their models had not sufficiently matched reality, and that the outcome of their decisions would be disastrous.
One strategy for dealing with uncertainty about the appropriateness of models is to construct and weight the predictions of multiple diverse models, rather than relying on a single one. However, in cases of radical uncertainty, not even this method may be enough. It may be that we think that there is a chance that none of the models that we have been able to generate is appropriate, and that we need to factor in what could happen if that were the case. Obviously, it is very hard to say something about such an uncertain case, but it may be possible to say some things. For instance, in their paper “Probing the Improbable,” Toby Ord, Rafaela Hillerbrand, and Anders Sandberg argue that in cases where our models about certain low probability, high-risk events - such as existential risks - are wrong, the chance of disaster may be substantially higher than if the models are right.
Frigg, Roman & Stephan Hartmann. 2012. Models in science. In Edward Zalta (ed.), Stanford encyclopedia of philosophy.
Hillerbrand, Rafaela, Toby Ord & Anders Sandberg. 2016. Probing the improbable: methodological challenges for risks with low probabilities and high stakes. Journal of risk research, 13(2): 191-205.
Karnofsky, Holden. 2016. Sequence thinking vs. cluster thinking.