You should make different recognizers for calls that are significantly different, especially with limited training data. With a small data set and significant variation in call types, you are likely to create a recognizer that will only look for a very close match to each variation; the new recording is unlikely to fit within any one of the tight definitions and thus will fail to match.