From a psychological perspective, there actually is set of criteria (a "formula") for determine whether a group is a cult in the popular, negatively perceived, mind-control manner. A pretty well-accepted list is summarized on wikipedia:

1. People are put in physically or emotionally distressing situations;
2. Their problems are reduced to one simple explanation, which is repeatedly emphasized;
3. They receive what seems to be unconditional love, acceptance, and attention from a charismatic leader or group;
4. They get a new identity based on the group;
5. They are subject to entrapment (isolation from friends, relatives and the mainstream culture) and their access to information is severely controlled.

From a psychological perspective, groups that follow the pattern above are very dangerous. The use of the word, however, does not always imply this dangerous cult-entity. So, it's not necessarily appropriate to correct someone who uses the word "cult" to describe a small following that doesn't meet these criteria.

Similar checklists have been put forth by various PhD's and psychological groups. One example:

http://www.csj.org/infoserv_cult101/checklis.htm

Another example of how to start a cult:

http://www.psychologytoday.com/blog/look-it-way/200906/how-start-cult

You'll find that a lot of small religious groups meet the first 4 criteria pretty quickly. I can't think of many well-known groups that satisfy the 5th indicator, though.

There are two senses

In the benign sense, a cult is any minority group with shared beliefs or practices that intentionally deviate from the majority. The connotation is one of foreignness and friction with a larger group.

In the malignant sense, the psychological perspective is generally applied, and there actually is set of criteria (a "formula") for determine whether a group is using dishonest (brain-washing) practices to recruit and retain members. A pretty well-accepted list of criteria is summarized on wikipedia:

1. People are put in physically or emotionally distressing situations;
2. Their problems are reduced to one simple explanation, which is repeatedly emphasized;
3. They receive what seems to be unconditional love, acceptance, and attention from a charismatic leader or group;
4. They get a new identity based on the group;
5. They are subject to entrapment (isolation from friends, relatives and the mainstream culture) and their access to information is severely controlled.

Groups that follow the pattern above are very dangerous. Variations of this checklists have been published by various PhD's and psychological groups. One example:

http://www.csj.org/infoserv_cult101/checklis.htm

And an example of how to start a cult, used to illustrate similar criteria:

http://www.psychologytoday.com/blog/look-it-way/200906/how-start-cult

Common to both senses

A "cult" is a deviation from some norm. And, the label can be accurately applied to any type of group, whether the group's focus is on art, film, history, science, religion, etc.. Thus, in both senses, it's clear that a "cult" does not accurately represent the beliefs or practices of the contextual group (the contrasted norm).

There may be "sub-senses" to each sense of the word. But, I believe the two "primary" senses, as I understand them, are sufficient in most cases. And in all cases, it's important to be clear as the speaker/author and request clarification as the listener/reader when the sense is not abundantly obvious.