John Stackhouse, Sangwoo Youtung Chee Professor of Theology and Culture at Regent College, offers the following advice for students seeking reference letters to get into PhD programs (see the full post here).
You, the student, want the ideal combination of (1) the highest
possible praise (2) in the most detail, (3) especially about the most
advanced (= independent & extended) work (e.g., a thesis or a
comprehensive examination or a large research paper), from (4) the most
prestigious ((a) in terms of the field or (b) in general) professor with
(5) the best connections to the school to which you are applying.
I would rank them then in this descending order of importance:
(1) is non-negotiable. If you didn’t get an “A” from him or her, don’t ask for a reference.
(5) is next most important, since everyone else’s letters will be strong, but only some will have an “inside track.”
(4a) is next, then (4b).
While I’m at it, if there is anything odd (or unimpressive) in your
record (e.g., a bad course or semester) to which a reference can speak
on your behalf, remind him or her to do so and provide all the
information necessary to help him or her do so. You can offer the excuse
yourself in your cover letter, but such excuses will be more powerful,
of course, if rendered by someone else.