karmakiro

Hi Steve,
I think to some extent you'll come to discover that there isn't an universal "mormon view" on how false prophets are determined. This is perhaps because our time spent worshiping is dedicated to learning about that which is uplifting and helps us to more closely emulate our Saviour Jesus Christ. Thus I've never heard and never expect to hear an official Mormon teaching specifically targeting what is wrong with another's belief system, that being said there are teachings on how to determine that which is true and to avoid deception.
Now then I'll first answer with why I personally reject the prophet of Islam starting with a scripture I've seen used recently (I believe in the Angels Thread) 
Revelation 19:10
10 And I fell at his feet to worship him. And he said unto me, See thou do it not: I am thy fellowservant, and of thy brethren that have the testimony of Jesus: worship God: for the testimony of Jesus is the spirit of prophecy.
To start with to declare that Christ is not deified seems to automatically disqualify him as others have mentioned.
Further the teachings of Islam simply don't reconcile with Biblical teachings. I know many assert that the same can be said of Mormonism, but I have to confess I don't understand that position - having been a student of the scriptures I cannot find any Mormon teaching that is contrary to Biblical teaching. In fact I find it more in line with Biblical teaching than any other religion's teaching I have ever studied... back to the OP now.
In the Doctrine and Covenants section 9 verse 8 one reads,
But, behold, I say unto you, that you must study it out in your mind; then you must ask me if it be right, and if it is right I will cause that your bosom shall burn within you; therefore, you shall feel that it is right.
This also goes along with and must pass the test of "by their fruits, ye shall know them." The fruits of the spirit are "love, joy, peace, longsuffering, gentleness, goodness, faith, Meekness, temperance" and would logically be (at least) expected to be among the fruits of true prophet. Now I'm not saying that the founder of Islam taught it to be so (I'm not extremely well versed with his teachings), but I find the idea of violently seeking to oppress and dominate the world with the "religion of peace" just doesn't live up to the fruits by which a true prophet is known.
So let me explain these two points in the context of how I use the above teaching to help me discern between truth and error. First I study it out in my mind and compare the teachings to those found in sources (such as the Bible) that I have come to believe to be true. If the idea that I am studying is contrary to what I believe to be true than I don't tend to concern myself with it further. If however I find no reason to reject it through study than I take that finding to God in prayer for further insight. If the item in question leaves me feeling the spirit as I've come to recognize it, then I believe it to be true and feel the need to act on it accordingly. If I feel no particular feeling when praying on it I also tend to put it on the shelf and figure that it is either not true or not important for me at that time.
 
So in the case of Mohammed, I haven't felt the need to even pray about it because of the apparent contradictions to revealed truth.
 
I hope this helps you understand how some Latter-day Saints work things out.