You should be concerned with head seek time and transfer rate. It is true that they depend on the form factor, but they also depend on many other variables. Looking only at the physical size and ignoring those variables would be wrong.
With this in mind, let's compare the most recent versions of some widely used disks: Seagate Cheetah 15K.7 and Savvio 10K.3.
Bottom line: 2.5" disks are comparable to 3.5" disks for random operations, and usually you can compensate or even win with 2.5" due to the larger number of spindles in the same physical volume (disk array or server). However, if you need to pump a lot of sequential data, 3.5" disk is still the king.
Prior to me starting my current job our ERP package had been moved to a new server by a predecessor, who had intended to decommission and scrap the old machine. The performance was appalling and both the vendor and that predecessor thought it was due to a configuration error, although they never found the cause. After taking over the job and examining all the factors it was clear to me that the 2.5 inch 10K SAS drives on the new server simply couldn't keep up. I therefore moved the ERP package back to the old server, which has 3.5 inch 15k SCSI drives. Performance is now back to where it should be. In both cases the drives are arranged as a 3 drive RAID 5 array.
On the subject of head movement actuator others have mentioned, they have failed to take into account that the actuator in 3.5 inch drives is generally considerably more powerful than that in a 2.5 inch drive. The consequence is that although it needs to travel a little further (about one tenth on an inch or less) it can do so more rapidly. This negates the perceived advantage of the smaller drive.
Every drive model is unique, and there are a ton of other factors that come into play beyond form factor and rotational speed, but in general:
More importantly, will the difference matter. If this is for a disk mirror set, you will not be able to notice the difference except in benchmarking tools, since controller cache, how the OS controls I/O and application I/O will all factor in to make the difference meaningless. However if you are loading up a SAN with 150 of these, you will notice a difference (as Chopper mentioned above) - you should still determine if that difference is worth the price, and many times that price difference could be the cost of more drives- which would make the difference irrelevant. See this storage advisors link for some published example details.