Look for newer versions that are in high demand. Freshman calculus books sell for a high prices - anything that's required course material for incoming students. You'll find big waves of demand at the beginning of each semester for those titles.
Higher level math books also sell for lots of money due to the subject matter. Even though there is less demand, these books are expensive to produce and are sold for a premium.
Other Subjects
Math textbooks are found in a variety of fields - finance, computer science, statistics, and more. These books are generally very valuable and command a high sale price. If you're buying and selling math books for a profit, there are tons of niches where you can sell for high prices.