The only reliable way to organise books on shelves is to not organise them at all. If you sort by ISBN, then you're stuffed if you need to insert in the middle as you have to shuffle everything along, perhaps onto other shelves, which in turn means you have to move stuff on those shelves, and so on. The best way to organise books on shelves is to just put them wherever there's space, and use a searchable database to keep track of what's where.

Anyway, sorting by ISBN is approximately the same as sorting by publisher. Which is what Foyles in London used to do. People hated it, and they only stayed in business because there were no other large bookshops nearby. Now that several others have opened, Foyles have adopted a much more sensible sorting algorithm - they mostly sort by subject, author and title. Although they do still have a seperate section for O'Reilly books.