I initially did it this way myself.  I discovered, though, that what tended to happen was that the kernel didn't know when the flash card had been removed and reinserted in the reader, and so it assumed that the previous contents were still valid -- even across mounts.  The only way to ensure that directory listings of the flashcard contents were actually correct was to remove the module after unmounting, and reload it again before mounting.

Of course, that was several years ago.  I haven't tested to see whether this problem still exists.  It would most certainly simplify life if it was fixed now.