Yes you will hit that problem with large files. The S/MIME encrypt streams with a low memory overhead while decrypt does not stream and needs the whole 
file in memory. It's a serious amount of effort to stream decrypt and so far there hasn't been enough interest to do that.