I highly recommend never using an envelope, only boxes. The last thing you want is more damage to your book in transit. Often the smaller the box, the better the shipping cost.
Large or heavy books especially should receive padding around ALL sides: bottom, sides and top. Fragile and or oversized books (e.g. family bibles) can’t receive too much padding.
There should be no empty space in the box for the book to shift around. Padding should reach all areas of the box and snuggly around the book.
All carriers toss and throw their boxes, but in my experience I’ve received more damaged boxes from the USPS and have had the most difficulty filing claims with them. I personally recommend UPS Ground services or FedEx. However, the USPS does offer a Media Mail rate on books which gives the best price when you need an economical option.