Judging by the dealer fee, sounds like you were at a Maroone dealership (Delray perhaps, I got mine there).
Dealer fees can be negotiated away, but keep in mind that the dealer wants to make a certain amount of money off of their car and whether it is invoice plus fees, or no dealer fee and $XXX over invoice, or invoice and they undervalue your trade by $XXX or they mark up the finance rate, the dealer will find a way to make what they want to make, or they will simply not make the deal.
One of the best ways to get around this issue when you are financing (or paying cash) is to talk in terms of an out-the-door price. Just figure the price you want to pay, add in your sales tax and a couple of hundred for tags and present this to the dealer, making sure to stipulate an OTD price. This way, they can't throw in any fees at the end.
Even though the dealer fee is pre-printed on the sales contract, it is easy enough to have them cross it out, if that is what the parties have agreed to.
As for the issue of the invoice price not being the true cost to the dealer, this is an area of confusion for most consumers. Most people who do their research on the net know what the invoice price is and they know the term "holdback" and even what that amount might be. What they don't realize is that money is more often than not already spent.
For example, my Murano had an MSRP of $37,387, an invoice price of $33,978 and a dealer net price of $32,540. The difference between the invoice price and the dealer net is the holdback amount ($1438). Most people think that even if the dealer sells the car at invoice, or even below invoice, they are still making a bunch of money. Well, this is not exactly the case.
First off, most dealers belong to regional advertising associations. They get charged a certain amount for every car they buy (and they do buy them, more on that below). While the amount can vary, it can be as much as 2% of the sticker price, or about $747 in the case of my car. There goes half of the holdback right there.
Next, when the car comes in, it must be prepped. This usually involves removing all shipping protection, final installation of some trim items, inspections, washing and gassing it up. Typically, this is done through the dealership's service department. Since most dealerships run the different parts of their stores (new, used, parts, service and finance) as separate profit centers, each one is accountable for their "billable" hours. The service department will usually charge the sales department 2-3 hours worth of labor. Depending on that dealership's labor charges, this can be as much as $50/hr, or another $150 off the holdback amount.
The dealership also has to pay their overhead, including mortgage, taxes, utilities and also paying that pesky salesperson (in addition to people like the receptionist, porters, file clerks, etc). Even on a mini-deal, most salespeople will make $100. The rest of the overhead charges are hard to put a number on, but I suspect it is at least $200.
So, out of the $1438 in holdback, we are left with about $241 in actual profit if the vehicle was sold at invoice.
But wait, you remember that I said the dealership buys the vehicle. This is true. Every new and used vehicle that a dealership gets is actually financed (the dealership borrows money from a lender). As with most things that are financed, interest must be paid. This interest begins to rack up from the day the car hits the lot. The faster a car is sold, the less interest the dealership has to pay on it and the more the remaining holdback can be put into the profit column of the balance sheet.
In the case of my car, I took delivery the day the car came in, so they paid little or no interest on the car, so the remaining holdback was added profit. However, if the car stays on the lot for days, weeks or months, the interest charges begin to add up, and this eats away at the remaining holdback.
This is not to say that you can't but a car at invoice, or less. My whole point is to show that the holdback amount that most people thing is always pure profit is usually far from it. Just something to keep in mind when you are negotiating and you want to talk intelligently to a dealer about their true invoice price.