Wow, I just had to change the same bulb (on the passenger side) on my 04 SE. This is only the third bulb (exterior AND interior) that I've had to change in 80K miles and almost 6 years of ownership. The other two were both of the license plate bulbs.
I was dreading changing this side marker bulb because I knew the fender protector (or at least part of it) had to be removed, but it turned out to be a very easy job. I pulled the wheel for better access but once you know how much of the protector has to be loosened up to get at the bulbs, it's probably possible to do with the wheel on.
First, only the front fender protector has to be removed, and it doesn't have to be completely removed - only loosened up on the top portion which then pulls away from the car for access to the bulbs. I started by taking out the 3 phillips screws that hold the front mud flap on, but I ended up leaving the flap in place (so don't bother taking these 3 screws out). To pull away the top part of the front frender protector, you only have to remove 5 of the plastic clips (type C205, shown on pages EI-11 and EI-22 in my 04 FSM) and NO screws. Once the protector is pulled away from all of its edge supports, the burned out bulb will be right in front of you. Unscrew the bulb holder, pop in a new bulb (I think it was a 194), replace the protector and the 5 clips (one of mine broke so I replaced it with a spare type C206 that I had left over from the trunk of my Maxima), and you're done.
Even with jacking the car up, removing the wheel, driving to Auto Zone for the bulb, and messing around way too long with those chintzy plastic clips, the whole job too less than 1/2 hour. Once you have access to the bulb and a new one in your hand, it takes about a minute to actually change the bulb out. The hardest part is loosening and pulling away the top part of the front fender protector.