The pier alone does not justify the drive for most people. Combined with what is around it, the day works well.
The fort is immediately here, a masonry coastal fortification at the mouth of the bay with its own long history and its own page on this site. An hour there and two hours on the pier is a reasonable morning.
The ferry runs from the same point across to Dauphin Island, which turns the trip into a genuine expedition: pier, fort, crossing, island, back. Check it is operating before you plan around it, because the ferry suspends for wind and sea and has been out of service at times.
The beaches along the peninsula on the way out are the quietest on the Alabama coast, with a fraction of the people of Gulf Shores and almost none of the infrastructure. No restrooms, no lifeguards, no parking machines, no shops.
The refuge sits along the same road, holding the last undeveloped dune habitat in the state.
The honest sequencing: drive out early, fish the morning tide, do the fort when the sun gets high, and stop at a beach on the way back. It is around forty-five minutes each way from Gulf Shores on a single road with no alternative if it closes.