Mix in stuffing mix and bread cubes and cook until well combined, a few more minutes. If your mixture is looking dry, you can add some broth. The mixture should be very moist throughout but not soupy.
Use your fingers to separate oysters into small bite-sized pieces (separate each oyster into about 2-3 pieces) as you add them to the mixture.
Add salt, pepper, cayenne and reserved parsley. Taste for seasoning, add more if desired.
Beat eggs with 1/2 cup broth and mix in. Transfer entire mixture to a buttered or nonstick sprayed baking dish. (If making ahead, cover with foil and store now.)
Top with Panko and drizzle with olive oil. Bake at 350 for 1 hour.