Preheat the grill for medium heat (~400°F/200°C).
Finely chop the reserved mushroom stems.
Warm the oil in a small skillet over medium heat. You can do this on your stovetop, grill side burner, or directly on the grill. Add the mushrooms and cook, stirring frequently, until they soften and release their liquid, about 5 minutes.
Stir in the garlic and cook for 1 minute, reducing the heat if the garlic starts to brown. Remove from the heat and let cool while you prepare the filling.
In a medium bowl, mix the cream cheese, Parmesan, panko, cayenne, and cooked mushroom stems and garlic. Taste a little nibble and season the cheese mixture to taste with salt and pepper.
Working with one mushroom at a time, hold the mushroom cap upside down. Use a small spoon to press about 1 tablespoon of the cheese mixture into each mushroom. Pack it in firmly, but don't press too hard or you will break the mushroom.
Place the mushrooms, stuffing side up on a grill tray, if using. Transfer the grill tray to the grill and cook the mushrooms until they begin to shrivel and the edges are bubbling, about 8 minutes. Alternatively, you can use tongs to transfer each mushroom directly to the grill.
Remove from the heat and serve warm.