Heat oil in a sauté pan over medium-high heat. Season shrimp with salt and pepper. When oil begins to smoke, add the shrimp, garlic, shallots, rosemary, crushed red pepper, and capers. Toss the pan to mix the ingredients.
Put the brandy in a nonbreakable measuring cup, then pour it from the cup to the pan (never directly from the bottle!). Hold your breath, as it will briefly flame up, and the smoke can choke you if inhaled. Add the lemon juice. Continue cooking over medium-high heat for about 30 seconds. Add the butter and cook until it’s melted. At this time, the shrimp should be just cooked. Stand the shrimp up in a circle around a bowl or plate, and pour sauce over the top. Serve with toast points.