If you go to Normandy you should combine it with Britanny - as Normandy is not very rich in mineral localities.
If you can get good contacts with the society in charge of the dump, you might visit Chessy on your way to the Alps (some members are here).
After visiting the Chamonix museum, I suggest you to stay in Bourg d’Oisans, so you can combine hiking in nice landscapes with mineral collecting (mainly quartz).
There are many regions to collect minerals in France, but you don't need to focus on quarries, or do visit quarries with the help of local contacts.
Choose the region first, then we will find helpful contacts for you!