10 points! How can I get the metal filings out that are stuck in my magnetic bit driver?

We putting up a storage building and using over 100 pounds of self-driving screws to hold it together. But whenever we go down to make a connection, the fines get stuck in the magnet bit driver. We've tried using a bigger magnet attached to a screwdriver to get them out, but it only works a little bit. The screws won't stay seated in the socket now and it's driving us nuts. Anyone know how to deal with this?