Trisodium phosphate (TSP) is an effective heavy-duty cleaner that's good on brick, but that's tough to find in some areas. If your region is one of them - or if the phosphates give you pause, there's a version of TSP that's phosphate free.
Whether you go with phosphate-free TSP or the good old-fashioned kind, wear protective gear, and do your best not to let the TSP contact your skin. In a bucket, mix 1/2 cup of the TSP with one gallon of hot water. and use a scrub brush to scrub it into the brick. Rinse it thoroughly and let it dry.
Good luck! Post before and after photos to Money Pit's Facebook page!