IMO if you're getting dirt on your drying towels you're not doing enough washing before you dry. Just like with a conventional wash, sometimes you have to go over an area more than once to get it clean. Try doing one pass with one side of the BRS, then going back over with the the other side to remove any remaining dirt. I don't have a BRS, but another thing I've done, with that kind of wash media, is you kind of have 4 sides, if you use the two edges. What I'm saying is kind of like if you were washing with a folded towel, where you flip or re-fold to get to a clean surface, you can wash a panel using the two sides of the BRS, then go back using the edges for a final pass.
Anyway, my recommendation is, if you're having this problem, is go back for a second pass with the other (clean) side of the BRS. You really want the dirt in your wash media and not your drying media.