Following Paul's suggestion, I changed the hole-filling method to use only neighboring pixels that are not themselves holes.  Previously, I simply regarded these as zeros.  The results are shown below.  There is obvious improvement: less blurring and darkening, and larger regions are filled.
 
 
 

Same 4 views as before, but will new hole-filling method.
There is less darkening and blurring, and more regions are filled

Here's another zoomed in view: