He probably bit you because he was hungry, and he's coming out now because he's hungry.

There are two things to address here: 1. the fact that you're becoming afraid of him, and 2. his behavior. You might as well feed him more, at least until his appetite drops off (I guarantee it will---he'll probably start stressing you out by skipping meals this winter). But you also ought to handle him more often without feeding, so he doesn't always associate "human handling" with "food".

I really recommend handing with gloves or mittens. My het pied male was mean little biter his first year, so I let him chew on my gloves. It's not terribly rewarding for him, and it got around my fear.

And I recommend feeding him in a cardboard box. Then he'll associate the box with food, not you. It really helps with cage aggression.