By moving him your stressing him out and giving him a much higher chance of showing aggression to you. The cage association is a myth unless you smell like mouse, you should not be bitten unless it's a nippy animal. (some are)
If the temperature is too cold in the separate cage, all the more reason to keep him in his main cage. Moving him out once a week could cause an RI due to change in temperatures so drastically and quickly.
Don't worry about him ingesting chips or bedding either. Unless it's sand (which is BAD) there's nothing to worry about, and impaction is very very rare.