Sounds like air is in the system;or busted accumulator.
On ABS brake systems you were supposed to pump the brake pedal numerous times to discharge pressure in the brake accumulator(keeps pressure up thoughout the brake system), then you can remove the pads and push in the caliper pistons. If you did not do that you can experiance the problems you are having. Thats properly why you couldn't push the piston back in all the way.
You should first bleed the system, if you can get hold of a power brake bleeder use that, if bleeding the system dosen't fix it, you might have to get a new accumulator.