Best is 3-4 months, but anything less than 2 months and you risk having some emotional problems.
Except for Baron, all of our cats are stray/rescue/HS cats. My Mother'* two youngest cats were both rescued as fuzzballs. Mickey is okay, but Patches is very skittish around anyone but Mom. The two little black cats(brother and sister), that my Brother and I have were just barely eyes open when my Mother found them in cabinets at a picnic shelter at her park. Gypsy is okay, if extremely hyper and a bit too dominant,, but Rover is extremely needy, to the point of neurosis. If we close our doors at night, he will spend hours begging to come in.
We didn't have any choice in any of these cases, but the longer you can leave the kitten with it'* mother, the better. 3-4 months is plenty old enough to avoid emotional damage, but still young enough that the kittens can imprint on you.
And you should NEVER take a kitten from it'* mother if it'* not eating solids yet(again, if it can be helped)..