The hosts were aiming to shock their neighbors Siwelele for a place in the last 16 after being knocked out by local top flight team Free State Stars in the first round last season in this competition.
Celtic caretaker coaches Lehlohonolo Seema and John Maduka made wholesale changes to the team that lost to SuperSport United the last time out, giving fringe players a chance to prove their qualities.
Neo Maema gave Phunya Sele Sele the lead in the 25th minute with a delightful strike that sailed into the top corner after the visitors showed their superiority in attack.
Celtic started gaining control of the game ten minutes before the break but there could not carve out clear chances, while the hosts struggled to get their attacking game going in the first stanza.
Motsamai Moloi brought the hosts level on the hour-mark with a left footed free-kick that took a deflection and wrong-footed Celtic goalkeeper Kabelo Dambe.
Celtic were looking for fresh legs two minutes later as Rendani Ndou got a run upfront in place of Victor Letsoalo as the visitors went in search of a winning goal.
Mpusana Tani netted the winner for Phunya Sele Sele two minutes before time and sent the ABC Motsepe League side crashing out of the competition.
Celtic: Dambe, Lingwati, Pfumbizai, Mofokeng, Maema, Potsane, Mabasa, Sam, Shikweni, Tani, Letsoalo.
Subs: Chaine, Ndou, Baloyi, Phalane, Dlamini, Masuku, Abondo.
