The defeat broke Amakhosi's four-match unbeaten run in the league as they failed to move into second position and keep up the pressure on leaders Mamelodi Sundowns.
"Of course, we didn't get off to a good start," Baxter told SuperSport TV.
"Of course that's a body blow. Then we've got a hill to climb. They play with confidence after that. They're slippery to try and get a co-ordinated press on. But I don't think we played with enough quality if I'm going to sum up the whole game. We didn't play with enough quality at any time, really.
"When you're not playing well with the ball then you end up going route one all of the time and that's not our game. Yes, you can play that way. All the training that we do is about the ball being under control. We've done a lot of work on our movement. But if you're playing balls to players with bad first touches and the straight ball that gets cut out. Without building you can't time any movements.
"But we still created a couple of chances and if we'd taken one of them it would have lifted our spirits. But it felt like we were treading water. It felt like we were going uphill the whole evening.
"I thought the back four for me was not working. They weren't co-ordinated enough. They weren't brave in holding our line. They weren't in sync and therefore we went to a back three. It gave us a higher press on their wide players. They tried in the beginning to hold a higher line but the longer it goes the more you're thinking of changing the shape and pushing on one more offensive player. So at the finish we ended up playing with a front four with Mshini and Bibo supporting the front two. So it becomes tactically a little bit messy but you've got to try and push on and try and change the game in some way.
"And as I say we could have got in. With Reeve getting higher, he got a couple of very good crosses in. But again the quality was just not there.
"There's lots of game to play and there's lots of work to do. We wanted to try and keep as much pressure as we could on the leaders. Now I think it's probably with us losing it's probably given them enough room to relax and then they'll pick up enough points to win it."