I had an issue with this my first year and have become a complete demon in following years to make sure it never happens again. Here's a brief rundown of what I do, and I've had exactly zero issues since I started. You've potentially done most of these things already, but I'll list out everything to be safe. I use Google Sheets to keep track of everything but know people use Excel or other programs instead.
1. I get all of the uniforms and put them in piles based off of what they are, then come up with a very specific name for them. I avoid terms like 'JV top' or 'Varsity skirt' because it may switch up year to year. A couple of examples are "GTM green shell, black side bands, 'YHS Roughriders' in white/black sparkle" or "REBEL skirt, dark green with black panel, white piping and "Riders". The more specific the better. I have a spreadsheet with every uniform name as well as how many of each size I have of that particular item, and the abbreviation I'll use for them moving forward (this is just to make it fit nice and neat in my passed out sheet)
2. Once I have all that information, I assign each uniform piece a number and write in in sharpie on the tag as well as somewhere inconspicuous on the fabric itself (usually the inside of the back of the neck). This means that even if the tag gets ripped off, the number is still on the piece itself so it cannot be confused. I do this for each uniform piece, and I always start with my smallest size as a 1 up until my largest size, which will be the same as the total number of that uniform piece that I have. I then document all of this on that Uniform spreadsheet that I mentioned above.
3. Now that everything is labelled and put into a spreadsheet, I make a separate spreadsheet for what I call the "pass out numbers". Before I give the kids their uniforms, I get them and their parents to sign a paper that basically says "the number uniform that is assigned to me is the one I'm responsible for, if that uniform is damaged, whether by me or another individual, I am the one who will be charged". This will generally get them pretty on top of making sure they always have their uniform and not somebody else's after they change out, so that they won't swap uniforms on accident. I also have them double check their numbers every couple of months just to make sure.
4. Any uniforms I'm not using I store in number order in a box with a printed off version of the spreadsheet that has the corresponding numbers to the sizes on it. This just helps me if I choose to use them in the future.
5. At the end of the season when they return the uniforms I inspect them and then store them back in number order. If any are damaged, I don't even look at which girl handed it to me, I just look at the spreadsheet to see who was responsible for it and apply the charge to their account. Haven't had this happen yet where it didn't match up, but hopefully their signed paper would help me be able to fight for the charge. I've only had three charges in my time though so it isn't a frequent issue.
Hopefully I explained that at least semi decently? I love spreadsheets and color coordination so I can go a little bit crazy with everything, but it definitely helps in the long run. Just takes a while to get everything set up - especially if your uniforms aren't already numbered.