Ever see a team invest a bunch of time into deciding how many story points they could claim at the end of a sprint?
You get what you teach people to give you.
Positioning story points as valuable encourages people to treat them as rewards. Rather than trying to be productive and "earning" points, time will be wasted trying to claim points for work that isn't really done.
Wasted time erodes productivity, and lowered productivity provides more incentive for credit-grabbing behaviors. A vicious cycle ensues.
This is what you get when you set story points up as these little food pellets meant to modify the behavior of your developers: Developers whose behavior is optimized for getting story points, not writing software.
The solution is simple but very hard: Start tracking what you want.
Not something you think will lead to what you want. Not something you consider to be an integral part of what you want. Track whether or not you are actually getting what you want.
