It’s hard to know without more context into the people you’re referring to. However, generally speaking, early adopters are very forgiving when it comes to UI. Over time, those same users will have higher standards. And, new users will have a higher standard to begin. So while you may see no issue today, the criticism could be a signal that issues might be around the corner and you’d probably want to take them seriously.