Air Pro 4 In-Ear Detection (Sometimes) Doesn't Work

Hello I recently got the Air Pro 4 and noticed that the in-ear detection feature sometimes doesn't work. One earbud could be working (sometimes left, sometimes right) while sometimes both doesn't work at all, the only way to fix this is to put both earbuds in the case, close the case then reconnect and then both earbuds would work flawlessly, pausing music/video whenever I take the earbuds off. 

I updated to the latest firmware and tried factory reset but the issue is still there. 

I turned off the feature for now because it's too inconsistent. I need help solving this, thanks!.