Is drifting really popular in Japan?

To put it short, yes, drift is a thing in Japan, although it's illegal. It's not that Japan is full of drifters at night, more likely you will never see them unless you go to the places where they often do their thing, because otherwise they would be easily found and caught by the police.