The few studies that have analyzed the factorial structure of early number skills have mainly used confirmatory factor analysis (CFA) and have yielded inconsistent results, since early numeracy is considered to be unidimensional, multidimensional or even underpinned by a general factor. Recently, the bifactor exploratory structural equation modeling (bifactor-ESEM)—which has been proposed as a way to overcome the shortcomings of both the CFA and the exploratory structural equation modeling (ESEM)—proved to be valuable to account for the multidimensionality and the hierarchical nature of several psychological constructs. The present study is the first to investigate the dimensionality of early number skills measurement through the application of the bifactor-ESEM framework. Using data from 644 prekindergarten and kindergarten children (4 to 6 years old), several competing models were contrasted: the one-factor CFA model; the independent cluster model (ICM-CFA); the exploratory structural equation modeling (ESEM); and their bifactor counterpart (bifactor-CFA and bifactor-ESEM, respectively). Results indicated acceptable fit indexes for the one-factor CFA and the ICM-CFA models and excellent fit for the others. Among these, the bifactor-ESEM with one general factor and three specific factors (Counting, Relations, Arithmetic) not only showed the best model fit, but also the best coherent factor loadings structure and full measurement invariance across gender. The bifactor-ESEM appears relevant to help disentangle and account for general and specific factors of early numerical ability. While early numerical ability appears to be mainly underpinned by a general factor whose exact nature still has to be determined, this study highlights that specific latent dimensions with substantive value also exist. Identifying these specific facets is important in order to increase quality of early numerical ability measurement, predictive validity, and for practical implications.