The uncertainty of renewable energy development in China has been exacerbated by institutional reforms including electricity marketization, the removal of subsidies for renewable power, and carbon emissions trading. This study analyses the investment decision of renewable energy when renewable and fossil power are connected at grid parity and in competition and discusses how intensity-based carbon trading and the intermittency of renewables affect energy investment and CO2 emissions. Meanwhile, using data from China’s electricity and carbon markets, it verifies the theoretical results with numerical simulations and estimates the scope for the further increase of the carbon price from the perspective of marginal abatement cost. The results suggest that: 1) Carbon trading does not provide a direct incentive for renewable energy investment in the grid-parity scenario, and its impact on fossil energy depends on the magnitude of the emission intensity of fossil power relative to its benchmark emission rate. 2) Under market competition, the lower the benchmark emission rate of fossil power, the more significant the incentive (disincentive) effect of carbon trading on renewable (fossil) energy investment. 3) Renewable energy investment under market competition is lower than the case at grid parity. However, reasonable carbon pricing under market competition can achieve a low-carbon transition of the power system. The research in this paper can provide a useful reference for the energy structure transition and the synergistic development of the electricity-carbon market.