In contrary, there should be the ‘healthy’ effect that your system will boot slightly faster as it will not have to load unused modules.