Do stick with your current job as per your 2 weeks notice, and be sure to thank your current supervisor/boss. Leave on a good note, then take your next job. Having a good reference is so important and will last forever. Even if it might be inconvenient to stay these last few days, make every effort to complete your current job with a good finish. btw, congrats on your new job. 
_________________
The ones who say “You can’t” and “You won’t” are probably the ones scared that you will. - Unknown