Trump said it was "inappropriate" to have a summit at this time amid heightened tensions with North Korea.