Q. Is there a way to manually add students who are not in a course to an appointment group in Canvas?

No.

Unless the student is in one of your courses, it is not possible to add them to an appointment group in Canvas. You can delete a slot (so it is no longer available) and add the student manually in your Outlook (or other) calendar system.