basically any portable operating system should do, whether linux or windows
as soon as the job's done, encrypt the whole (flash-)drive (also empty space!) the OS was running on and format it afterwards
or even better, destroy it physically
not sure though if the firmware stores any traces
average windows users might just uninstall it from device manager and that's it. personally I wouldn't worry about it, since I'm not engaged in criminal activities
but after all: the only evidenve that's left, is that this DEVICE had been used to install GOS, but not WHO actually installed it