https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md#trainingtest-tips
Preprocessing
Images can be resized and cropped in different ways using--preprocess option. The default option'resize_and_crop'resizes the image to be of size(opt.load_size, opt.load_size)and does a random cropof size(opt.crop_size, opt.crop_size).'crop'skips the resizing step and only performs random cropping.'scale_width'resizes the image to have width opt.crop_sizewhile keeping the aspectratio.'scale_width_and_crop'first resizes the image to have widthopt.load_sizeand then does random cropping of size(opt.crop_size, opt.crop_size).'none'tries to skip all these preprocessing steps.However, if the image size is not a multiple of some number depending on the number of downsamplings of the generator, you will get an error because the size of the output image may be different from the size of the input image. Therefore,'none' option still tries to adjust the image size to be a multiple of 4.You might need a bigger adjustment if you change the generator architecture. Please see data/base_datset.py do see how all these were implemented.
Fine-tuning/resume training
To fine-tune a pre-trained model, or resume the previous training, use the--continue_trainflag. The program will then load the model based one poch. By default, the program will initialize the epoch count as 1. Set--epoch_countto specify a different starting epoch count.
About image size
Since the generator architecture in CycleGAN involves a series of downsampling / upsampling operations, the size of the input and output image may not match if the input image size is not a multiple of 4. As a result, you may get a runtime error because theL1 identity loss cannot be enforced with images of different size. Therefore, we slightly resize the image to become multiples of 4even with--preprocess none option. For the same reason,--crop_sizeneeds to be a multiple of 4.
Training/Testing with high res images
CycleGAN is quite memory-intensive as four networks (two generators and two discriminators) need to be loaded on one GPU, so a large image cannot be entirely loaded. In this case, we recommend training with cropped images. For example, to generate 1024pxresults, you can train with--preprocess scale_width_and_crop--load_size 1024 --crop_size 360, and test with--preprocess scale_width --crop_size 1024. This way makes sure the training and test will be at the same scale. At test time, you can afford higher resolution because you don’t need to load all networks.
About loss curve
Unfortunately, the loss curve does not reveal much information in training GANs, and CycleGAN is no exception. To check whether the training has converged or not, we recommend periodically generating a few samples and looking at them.
About batch size
For all experiments in the paper, we set the batch size to be 1.If there is room for memory, you can use higher batch size with batch norm or instance norm. (Note that the default batch norm does not work well with multi-GPU training. You may consider using synchronized batch norm instead). But please be aware that it can impact the training. In particular, even with Instance Normalization, different batch sizes can lead to different results. Moreover, increasing--crop_size may be a good alternative to increasing the batch size.
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/qa.md#frequently-asked-questions
ValueError: empty range for randrange()
Similar error messages include "Connection Refused Error: [Errno111] Connection refused"
It is related to data augmentation step. It often happens when you use--preprocess crop. The program will crop random crop_size x crop_size patches out of the input training images. But if some of your image sizes (e.g.,256x384) are smaller than the crop_size(e.g., 512), you will get this error. A simple fix will be to use other data augmentation methods such as resize_and_cropor scale_width_and_crop.Our program will automatically resize the images according to load_size before applycrop_size x crop_sizecropping. Make sure thatload_size >=crop_size.
Can I continue/resume my training?
You can use the option--continue_train. Also set--epoch_countto specify a different starting epoch count. See more discussion in [training/test tips](https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/docs/tips.md#training test-tips.
Multi-GPU Training
You can use Multi-GPU training by setting--gpu_ids(e.g.,--gpu_ids 0,1,2,3for the first four GPUs on your machine.) To fully utilize all the GPUs, you need to increase your batch size. Try--batch_size 4,--batch_size 16, or even a larger batch_size. Each GPU will process batch_size/#GPUs images. The optimal batch size depends on the number of GPUs you have, GPU memory per GPU, and there solution of your training images.
We also recommend that you use the instance normalization for multi-GPU training by setting--norm instance. The current batch normalization might not work for multi-GPUs as the batch norm parameters are not shared across different GPUs. Advanced users can try synchronized batch norm.
Can I run the model on CPU?
Yes, you can set--gpu_ids -1. See training/test tips for more details.
Out of memory
CycleGAN is more memory-intensive than pix2pix as it requires two generators and two discriminators. If you would like to produce high-resolution images, you can do the following. During training, train CycleGAN on cropped images of the training set. Please be careful not to change the aspect ratio or the scale of the original image, as this can lead to the training/test gap. You can usually do this by using--preprocess crop option, or--preprocess scale_width_and_crop.
Then at test time, you can load only one generator to produce the results in a single direction. This greatly saves GPU memory as you are not loading the discriminators and the other generator in the opposite direction. You can probably take the whole image as input. You can do this using--model test --dataroot [path to the directory that contains your test images (e.g., ./datasets/horse2zebra/trainA)] --model_suffix _A --preprocess none . You can use either--preprocess none or--preprocess scale_width --crop_size [your_desired_image_width]. Please see the model_suffix and preprocess for more details.
What is the identity loss?
We use the identity loss for our photo to painting application. The identity loss can regularize the generator to be close to an identity mapping when fed with real samples from the target domain. If something already looks like from the target domain, you should preserve the image without making additional changes. The generator trained with this loss will often be more conservative for unknown content. Please see more details in Sec 5.2 ''Photo generation from paintings'' and Figure 12 in the CycleGAN paper. The loss was first proposed in the Equation 6 of the prior work [Taigman et al., 2017].
The color gets inverted from the beginning of training
The authors also observe that the generator unnecessarily inverts the color of the input image early in training, and then never learns to undo the inversion. In this case, you can try two things.
First, try using identity loss--lambda_identity 1.0or--lambda_identity 0.1. We observe thatthe identity loss makes the generator to be more conservative and make fewer unnecessary changes. However, because of this, the change may not be as dramatic.
Second, try smaller variance when initializing weights by changing--init_gain. We observe that smaller variancein weight initialization results in less color inversion.
pix2pix/CycleGAN has no random noise z
The current pix2pix/CycleGAN model does not take z as input. In both pix2pix and CycleGAN, we tried to add z to the generator: e.g., adding z to a latent state, concatenating with a latent state, applying dropout, etc., but often found the output did not vary significantly as a function of z. Conditional GANs do not need noise as long as the input is sufficiently complex so that the input can kind of play the role of noise. Without noise, the mapping is deterministic. Please check out the following papers that show ways of getting z to actually have a substantial effect: e.g.,BicycleGAN, AugmentedCycleGAN, MUNIT, DRIT,etc.
