Apply custom randomization on __getitem__ during training.

[ysig]

Hi,

I have an extremely simple use case which I cannot understand how to do with your library.

With load_dataset I load a text dataset and I want to apply randomization to the input everytime __getitem__ (or the equivalent - however it is called) is called on the huggingface dataset object.

(example: getitem normally returns a text and let's say I want to shuffle the words)

I really cannot understand how to do it and I've been searching for more than 2 hours.

Please help!

Thanks,

[mariosasko]

Hi! You can use set_transform for that:

def transform(batch):
    batch["text"] = [shuffle_words(text) for text in batch["text"]]
    return batch

dset.set_transform(transform)

[ysig]

Thank you so much ::)))