Autocast

[haytham2597]

Soon i will try make AMP (Automatic Mixed Precision) with GradScaler.

[haytham2597]

@dotnet-policy-service agree

[NiklasGustafsson]

Is this a work-in-progress PR, or something you're submitting for approval and merging? If the latter, then please create an issue to track "to do" items and add some unit tests.

[haytham2597]

U can merging this if you want, this dont break anything (far as I know). But may useful for someone who want use that autocast function manually. My idea and plan is to make AMP, GradScaler, these modules use the functions I added.
Thank, I will try take into account about issue "to do" and unit tests. Sorry.

[NiklasGustafsson]

@haytham2597 -- thank you for your first PR! Much appreciated. Please see the comment I made in the review.

[haytham2597]

Do not merge, i keep have some issue.

[NiklasGustafsson]

Lots of errors in the build on everything except the .NET FX builds (which don't have System.Range):

https://dev.azure.com/dotnet/TorchSharp/_build/results?buildId=103093&view=logs&j=80b813b5-9a08-5859-11a8-dc0e5b556e52&t=d3977768-5d05-5555-eccf-169680cb7093

[lintao185]

I am very happy to see this proposal.

[NiklasGustafsson]

@haytham2597 -- just a gentle ping! I think this PR would be very valuable, but it's still a draft, and thus I will not merge it. I also had some comments in my review.

[haytham2597]

@haytham2597 -- just a gentle ping! I think this PR would be very valuable, but it's still a draft, and thus I will not merge it. I also had some comments in my review.

Yeah, but sorry i am very busy with studied and work. I need managed very well about my time for making some progress on this pull requests, i mean this is very useful for me too.
But i can provide some idea about this if you want continue.

• While the autocast is inside on scope automatically convert the tensor to dtype of autocast.
  For example

torch.Tensor a;
using(var ac = torch.NewAutocast()){
      torch.Tensor b = a;
      torch.Tensor c = torch.arange(...)
}

The b and c should automatically converted to float16 (if that is dtype of mixed precision from f32) including all weight/bias of modules that found inside i mean the module, example: ResNet should passed to mixed precision.

The idea Is very similar that you do with

using (var d = torch.NewDisposeScope())

And in outer scope need back to original dtype. Because the neural should backward with original dtype (on my understood)
With my external THS_Autocast u can determine the dtype that should passed/work and if is enabled/disabled too
I don't know if I explained myself correctly, but feel free to ask.

[NiklasGustafsson]

Yeah, no pressure!

We all have other things to do, so I understand completely. Just wanted to let you know we haven't forgotten about your work, and that it will be appreciated, if and when you find time.

[GilesBathgate]

I would also like to see this completed. It should help with #1136 as well.

[ingted]

Really need this!! Thank you!!

[haytham2597]

About AMP or Autocast, @NiklasGustafsson do you have any idea what the "only" (or more abstraction) method is to obtain the tensor? Because in autocast for example, inner-scope on Autocast should all tensors pass to Float16, So the problem is Tensor have so much operation (ie: sum, prod, some linalg, div, etc.) And i should in every method cast the tensor to specific ScalarType. But I want to see where is one method for that, I thinking about using the IntPtr of Tensor and each call of this (because some method uses that, like prod, sum, etc use that IntPtr) and casting to that ScalarType. Is best idea work with IntPtr tensor right?

P.D: I don't know why i can Compile but cannot run Test so rare.

[NiklasGustafsson]

@haytham2597:

This PR is still labeled 'Draft' -- how close do you think you're getting to having it ready to review and merge?

[haytham2597]

This PR is still labeled 'Draft' -- how close do you think you're getting to having it ready to review and merge?

I am closest but not enough. I need write and Test the GradScaler And need find out how autocast the Module. Including i try use the BF16 of C10 LibTorch because some operator of CPU can pass as BFloat16 also GPU and how we know the netstandard do not have Half struct only Net 5 or newer, i added the Half Struct for Older than Net 5.

TODO:

• C10::BFloat16 and Test
• Finish and Test GradScaler
• Test Half Struct for older Net
• Autocast Cuda Ops
• Autocast CPU Ops Bfloat16
• Autocast Model, Sequential Module
• Implement Test of TestGradScalingMultiple

[GeorgeS2019]

Any update?