Using flow.js to upload to Google Cloud Storage?

[maclav3]

Hello flow.js developers,

I am currently developing a web application that enables users to upload files, but the requirement is that they be stored on Google Cloud Storage instead of the server providing the application. Handling multiple uploads concurrently is also required.

To reduce bandwidth usage and server loads, Google recommends communicating from the client directly with GCS servers, omitting the server providing the web application. The preferred way for me would be probably either with Signed URLs or Signed Policy Documents, because they do not require logging in with a Google Account https://cloud.google.com/storage/docs/access-control

Providing a test handler and an upload handler is easy when in control of the destination server, but to work with GCS one would need to adjust to their API.

I am thoroughly impressed by how flow.js works when uploading to a server that provides a suitable API and had little success finding any good prefabricated front-end solutions that would work with GCS. I thought that a whole lot of existing flow.js code (and ng-flow, which is equally adorable) could be put to use here, if only the parts involved in POST requests were adjusted.

As to the Google Cloud Storage API, it is a REST interface, is reasonably well documented, supports resumable and chunked uploads.
https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload#resumable

I don't think that it has an equivalent of flow's test handler, but the responses from chunk uploads report of error/success of each chunk's upload.

So here comes my question:
Is it realistically possible to divert the relevant XHR parts of flow.js's code to communicate with Google Cloud Storage's API or do you foresee any design barriers that would make attempts to rewrite it a wasted effort? I have done a preliminary scan of flow's code, but having no prior experience with its inner workings I fear I might miss some crucial and obvious facts.

The biggest advantage of adapting the code of flow.js to this task would be reusing the parts of user interface and ng-flow instead of reinventing the wheel for just communicating with a different API. And that the code is already mature, tested and widely compatible.

I suspect that most of you would have no motivation to implement such functionality right now, so I guess that I could fork and contribute to the project, but I would be really grateful for your expertise and highlighting possible issues with this undertaking.

[maclav3]

FYI: After debate, we've changed the flow so that XHRs to Google Cloud Storage are no longer needed. I leave this issue opened for consideration in the future, but don't plan to implement it at the moment.

[askdesigners]

So what was your solution? I'm needing to exactly the same thing.

[maclav3]

@askdesigners For now, we decided to use a ng-flow-compatible backend that writes to the local file system on our own server, sorry to dissapoint you.

Another solution would be probably to migrate to Google App Engine and write a backend using native GCS API, exposing a flow-compatible REST API to the frontend.

[askdesigners]

ya I was afraid of that. :) I'm using node on the backend, so I'm going to look at streaming the chunks straight through the app to GCS. No idea how that will work out. I plan to dig into that after this week so I'll post my experience here for posterity.

[Roycohen]

Uploading files to GCS can be done without changing the library.
You just need to hook to the right places.

[dcardosods]

@Roycohen I was able to upload to GCS by setting the target url properly and chunkSize = Number.MAX_SAFE_INTEGER (so the file is not split into chunks). But I'm still to find a way to hook up "Resuming an interrupted upload" from GCS and flow.js "retries". What was your solution?

[aayusharyan]

@Roycohen

Uploading files to GCS can be done without changing the library.
You just need to hook to the right places.

Please share the steps to do so.

[aayusharyan]

FYI, I made PR which might help to get it resolved. #276