Skip to content

Remove 100 Continue from boto3 requests since it messes with RGW/Ceph in prod

Yuto Takano requested to merge fix-boto3-expect into master

I think this should fix the issue of not being able to upload to S3 from ComSol.

boto3 by default splits every upload request into 2 parts:

  1. First boto3 request is only headers and uses "Expect: 100 Continue".
  2. (Then, the S3 service should reply with 100 Continue if the upload is allowed)
  3. Then boto3 sends the actual file body

Theoretically this is useful for preventing sending the big file if the request isn't allowed anyway.

In the prod cluster, it seems like RGW/Ceph doesn't actually send us back "100 Continue" when we want it to.

See network trace below where the S3 gateway responds with 200 OK instead of 100 Continue, and boto3 just waits forever as a result. In Slack, @jhoffmann mentioned also 504 Timeout being returned from the gateway, I wasn't able to reproduce that so I don't know if it's the same issue.

Screenshot

Anyway, we can hotfix this (not the real solution) by telling boto3 not to split the request in two parts. We can do this by adding a boto3 hook to remove the "Expect" header. Now, everything is sent in one go.

Merge request reports