How do I handle chunked http responses?

Question

Hi.
Problems arise when using Https.
After the Https request, the response is received as a chunked.
After receiving the response, the data is read and copied to the Memory Stream.
When trying to read data from the response stream, "invalid chunk." I get an error.
How to deal with this case?

var sourceStream = Response.GetResponseStream();
MemoryStream ms = new MemoryStream();
int count = 0;
byte[] buffer = new byte[65536];

try
{
    do
    {
        count = sourceStream.Read(buffer, 0, 65536);
        if (count <= 0)
            break;

        ms.Write(buffer, 0, count);

    } while (count > 0);

    ms.Position = 0;
    return ms;
}
catch(System.IO.IOException)
{
    ms.Position = 0;
    return ms;
}
catch // Invalid Chunk Exception
{
    throw;
}

-       $exception  {"Invalid chunk."}  System.Net.WebException
+       Data    {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
        HResult -2146233079 int
        HelpLink    null    string
+       IPForWatsonBuckets  0x00007ffb3481d3ed  System.UIntPtr
+       InnerException  {"Invalid chunk."}  System.Exception {uspns.nhitg}
        InternalStatus  RequestFatal    System.Net.WebExceptionInternalStatus
        IsTransient false   bool
        Message "Invalid chunk."    string
        RemoteStackTrace    null    string
+       Response    {Rebex.Net.HttpResponse}    System.Net.WebResponse {Rebex.Net.HttpResponse}
        Source  "Rebex.Networking"  string
        StackTrace  "   위치: uspns.imxim.wwocn(Byte[] p0, Int32 p1, Int32 p2)\r\n   위치: uspns.imxim.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   위치: uspns.dhhhe.twrde(Byte[] p0, Int32 p1, Int32 p2)\r\n   위치: uspns.dhhhe.pewak[T](Func`1 p0)\r\n   위치: RaiDrive.Http.CommonHttpStreamResponseInfo.ReadToMemoryStream() 파일 C:\\Users\\lanop\\source\\repos\\raidrive.service\\RaiDrive.Service\\RaiDrive.Service.Share\\Http\\CommonHttpResponseInfo.cs:줄 153" string
        Status  ReceiveFailure  System.Net.WebExceptionStatus
+       TargetSite  {Int32 wwocn(Byte[], Int32, Int32)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}
        WatsonBuckets   null    object
        _HResult    -2146233079 int
        _className  null    string
+       _data   {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
        _dynamicMethods null    object
+       _exceptionMethod    {Int32 wwocn(Byte[], Int32, Int32)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}
        _exceptionMethodString  null    string
        _helpURL    null    string
+       _innerException {"Invalid chunk."}  System.Exception {uspns.nhitg}
+       _ipForWatsonBuckets 0x00007ffb3481d3ed  System.UIntPtr
        _message    "Invalid chunk."    string
        _remoteStackIndex   0   int
        _remoteStackTraceString null    string
+       _safeSerializationManager   {System.Runtime.Serialization.SafeSerializationManager} System.Runtime.Serialization.SafeSerializationManager
        _source "Rebex.Networking"  string
+       _stackTrace {sbyte[192]}    object {sbyte[]}
        _stackTraceString   null    string
        _watsonBuckets  null    object
        _xcode  -532462766  int
+       _xptrs  0x0000000000000000  System.IntPtr
        m_InternalStatus    RequestFatal    System.Net.WebExceptionInternalStatus
+       m_Response  {Rebex.Net.HttpResponse}    System.Net.WebResponse {Rebex.Net.HttpResponse}
        m_Status    ReceiveFailure  System.Net.WebExceptionStatus
+       Static members      

Thank you.

Best Regards.

BooKyung Oh.
OpenBoxLab Inc.

Comments

using (System.IO.StreamReader reader = new System.IO.StreamReader(ResponseResult, System.Text.Encoding.UTF8))
{
    data = reader.ReadToEnd();
}

This code also gives the same error.

Answer 1

Rebex HTTPS is supposed to handle chunked responses automatically. When "invalid chunk" error occurs, it indicates that the library was unable to parse the chunked response received from the server. This usually indicates invalid response (such as non-chunked data when chunked data was expected), although it could also indicate a bug in our chunked response parser.

Unfortunately, it's not possible to determine the exact cause based on an error message and stack trace alone. Would it be possible to create a log at LogLevel.Verbose and either post the relevant section (showing incoming data before the failure occured) here or mail it to support@rebex.net for analysis? Thanks!

Comments

Thanks for the reply.
Analyzing the server's response, the server sends incomplete chunked data.
The server immediately disconnects the socket after sending some data. (Although KeepAlive is set)
It's probably a server bug.

In this case, if only partial data is received, .Net's HttpWebRequest can read all of the partial data (eventually, System.IOException will appear. If the error is ignored, the data can be processed.),
Rebex Https throws an error immediately.

Below is the error part in the log.
Detailed logs will be sent by e-mail.

<pre>
2021-03-16 11:06:56.140 VERBOSE HttpRequest(8)[29] HTTP: Received data:
 0000 |0D-0A                                          | ..
2021-03-16 11:06:56.140 VERBOSE HttpRequest(8)[29] HTTP: Socket closed by the server.
2021-03-16 11:06:56.191 ERROR HttpRequest(8)[29] HTTP: Error while reading response: System.ArgumentNullException: 값은 null일 수 없습니다.
매개 변수 이름: String
   위치: System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   위치: System.Number.ParseInt64(String value, NumberStyles options, NumberFormatInfo numfmt)
   위치: uspns.imxim.wwocn(Byte[] p0, Int32 p1, Int32 p2)
   위치: uspns.imxim.Read(Byte[] buffer, Int32 offset, Int32 count)
2021-03-16 11:06:58.396 DEBUG HttpRequest(8)[29] HTTP: Closing response stream.
2021-03-16 11:06:58.397 DEBUG HttpRequest(8)[29] HTTP: Closing HTTP session (4).
</pre>

---

Thanks! Our receiver apparently tries to fill the user-supplied buffer in the Stream.Read call, and fails when the connection is unexpectedly closed. We can tweak it to at least return the data received so far (and fix the internal ArgumentNullException as well). I'll send you a link to a hotfix to try when it's ready.

---

This has been addressed in Rebex HTTPS R5.2: https://www.rebex.net/https/history.aspx#R5.2