How do I handle chunked http responses?

Question

var sourceStream = Response.GetResponseStream();
MemoryStream ms = new MemoryStream();
int count = 0;
byte[] buffer = new byte[65536];

try
{
    do
    {
        count = sourceStream.Read(buffer, 0, 65536);
        if (count <= 0)
            break;

        ms.Write(buffer, 0, count);

    } while (count > 0);

    ms.Position = 0;
    return ms;
}
catch(System.IO.IOException)
{
    ms.Position = 0;
    return ms;
}
catch // Invalid Chunk Exception
{
    throw;
}

-       $exception  {"Invalid chunk."}  System.Net.WebException
+       Data    {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
        HResult -2146233079 int
        HelpLink    null    string
+       IPForWatsonBuckets  0x00007ffb3481d3ed  System.UIntPtr
+       InnerException  {"Invalid chunk."}  System.Exception {uspns.nhitg}
        InternalStatus  RequestFatal    System.Net.WebExceptionInternalStatus
        IsTransient false   bool
        Message "Invalid chunk."    string
        RemoteStackTrace    null    string
+       Response    {Rebex.Net.HttpResponse}    System.Net.WebResponse {Rebex.Net.HttpResponse}
        Source  "Rebex.Networking"  string
        StackTrace  "   위치: uspns.imxim.wwocn(Byte[] p0, Int32 p1, Int32 p2)\r\n   위치: uspns.imxim.Read(Byte[] buffer, Int32 offset, Int32 count)\r\n   위치: uspns.dhhhe.twrde(Byte[] p0, Int32 p1, Int32 p2)\r\n   위치: uspns.dhhhe.pewak[T](Func`1 p0)\r\n   위치: RaiDrive.Http.CommonHttpStreamResponseInfo.ReadToMemoryStream() 파일 C:\\Users\\lanop\\source\\repos\\raidrive.service\\RaiDrive.Service\\RaiDrive.Service.Share\\Http\\CommonHttpResponseInfo.cs:줄 153" string
        Status  ReceiveFailure  System.Net.WebExceptionStatus
+       TargetSite  {Int32 wwocn(Byte[], Int32, Int32)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}
        WatsonBuckets   null    object
        _HResult    -2146233079 int
        _className  null    string
+       _data   {System.Collections.ListDictionaryInternal} System.Collections.IDictionary {System.Collections.ListDictionaryInternal}
        _dynamicMethods null    object
+       _exceptionMethod    {Int32 wwocn(Byte[], Int32, Int32)} System.Reflection.MethodBase {System.Reflection.RuntimeMethodInfo}
        _exceptionMethodString  null    string
        _helpURL    null    string
+       _innerException {"Invalid chunk."}  System.Exception {uspns.nhitg}
+       _ipForWatsonBuckets 0x00007ffb3481d3ed  System.UIntPtr
        _message    "Invalid chunk."    string
        _remoteStackIndex   0   int
        _remoteStackTraceString null    string
+       _safeSerializationManager   {System.Runtime.Serialization.SafeSerializationManager} System.Runtime.Serialization.SafeSerializationManager
        _source "Rebex.Networking"  string
+       _stackTrace {sbyte[192]}    object {sbyte[]}
        _stackTraceString   null    string
        _watsonBuckets  null    object
        _xcode  -532462766  int
+       _xptrs  0x0000000000000000  System.IntPtr
        m_InternalStatus    RequestFatal    System.Net.WebExceptionInternalStatus
+       m_Response  {Rebex.Net.HttpResponse}    System.Net.WebResponse {Rebex.Net.HttpResponse}
        m_Status    ReceiveFailure  System.Net.WebExceptionStatus
+       Static members      

Comments

using (System.IO.StreamReader reader = new System.IO.StreamReader(ResponseResult, System.Text.Encoding.UTF8))
{
    data = reader.ReadToEnd();
}

This code also gives the same error.

Answer 1

Comments

Thanks for the reply.
Analyzing the server's response, the server sends incomplete chunked data.
The server immediately disconnects the socket after sending some data. (Although KeepAlive is set)
It's probably a server bug.

In this case, if only partial data is received, .Net's HttpWebRequest can read all of the partial data (eventually, System.IOException will appear. If the error is ignored, the data can be processed.),
Rebex Https throws an error immediately.

Below is the error part in the log.
Detailed logs will be sent by e-mail.

<pre>
2021-03-16 11:06:56.140 VERBOSE HttpRequest(8)[29] HTTP: Received data:
 0000 |0D-0A                                          | ..
2021-03-16 11:06:56.140 VERBOSE HttpRequest(8)[29] HTTP: Socket closed by the server.
2021-03-16 11:06:56.191 ERROR HttpRequest(8)[29] HTTP: Error while reading response: System.ArgumentNullException: 값은 null일 수 없습니다.
매개 변수 이름: String
   위치: System.Number.StringToNumber(String str, NumberStyles options, NumberBuffer& number, NumberFormatInfo info, Boolean parseDecimal)
   위치: System.Number.ParseInt64(String value, NumberStyles options, NumberFormatInfo numfmt)
   위치: uspns.imxim.wwocn(Byte[] p0, Int32 p1, Int32 p2)
   위치: uspns.imxim.Read(Byte[] buffer, Int32 offset, Int32 count)
2021-03-16 11:06:58.396 DEBUG HttpRequest(8)[29] HTTP: Closing response stream.
2021-03-16 11:06:58.397 DEBUG HttpRequest(8)[29] HTTP: Closing HTTP session (4).
</pre>

---

Thanks! Our receiver apparently tries to fill the user-supplied buffer in the Stream.Read call, and fails when the connection is unexpectedly closed. We can tweak it to at least return the data received so far (and fix the internal ArgumentNullException as well). I'll send you a link to a hotfix to try when it's ready.

---

This has been addressed in Rebex HTTPS R5.2: https://www.rebex.net/https/history.aspx#R5.2