Https "Unexpected data appeared after end of response" Error

Question

I'm curious when and why the "Unexpected data appeared after end of response" error occurs.

When downloading files, this error occurs quite occasionally.
However, we have not yet found the cause of it.

It seems to be due to more read requests than the given Content-Length.

It is common to return 0 if there is no data when requesting Read.

How do I handle this exception?

I am using the latest fixed version (5.0.7733.0) that you sent me.

Thank you.

Best regards.

BooKyung Oh.
OpenBoxLab Inc.

Answer 1

This error occurs when the server sends more data that it was expected to. A mismatch between the number of bytes reported in Content-Length and actual number of bytes received is the most common cause.

We choose to fail in this case because silently ignoring the problem could lead to data corruption. Sending more data than announced indicates a serious server-side issue that should not be hidden from the caller.

Comments

I understood well.

In this case, if I read data as much as Content-Length size, does not an exception occur?

Thank you.

---

I don't know the details inside Http yet, so there's always something to worry about.
When I use it in KeepAlive mode
Doesn't there conflict with the previous data when I cancel the download and download a new one?

---

Well. That's a little weird :
Http Request : 1105
Http Response : Content-length : 1105 (OK)
Read Stream Start :  stream.read(buffer, 0, 1105)
Exception (Unexpected data..)

stack trace :
uspns.imxim.Read(Byte[] buffer, Int32 offset, Int32 count)
uspns.dhhhe.twrde(Byte[] p0, Int32 p1, Int32 p2)
uspns.dhhhe.pewak[T](Func`1 p0)
RaiDrive.Service.ThrottledStream.Read(Byte[] buffer, Int32 offset, Int32 count)

---

There are two cases in my download code.
I always use KeepAlive.
One is when Http is aborted by using Cancellation(request.abort call),
the other is when I get enough data while downloading and close Response and Response stream without reading the rest.
Then it continues to download the next file or another location of the same file.
These tasks are operated in multi-thread.
If I don't properly call request abort, wouldn't it be possible for this to happen?
Is it normal to have to abort request all the time when I haven't read the data completely?

---

The error will occur even if you only read as much data as indicated by Content-Length.

But it should not affect other downloads - when an error occurs, the HTTP session is discarded and not reused for subsequent requests.

This also applies to HTTP sessions from which not all data has been read - these are discarded and not reused. In this case, it might be better to actually read the rest of the data as well and discard it, because that would make it possible to reuse the session.

As usual, a communication log at LogLevel.Debug might make it possible to tell what is actually going on.

---

I sent a log file.

---

Thanks! The log confirms that the server really sent more data than it should. So this is a server-side bug.

We will modify Rebex HTTPS to return the data up to the number of bytes of Content-Length, and only fail in the next Stream.Read call.

---

I still get the error when the server sends more data than Content-Length.

In Rebex R6.0, "Unexpected data appeared after end of response." occurs when Content-Length is 1105 and stream.Read(buffer, 0, 1105) is performed.

You said you would fix it, but when I read exactly Content-Length I get an error.

There is no problem until 1104 bytes are read, but it happens when the last 1 byte is read.

Reading 1105 bytes at a time, or even small multiple reads, throws an error when reading the last byte.

.Net HTTP returns as much as I requested to read, and it doesn't throw an error.

Happy new year!.

---

Happy new year!

I have analyzed the issue and I found this:
- In R5.5 we modified the read behavior as you requested to not fail if you read exact Content-Length.
- However, the modified behavior caused an issue with caching HTTP sessions, which we fixed in R5.7. Unfortunately, our tests did not detect that the previously modified read behavior is not working as requested.

I have fixed the tests. I have also modified the read behavior to work as requested and still cache HTTP sessions if possible.

The change will be in the next public release. Please let me know, if you want to receive beta version to test it yourself.

---

I want to get a beta version.
thank you.

---

I just sent a link to the latest build to your e-mail address. Please give it a try and let us know whether it resolves the issue.

---

R6.1 RC1 test version works correctly.
Thank you.

---

Thanks for testing. R6.1 has been published: https://www.rebex.net/total-pack/history.aspx#R6.1