How to set TIdHTTPResponseInfo RawHeaders from a nother backend server?

Hi, I finally managed to create some kind of a load balancer and failover using indy TIdHttpServer.

I just have a problem for now is that i can't set the Headers from an THTTPClient object to TIdHTTPResponseInfo RawHeaders

This is my code for the On GET event

 

void __fastcall TMain::IdHTTPServer1CommandGet(TIdContext *AContext, TIdHTTPRequestInfo *ARequestInfo,
		  TIdHTTPResponseInfo *AResponseInfo)
{
unique_ptr<THTTPClient> BackendRequest(THTTPClient::Create());
BackendRequest->ConnectionTimeout = 1000;
BackendRequest->ResponseTimeout = 1000;

TNameValueArray Headers;
unique_ptr<TStringList> str(new TStringList);

if (ARequestInfo->RawHeaders->Count > 0)
	{
	ARequestInfo->RawHeaders->ConvertToStdValues(str.get());

	Headers.Length = ARequestInfo->RawHeaders->Count;

	TNameValuePair *H = &Headers[0];

	for (int i = 0; i < ARequestInfo->RawHeaders->Count; ++i, *H++)
		{
		H->Name = str->Names[i];
		H->Value = str->ValueFromIndex[i];
		}
	}

AResponseInfo->ContentStream = new TMemoryStream;
BackendRequest->CustomHeaders["X-Forwarded-For"] = AContext->Connection->Socket->Binding->PeerIP;
TStringDynArray PortsArr = SplitString(Ports->Text, ";");

String *Port = &PortsArr[0];
int Count = PortsArr.High;

ProxyRequest :
	{
	try
		{
		_di_IHTTPResponse Response = BackendRequest->Execute(ARequestInfo->Command,
								"http://localhost:" + *Port + ARequestInfo->URI,
								ARequestInfo->PostStream,
								AResponseInfo->ContentStream,
								Headers);

		AResponseInfo->CustomHeaders->Clear();
		
		forDynArray(Head, Response->Headers)
			{
			AResponseInfo->CustomHeaders->Values[(*Head).Name] = (*Head).Value;  //this causes the client to the error
			}

		AResponseInfo->ResponseNo = Response->StatusCode;
		AResponseInfo->ResponseText = Response->StatusText;
		}
	catch (ENetHTTPClientException &NetHttpEx)
		{
		*Port++;
		if (Port <= &PortsArr[Count])
			goto ProxyRequest;
		else
            throw Exception("All backend servers are down");
		}
	}
}
//---------------------------------------------------------------------------

I get "Invalid encoding name" error on the client side when I make a request to that server. I know that Indy clears the RawHeaders after the ObGet event is handled. and set its values from the actual properties like ContentType or ContentLength. But i was wondering if there is a workaround this issue to set the headers of the proxy server from the response headers of the backend server like i demonstrated in the code. 

Any help will be appreciated

Thanks in advance

Parents Reply
  • The header sent by the HTTP serve is the default the one that is computed from WriteHeader method
    for example: Content-Type: text/html. etc. But that is only visible when i remove the lines of code that iterate for the backend response header.

    That is not what I asked you for.  I asked you for the ACTUAL headers that are being transmitted over the wire.  Sniff the network traffic with Wireshark, or hook up an Intercept component to the IOHandler, either way can capture the raw data going out on the wire.

    I tried to set the Response header one by one to see which one is causing the error on client side and i found it was "Content-Type". If i set it using the "AResponseInfo->ContentType" property it is ok and works fine but if i set it using this line:
    AResponseInfo->CustomHeaders->Values["Content-Type"] = "application/json";
    this causes the error "Invalid encoding name" on client side.

    You can't set the "Content-Type" header using the CustomHeaders property.  The CustomHeaders is merely appended as-is after the default headers that are generated by TIdHTTPResponseInfo.  The CustomHeaders do not replace any of the default headers.  So you end up with two "Content-Type" headers, the one generated by TIdHTTPResponseInfo, and the one received from the proxy.  Which, per the HTTP standard, will be concatenated together into a single value by a compliant HTTP receiver, thus producing an invalid result.

    So, to work around this, you can look at the name of every header received from the proxy, and for any name that corresponds to a header belonging to a TIdHTTPResponseInfo property, use that property, otherwise use the CustomHeaders instead, eg:

    AResponseInfo->CustomHeaders->Clear();
    		
    forDynArray(Head, Response->Headers)
        {
        if (TextIsSame((*Head).Name, "Content-Type"))
            AResponseInfo->ContentType = (*Head).Value;
        else if (...) // repeat of other properties as needed...
            ...
        else
            AResponseInfo->CustomHeaders->Values[(*Head).Name] = (*Head).Value;  //this causes the client to the error
        }

    I only managed to do it by tricking the response as if it was already written with this:

    That is another way to go, though I would suggest some small changes to the code you showed:

    //AResponseInfo->ResponseNo = Response->StatusCode;
    //AResponseInfo->ResponseText = Response->StatusText;
    AResponseInfo->HeaderHasBeenWritten = true;
    
    IOSocket->WriteBufferOpen();
    try
        {
        IOSocket->WriteLn("HTTP/1.1 " + IntToStr(Response->StatusCode) + " " + Response->StatusText);
    
        if (Response->Headers.Length > 0)
            {
            forDynArray(Head, Response->Headers)
                {
                //AResponseInfo->CustomHeaders->Values[(*Head).Name] = (*Head).Value;
                IOSocket->WriteLn((*Head).Name + ": " + (*Head).Value);
        		}
            }
    
        //IOSocket->Write(AResponseInfo->CustomHeaders);
        IOSocket->WriteLn();
        IOSocket->WriteBufferClose();
        }
    catch (const Exception &)
        {
        IOSocket->WriteBufferCancel();
        throw;
        }

    Also, I was wondering if i can do this in TCP server instead.

    Have you looked at TIdHTTPProxyServer yet?  It handles those kind of details for you, and gives you access to per-message headers and building data (NOTE: it doesn't currently handle "chunked" data if you need to access the building data.  But just acting as a forwarder, "checked" data should work, I think).

Children