How to set TIdHTTPResponseInfo RawHeaders from a nother backend server?

Hi, I finally managed to create some kind of a load balancer and failover using indy TIdHttpServer.

I just have a problem for now is that i can't set the Headers from an THTTPClient object to TIdHTTPResponseInfo RawHeaders

This is my code for the On GET event

 

void __fastcall TMain::IdHTTPServer1CommandGet(TIdContext *AContext, TIdHTTPRequestInfo *ARequestInfo,
		  TIdHTTPResponseInfo *AResponseInfo)
{
unique_ptr<THTTPClient> BackendRequest(THTTPClient::Create());
BackendRequest->ConnectionTimeout = 1000;
BackendRequest->ResponseTimeout = 1000;

TNameValueArray Headers;
unique_ptr<TStringList> str(new TStringList);

if (ARequestInfo->RawHeaders->Count > 0)
	{
	ARequestInfo->RawHeaders->ConvertToStdValues(str.get());

	Headers.Length = ARequestInfo->RawHeaders->Count;

	TNameValuePair *H = &Headers[0];

	for (int i = 0; i < ARequestInfo->RawHeaders->Count; ++i, *H++)
		{
		H->Name = str->Names[i];
		H->Value = str->ValueFromIndex[i];
		}
	}

AResponseInfo->ContentStream = new TMemoryStream;
BackendRequest->CustomHeaders["X-Forwarded-For"] = AContext->Connection->Socket->Binding->PeerIP;
TStringDynArray PortsArr = SplitString(Ports->Text, ";");

String *Port = &PortsArr[0];
int Count = PortsArr.High;

ProxyRequest :
	{
	try
		{
		_di_IHTTPResponse Response = BackendRequest->Execute(ARequestInfo->Command,
								"http://localhost:" + *Port + ARequestInfo->URI,
								ARequestInfo->PostStream,
								AResponseInfo->ContentStream,
								Headers);

		AResponseInfo->CustomHeaders->Clear();
		
		forDynArray(Head, Response->Headers)
			{
			AResponseInfo->CustomHeaders->Values[(*Head).Name] = (*Head).Value;  //this causes the client to the error
			}

		AResponseInfo->ResponseNo = Response->StatusCode;
		AResponseInfo->ResponseText = Response->StatusText;
		}
	catch (ENetHTTPClientException &NetHttpEx)
		{
		*Port++;
		if (Port <= &PortsArr[Count])
			goto ProxyRequest;
		else
            throw Exception("All backend servers are down");
		}
	}
}
//---------------------------------------------------------------------------

I get "Invalid encoding name" error on the client side when I make a request to that server. I know that Indy clears the RawHeaders after the ObGet event is handled. and set its values from the actual properties like ContentType or ContentLength. But i was wondering if there is a workaround this issue to set the headers of the proxy server from the response headers of the backend server like i demonstrated in the code. 

Any help will be appreciated

Thanks in advance

Parents Reply
  • I already tried TIdHTTPProxyServer but it does require changing the internet proxy settings for the whole OS

    No, it doesn't. It doesn't know or care anything about the OS at all.  It is just a TCP server like any other.  But, from an HTTP perspective, it does require clients to understand that they are connected to a proxy in order to tailor their HTTP requests correctly, but you can setup proxy settings on a per-client basis, assuming client TCP apps actually support proxies in their setups.  You don't need to setup the proxy settings on a per-OS basis.

    As for the TCP server code what do you think is wrong with it.

    You are not reading the full HTTP message, you are reading only its headers but not its building data.  If there is any building data present, like in a POST request (which is the case in the screenshot you provided), you are not reading that data at all (you commented out that code - and also, String::ToInt() may be too small! You should use StrToInt64() instead).  And the "Content-Length" header is not the only thing you have to look for when dealing with building data, either.  Read RFC 21616 Section 4.4 and RFC 7230 Section 3.3.3 for details about that.  So using ReadStream() may not always be appropriate, at least not in a single call.  Bu, in your particular example shown, it will suffice.

    Also, when sending back a "Content-Length" to the client, you are using the raw byte count from Response->Text.BytesOf().Length, but then you are ignoring the bytes generated and instead sending the response data using Response->Lines instead, which is NOT guaranteed to produce the same (number of) bytes!  I would suggest saving the result of Response->Text.BytesOf() to a local variable, and then you can send the byte length and raw bytes as-is so they are always guaranteed to match (I really would not suggest using BytesOf() though, as it will lose data for non-ASCII characters - use TEncoding.UTF8.GetBytes(Response->Text) or IndyTextEncoding_UTF8.GetBytes(Response->Text) instead, and don't forget to include a "charset" attribute on the "Content-Type" header or else the client will assume ASCII).

    The client also sometime not always gives me this error 

    I can't answer that, since you did not show what the client is actually receiving when the error happens.

    I know that all of this is already done in http server but I don't want all of http server features on this one so i guess a tcp server will be better for me or more lightweight

    Why?  You are going to end up replicating much of what TIdHTTPServer already provides for you, so why not just use it?  HTTP is not a trivial protocol to implement from scratch, so you should save yourself some work.  I would suggest getting your project working with TIdHTTPServer first, let it handle the parsing for you.  And then after you get your logic working, then optimize later by switching to TIdTCPServer and manual parsing, if still needed.

Children