Download Source Code: WebRequestSpy.zip - 15.03KB
 |  |  |
| Create a Web Request Spy, a simple UI framework for transparent and interactive HTTP requests. Tutorial on client-side and ASP.NET server-side Framework classes for HTTP requests. WebUtilities class for transparent web requests. Automatic redirections. | ||
 | |  |
Overview
HTTP (HyperText Transfer Protocol) is the main TCP/IP protocol used to return HTML and other kind of content, as web pages. While TCP/IP uses ASCII for any data transfer, HTTP commands are sent back and forth as concatenations of "name: value" substring, each name-value pair being known as a "header". Additionally, a client can send content as POST data with a request, and get back a content string from a GET method's response. There is also some other information associated with this round-trip or handshake between a client and a web server, like HTTP's protocol version, the returned HTTP status code, the address of the page you are looking for.
In your applications, you may need to issue and process HTTP requests at different levels. In some cases, sending the URL of the page you are looking for, and getting back its text content, it's all you need. You don't want to handle intermediate steps and details of the HTTP protocol. We might say you need a transparent HTTP request.
In other cases, you may need more control and information, about the data exchanged and the actual steps required to perform the operation. Being able to step through the executed code or on a more friendly interface, makes this HTTP request kind of interactive.
.NET Framework Classes offer rich support for processing HTTP web requests. Depending on the degree of detail and control you may want to have over the HTTP data transfer, you can use, on the client-side, classes defined at three different levels.
In Microsoft's .NET Framework, client-side web request functionality is implemented in System.Net namespace, as part of the generic network support, while ASP.NET server-side functionality is mostly implemented in System.Web namespace. Standalone .NET applications using server-side functionality - such as the HttpUtility class for its static encode-decode methods - usually need to explicitly reference the System.Web.dll assembly, not just the namespace.
Transparent Requests with WebClient
WebClient offers a high level of abstraction. For synchronous transfers, it's enough to call one of the Download methods, based on what type of data you want in return. For text-based content, WebUtilities.WebRequestClient implements one of the shortest sequence you can use to get the content of a web page. WebClient is a disposable class, so using will make sure the instance will be properly closed:
/// <summary>
/// High-level transparent HTTP transfer, with WebClient class
/// </summary>
/// <param name="url">web address</param>
/// <returns>response stream</returns>
public static string WebRequestClient(string url)
{
using (WebClient webClient = new WebClient())
return webClient.DownloadString(url);
}From our downloadable project, Web Request Spy, call Transparent Requests - Using WebClient menu command, for the default google.com URL, and the response stream box will be filled with the source code of the same page you can actually get and show, in a more friendly manner, with your browser.
Transparent HttpWebRequest
An intermediate level of abstraction is handled by HttpWebRequest and HttpWebResponse classes. Based and obtained from the abstract classes WebRequest and WebResponse, here is how they can be used, in three easy steps, to get a content string passing a URL. Remark the necessary explicit casts to concrete instance types:
/// <summary>
/// Typical transparent HTTP transfer,
/// with HttpWebRequest/Response classes
/// </summary>
/// <param name="url">web address</param>
/// <returns>response stream</returns>
public static string WebRequestHttp(string url)
{
// (1) create request
HttpWebRequest request
= (HttpWebRequest)HttpWebRequest.Create(url);
// (2) get response
using (HttpWebResponse response
= (HttpWebResponse)request.GetResponse())
{
// (3) get response stream
using (StreamReader reader = new StreamReader(
response.GetResponseStream(), Encoding.Default))
{
return reader.ReadToEnd();
}
}
}From Web Request Spy, call Transparent Requests - Using HttpWebClient menu command, for the default google.com URL, to issue a transparent HTTP request based on these classes. The response stream box will be filled with the same source code as for previous case, using WebClient.
Interactive Requests
In Web Request Spy, we also presented these three steps at the UI level, one by one, through interactive HTTP web requests. After you enter a URL, use, in order, the first three main menu commands - Create Request, Get Response and Get Response Stream, to completely perform a full HTTP data transfer.
After each step, you can check (and change, if you want) intermediate property values for the Uri, HttpWebRequest and HttpWebResponse involved objects. The URI is an encapsulation or the URL in a System.Uri object. The response stream is returned and dumped, in raw format, in a multiline text box. Here is the separate implementation of these three steps, in MainForm class:
/// <summary>
/// (1) Create HTTP Web Request
/// </summary>
private HttpWebRequest _request = null;
private void mnuCreateRequest_Click(object sender, EventArgs e)
{
Debug.Assert(txtUrl.Text.Length > 0);
Reset();
// Show Uri object properties
propsUri.SelectedObject = _uri = new Uri(txtUrl.Text);
// Show HttpWebRequest object properties
propsRequest.SelectedObject = _request
= (HttpWebRequest)HttpWebRequest.Create(_uri);
mnuGetResponse.Enabled = true;
}
/// <summary>
/// (2) Get HTTP Web Response
/// </summary>
private HttpWebResponse _response = null;
private void mnuGetResponse_Click(object sender, EventArgs e)
{
Debug.Assert(_request != null);
txtContent.Text = "";
Debug.WriteLine(">>> Request Headers:");
WebUtilities.Dump(_request.Headers);
// Show HttpWebResponse object properties
propsResponse.SelectedObject = _response
= (HttpWebResponse)_request.GetResponse();
Debug.WriteLine(">>> Response Headers:");
WebUtilities.Dump(_response.Headers);
mnuGetResponse.Enabled = false;
mnuGetResponseStream.Enabled = true;
}
/// <summary>
/// (3) Get the HTTP Web Response Data
/// </summary>
/// <param name="sender"></param>
/// <param name="e"></param>
private void mnuGetResponseStream_Click(object sender, EventArgs e)
{
Debug.Assert(_uri != null
&& _request != null && _response != null);
txtContent.Text = "";
using (StreamReader reader = new StreamReader(
_response.GetResponseStream(), Encoding.Default))
txtContent.Text = reader.ReadToEnd();
// Show Uri/HttpWebRequest/HttpWebResponse object properties
mnuGetResponseStream.Enabled = false;
propsUri.SelectedObject = _uri;
propsRequest.SelectedObject = _request;
propsResponse.SelectedObject = _response;
}We needed to dump the content of name-value pairs of the request and response Headers in the Debug window, because the PropertyGrid control does not show dictionary values, just keys.
Here are some possible response headers, for google.com page:
Transfer-Encoding=chunked
Cache-Control=private
Content-Type=text/html; charset=ISO-8859-1
Date=Sun, 22 Apr 2007 15:52:53 GMT
Set-Cookie=PREF=ID=a1ad9aa25133b430:TM=1177257173:LM=1177257173:S=YVVAG4F28XMYonIT; expires=Sun, 17-Jan-2038 19:14:07 GMT; path=/; domain=.google.ca
Server=GWS/2.1
Subscribe and Share: 
