View definition


Defined in


Fetch requests the specified URL using the given user agent string. It uses a custom http Client instance that doesn't follow redirections. Instead, the redirected-to URL is enqueued so that it goes through the same Filter and Fetch process as any other URL.

Two options were considered for the default Fetch implementation :

1- Not following any redirections, and enqueuing the redirect-to URL,
failing the current call with the 3xx status code.

2- Following all redirections, enqueuing only the last one (where redirection
stops). Returning the response of the next-to-last request.

Ultimately, 1) was implemented, as it is the most generic solution that makes sense as default for the library. It involves no "magic" and gives full control as to what can happen, with the disadvantage of having the Filter being aware of all possible intermediary URLs before reaching the final destination of a redirection (i.e. if A redirects to B that redirects to C, Filter has to allow A, B, and C to be Fetched, while solution 2 would only have required Filter to allow A and C).

Solution 2) also has the disadvantage of fetching twice the final URL (once while processing the original URL, so that it knows that there is no more redirection HTTP code, and another time when the actual destination URL is fetched to be visited).

Fetch is referenced in 0 repositories