cURL - command line tool and library for transferring data with URLs

for webHost in www.example.com apache.org; do
	for scheme in http https; do
		url="${scheme}://$webHost"
		echo -e "$url"
		curl -sI "$url" | grep '^HTTP/1.1'
	done
	echo
done

http://www.example.com
HTTP/1.1 200 OK
https://www.example.com
HTTP/1.1 200 Connection established		different reason phrases

http://apache.org
HTTP/1.1 301 Moved Permanently
https://apache.org
HTTP/1.1 200 Connection established		different responses to HTTP / HTTPS requests

• different status codes
  • websites administrators are free to serve content via HTTP or HTTPS or both
  • when both HTTP and HTTPS are supported, it is a common practice that a request to an HTTP resource is responded with an HTTP 301 code, redirecting the requester to the HTTPS version of that resource
• different reason phrases
  • the reason phrase is optional in the response message (source)
  • reason phrases listed in the specification are only recommendations and can be replaced or removed (source)
  • reason phrases are made up by the HTTP server developers and can safely be ignored (source)
    HTTP/2 and HTTP/3 have no reason phrase

• HTTP status codes are not protocol-specific
• there is no reason phrase in HTTP/2, but if we if we inspect response headers more thoroughly :

  for url in https://www.example.com https://www.google.com https://www.wikipedia.com https://nginx.org; do
  	echo -e "\n$url"
  	curl -sI "$url" | grep '^HTTP' | tr ' ' '_'	this is to highlight [SPACE] characters
  done

  https://www.example.com
  HTTP/1.1_200_Connection_established
  HTTP/2_200_
  
  https://www.google.com
  HTTP/1.1_200_Connection_established
  HTTP/2_204_
  
  https://www.wikipedia.com
  HTTP/1.1_200_Connection_established
  HTTP/2_301_
  
  https://nginx.org
  HTTP/1.1_200_Connection_established
  HTTP/1.1_200_OK

  We can see that :
  • cURL reports 2 HTTP status messages for each host :
    • one for HTTP/1.1
    • and one for HTTP/2. If not available, it shows a different version of the HTTP/1.1 status message
  • HTTP/2 status messages have a trailing [SPACE] since their reason phrases are empty strings

• curl -Iv http://www.example.com/ 2>&1 | grep -i 'connection #0'
• curl -Iv -H 'Host: www.example.com' http://12.34.56.78/ 2>&1 | grep -i 'connection #0'

* Connection #0 to host www.example.com left intact
* Closing connection #0

the HTTP keep-alive is enabled

* Closing connection #0

the HTTP keep-alive is not enabled

* Connection #0 to host www.example.com left intact
* Re-using existing connection! (#0) with www.example.com
* Connection #0 to host www.example.com left intact

return page content to stdout

curl http://www.example.com

return HTTP response headers + page content to stdout

• curl -i http://www.example.com
• curl -i http://www.example.com | less

same as above (HTTP headers + page content), but written to outputFile rather than stdout

curl -i -o outputFile http://www.example.com

This only outputs a progress meter to stdout :

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1270  100  1270    0     0  75107      0 --:--:-- --:--:-- --:--:-- 79375

same as above : HTTP headers + page content written to outputFile, without the progress meter

• curl -i -o outputFile -s http://www.example.com
• curl -iso outputFile http://www.example.com

display the HTTP headers only (see also the wget method) :

• curl -I http://www.example.com
• curl -IXGET http://www.example.com
  Forces a GET HTTP request rather than the HEAD done with -I.
• curl -iso /dev/null -D /dev/stdout http://www.example.com

curl -iso /dev/null -D /dev/stdout -H "Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id" -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: old.example.com" http://example.com.edgesuite.net/

siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$siteName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Se+connecter&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt $baseUrl/wp-login && grep --color=auto "Tableau de bord" $resultFile

• Even though ./cookie.txt isn't visible after executing this command, it won't work without specifying it (?)
• So far, I've not been able to do the same with wget.
• ./index.html contains both the HTTP headers and the page contents.
• The form field names may change : log, password, wp-submit, as well as the submit button name : Se connecter.

serverName='srvXYZ'; siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$serverName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Connexion&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt --header "Host: $siteName" $baseUrl/wp-login.php

• Don't forget to post data to wp-login.php, some redirects may not work (?)

hostnameFake='my.productionSite.com'; hostnameReal='serverName'; resultFile='./result.txt'; cookieFile='./cookie.txt'; curl -iso $resultFile -D /dev/stdout --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginPage; curl -iso $resultFile -D /dev/stdout --data 'login=admin& password=secret' --cookie $cookieFile --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginCredentialsCheck; curl -iso $resultFile -D /dev/stdout --cookie $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/contentPage; grep --color=auto "some text" $resultFile

• The data to be sent through HTTP POST must not be urlencoded
• Albeit we're sending a tampered HTTP host header, no need to modify the cookie-jar file to translate the fake hostname into the real one. Just leave the cookie-jar file as-is.
• Depending on the website, it may/may not be necessary to :
  • GET the login page first, for session cookies
  • specify the HTTP Referrer when presenting credentials

Times :

1. DNS lookup time : to look up the IP address of the domain in question
2. Connect time : to set up the TCP connection
3. Wait time : to receive the first byte after the connection has been setup aka TTFB
4. Content download time

curl -s -o /dev/null -iD /dev/stdout -w "Effective URL :\t\t%{url_effective}\nContent-type :\t\t%{content_type}\nHTTP CODE :\t\t%{http_code}\nDNS lookup duration :\t%{time_namelookup} s\nConnect duration :\t%{time_connect} s\nTTFB :\t\t\t%{time_starttransfer} s\nTotal time :\t\t%{time_total} s\nDownloaded :\t\t%{size_download} bytes\n" http://www.example.com

Back to the origin + bypass host cache (Varnish) :

curl --insecure --ftp-ssl --ftp-pasv --user "login:password" "ftp://ftpHost:ftpPort/path/to/upload/directory/"

The trailing / looks mandatory (?).