cURL - command line tool and library for transferring data with URLs

for webHost in www.example.com apache.org; do
	for scheme in http https; do
		url="${scheme}://$webHost"
		echo -e "$url"
		curl -sI "$url" | grep '^HTTP/1.1'
	done
	echo
done

http://www.example.com
HTTP/1.1 200 OK
https://www.example.com
HTTP/1.1 200 Connection established		different reason phrases

http://apache.org
HTTP/1.1 301 Moved Permanently
https://apache.org
HTTP/1.1 200 Connection established		different responses to HTTP / HTTPS requests

• different status codes
  • websites administrators are free to serve content via HTTP or HTTPS or both
  • when both HTTP and HTTPS are supported, it is a common practice that a request to an HTTP resource is responded with an HTTP 301 code, redirecting the requester to the HTTPS version of that resource
• different reason phrases
  • the reason phrase is optional in the response message (source)
  • reason phrases listed in the specification are only recommendations and can be replaced or removed (source)
  • reason phrases are made up by the HTTP server developers and can safely be ignored (source)
    HTTP/2 and HTTP/3 have no reason phrase

• HTTP status codes are not protocol-specific
• there is no reason phrase in HTTP/2, but if we if we inspect response headers more thoroughly :

  for url in https://www.example.com https://www.google.com https://www.wikipedia.com https://nginx.org; do
  	echo -e "\n$url"
  	curl -sI "$url" | grep '^HTTP' | tr ' ' '_'	this is to highlight [SPACE] characters
  done

  https://www.example.com
  HTTP/1.1_200_Connection_established
  HTTP/2_200_
  
  https://www.google.com
  HTTP/1.1_200_Connection_established
  HTTP/2_204_
  
  https://www.wikipedia.com
  HTTP/1.1_200_Connection_established
  HTTP/2_301_
  
  https://nginx.org
  HTTP/1.1_200_Connection_established
  HTTP/1.1_200_OK

  We can see that :
  • cURL reports 2 HTTP status messages for each host :
    • one for HTTP/1.1
    • and one for HTTP/2. If not available, it shows a different version of the HTTP/1.1 status message
  • HTTP/2 status messages have a trailing [SPACE] since their reason phrases are empty strings

• curl -Iv http://www.example.com/ 2>&1 | grep -i 'connection #0'
• curl -Iv -H 'Host: www.example.com' http://12.34.56.78/ 2>&1 | grep -i 'connection #0'

* Connection #0 to host www.example.com left intact
* Closing connection #0

the HTTP keep-alive is enabled

* Closing connection #0

the HTTP keep-alive is not enabled

* Connection #0 to host www.example.com left intact
* Re-using existing connection! (#0) with www.example.com
* Connection #0 to host www.example.com left intact

return page content to stdout

curl http://www.example.com

return HTTP response headers + page content to stdout

• curl -i http://www.example.com
• curl -i http://www.example.com | less

same as above (HTTP headers + page content), but written to outputFile rather than stdout

curl -i -o outputFile http://www.example.com

This only outputs a progress meter to stdout :

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1270  100  1270    0     0  75107      0 --:--:-- --:--:-- --:--:-- 79375

same as above : HTTP headers + page content written to outputFile, without the progress meter

• curl -i -o outputFile -s http://www.example.com
• curl -iso outputFile http://www.example.com

display the HTTP headers only (see also the wget method) :

• curl -I http://www.example.com
• curl -IXGET http://www.example.com
  Forces a GET HTTP request rather than the HEAD done with -I.
• curl -iso /dev/null -D /dev/stdout http://www.example.com

curl -iso /dev/null -D /dev/stdout -H "Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id" -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: old.example.com" http://example.com.edgesuite.net/

siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$siteName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Se+connecter&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt $baseUrl/wp-login && grep --color=auto "Tableau de bord" $resultFile

• Even though ./cookie.txt isn't visible after executing this command, it won't work without specifying it (?)
• So far, I've not been able to do the same with wget.
• ./index.html contains both the HTTP headers and the page contents.
• The form field names may change : log, password, wp-submit, as well as the submit button name : Se connecter.

serverName='srvXYZ'; siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$serverName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Connexion&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt --header "Host: $siteName" $baseUrl/wp-login.php

• Don't forget to post data to wp-login.php, some redirects may not work (?)

hostnameFake='my.productionSite.com'; hostnameReal='serverName'; resultFile='./result.txt'; cookieFile='./cookie.txt'; curl -iso $resultFile -D /dev/stdout --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginPage; curl -iso $resultFile -D /dev/stdout --data 'login=admin& password=secret' --cookie $cookieFile --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginCredentialsCheck; curl -iso $resultFile -D /dev/stdout --cookie $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/contentPage; grep --color=auto "some text" $resultFile

• The data to be sent through HTTP POST must not be urlencoded
• Albeit we're sending a tampered HTTP host header, no need to modify the cookie-jar file to translate the fake hostname into the real one. Just leave the cookie-jar file as-is.
• Depending on the website, it may/may not be necessary to :
  • GET the login page first, for session cookies
  • specify the HTTP Referrer when presenting credentials

Times :

1. DNS lookup time : to look up the IP address of the domain in question
2. Connect time : to set up the TCP connection
3. Wait time : to receive the first byte after the connection has been setup aka TTFB
4. Content download time

curl -s -o /dev/null -iD /dev/stdout -w "Effective URL :\t\t%{url_effective}\nContent-type :\t\t%{content_type}\nHTTP CODE :\t\t%{http_code}\nDNS lookup duration :\t%{time_namelookup} s\nConnect duration :\t%{time_connect} s\nTTFB :\t\t\t%{time_starttransfer} s\nTotal time :\t\t%{time_total} s\nDownloaded :\t\t%{size_download} bytes\n" http://www.example.com

Back to the origin + bypass host cache (Varnish) :

curl --insecure --ftp-ssl --ftp-pasv --user "login:password" "ftp://ftpHost:ftpPort/path/to/upload/directory/"

The trailing / looks mandatory (?).

fieldSeparator='|'; for curlOption in '-s' '-s -f'; do
	echo -e "\ncURL option : '$curlOption'"
	for httpStatusCode in 200 204 301 302 304 400 401 403 404 410 501 503; do
		echo -ne "$httpStatusCode$fieldSeparator\""
		curl $curlOption "https://httpstat.us/$httpStatusCode" && answer='\e[1;32mOK\e[0m' || answer='\e[1;31mKO\e[0m'
		echo -e "\"$fieldSeparator$answer"
	done | column -s "$fieldSeparator" -t
done

cURL option : '-s'
200	"200 OK"			OK
204	""				OK
301	"301 Moved Permanently"		OK
302	"302 Found"			OK
304	""				OK
400	"400 Bad Request"		OK
401	"401 Unauthorized"		OK
403	"403 Forbidden"			OK
404	"404 Not Found"			OK
410	"410 Gone"			OK
501	"501 Not Implemented"		OK
503	"503 Service Unavailable"	OK

cURL option : '-s -f'
200	"200 OK"			OK
204	""				OK
301	"301 Moved Permanently"		OK
302	"302 Found"			OK
304	""				OK
400	""				KO
401	""				KO
403	""				KO
404	""				KO
410	""				KO
501	""				KO
503	""				KO