cURL - command line tool and library for transferring data with URLs

> Under what circumstances will cURL print the following message:
> * additional stuff not fine transfer.c:1037: 0 0

That's a debug output I put there once to aid me debugging a transfer case I
had problems with and I then left it there. It is only present in debug builds
and basically if you need to ask about it, the message is not for you...

• curl -Iv http://www.example.com/ 2>&1 | grep -i 'connection #0'
• curl -Iv -H 'Host: www.example.com' http://12.34.56.78/ 2>&1 | grep -i 'connection #0'

* Connection #0 to host www.example.com left intact
* Closing connection #0

the HTTP keep-alive is enabled

* Closing connection #0

the HTTP keep-alive is not enabled

* Connection #0 to host www.example.com left intact
* Re-using existing connection! (#0) with www.example.com
* Connection #0 to host www.example.com left intact

curl http://12.34.56.78:443

curl: (52) Empty reply from server

curl https://12.34.56.78

curl: (35) Unknown SSL protocol error in connection to 12.34.56.78:443

When you want to use HTTPS (i.e. SSL / TLS over HTTP), you'll have to specify it in the scheme part of the URL, not with a trailing :443 (source) :

• DO : curl https://example.com
• DON'T : curl http://example.com:443

This is why commands using :443 fail : they're sending HTTP requests to the port 443 of the remote host, whereas this one is actually expecting HTTPS requests.

Let's have a look at this in more details :

1. you get a similar empty response, no data sent from server, or something like that with an other tool such as wget or the Web Developer Firefox extension
2. DNS resolution works as expected (and you're actually sending requests to the right server )
3. you're contacting the right virtualhost (if any). Have a look at the access.log / error.log
4. some process is actually listening on the specified IP+port
5. the network configuration (route, firewall, forward + reverse proxy, NAT, ...) actually leads requests to this listening socket.
   Don't hesitate to sketch the desired vs observed setup : the bug is on one of those arrows

return page content to stdout

curl http://www.example.com

return HTTP response headers + page content to stdout

• curl -i http://www.example.com
• curl -i http://www.example.com | less

same as above (HTTP headers + page content), but written to outputFile rather than stdout

curl -i -o outputFile http://www.example.com

This only outputs a progress meter to stdout :

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1270  100  1270    0     0  75107      0 --:--:-- --:--:-- --:--:-- 79375

same as above : HTTP headers + page content written to outputFile, without the progress meter

• curl -i -o outputFile -s http://www.example.com
• curl -iso outputFile http://www.example.com

display the HTTP headers only (see also the wget method) :

• curl -I http://www.example.com
• curl -IXGET http://www.example.com
  Forces a GET HTTP request rather than the HEAD done with -I.
• curl -iso /dev/null -D /dev/stdout http://www.example.com

curl -iso /dev/null -D /dev/stdout -H "Pragma: akamai-x-get-cache-key, akamai-x-cache-on, akamai-x-cache-remote-on, akamai-x-get-true-cache-key, akamai-x-get-extracted-values, akamai-x-check-cacheable, akamai-x-get-request-id, akamai-x-serial-no, akamai-x-get-ssl-client-session-id" -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: www.example.com" http://12.34.56.78/

akcurl -iso /dev/null -D /dev/stdout -H "Host: old.example.com" http://example.com.edgesuite.net/

siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$siteName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Se+connecter&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt $baseUrl/wp-login && grep --color=auto "Tableau de bord" $resultFile

• Even though ./cookie.txt isn't visible after executing this command, it won't work without specifying it (?)
• So far, I've not been able to do the same with wget.
• ./index.html contains both the HTTP headers and the page contents.
• The form field names may change : log, password, wp-submit, as well as the submit button name : Se connecter.

serverName='srvXYZ'; siteName='www.example.com'; login='username'; password='password'; baseUrl='http://'$serverName; resultFile='./index.html'; redirectTo='http%3A%2F%2F'$siteName'%2Fwp-admin%2F'; curl -L -iso $resultFile -D /dev/stdout --data "log=$login&pwd=$password&wp-submit=Connexion&redirect_to=$redirectTo&testcookie=1" --cookie ./cookie.txt --header "Host: $siteName" $baseUrl/wp-login.php

• Don't forget to post data to wp-login.php, some redirects may not work (?)

hostnameFake='my.productionSite.com'; hostnameReal='serverName'; resultFile='./result.txt'; cookieFile='./cookie.txt'; curl -iso $resultFile -D /dev/stdout --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginPage; curl -iso $resultFile -D /dev/stdout --data 'login=admin& password=secret' --cookie $cookieFile --cookie-jar $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/loginCredentialsCheck; curl -iso $resultFile -D /dev/stdout --cookie $cookieFile -H "Host: $hostnameFake" http://$hostnameReal/contentPage; grep --color=auto "some text" $resultFile

• The data to be sent through HTTP POST must not be urlencoded
• Albeit we're sending a tampered HTTP host header, no need to modify the cookie-jar file to translate the fake hostname into the real one. Just leave the cookie-jar file as-is.
• Depending on the website, it may/may not be necessary to :
  • GET the login page first, for session cookies
  • specify the HTTP Referrer when presenting credentials

Times :

1. DNS lookup time : to look up the IP address of the domain in question
2. Connect time : to set up the TCP connection
3. Wait time : to receive the first byte after the connection has been setup aka TTFB
4. Content download time

curl -s -o /dev/null -iD /dev/stdout -w "Effective URL :\t\t%{url_effective}\nContent-type :\t\t%{content_type}\nHTTP CODE :\t\t%{http_code}\nDNS lookup duration :\t%{time_namelookup} s\nConnect duration :\t%{time_connect} s\nTTFB :\t\t\t%{time_starttransfer} s\nTotal time :\t\t%{time_total} s\nDownloaded :\t\t%{size_download} bytes\n" http://www.example.com

Back to the origin + bypass host cache (Varnish) :

curl --insecure --ftp-ssl --ftp-pasv --user "login:password" "ftp://ftpHost:ftpPort/path/to/upload/directory/"

The trailing / looks mandatory (?).