cURL

cURL is a computer software project providing a library and command-line tool for transferring data using various protocols. The cURL project produces two products, libcurl and cURL. It was first released in 1997.

curl is a command line tool for transferring data with URL syntax, supporting DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet and TFTP. curl supports SSL certificates, HTTP POST, HTTP PUT, FTP uploading, HTTP form based upload, proxies, cookies, user+password authentication (Basic, Digest, NTLM, Negotiate, kerberos…), file transfer resume, proxy tunneling and a busload of other useful tricks.

Working with HTTP from the command-line is a valuable skill for HTTP architects and API designers to have. The cURL library and curl command give you the ability to design a Request, put it on the pipe, and explore the Response. The downside to the power of curl is how much breadth its options cover. Running curl --help spits out 150 different flags and options. This article demonstrates nine basic, real-world applications of curl.

In this tutorial we’ll use the httpkit echo service as our end point. The echo server’s Response is a JSON representation of the HTTP request it receives.

Make a Request

Let’s start with the simplest curl command possible.

Request
curl http://echo.httpkit.com
Response
{
  "method": "GET",
  "uri": "/",
  "path": {
    "name": "/",
    "query": "",
    "params": {}
  },
  "headers": {
    "host": "echo.httpkit.com",
    "user-agent": "curl/7.24.0 ...",
    "accept": "*/*"
  },
  "body": null,
  "ip": "28.169.144.35",
  "powered-by": "http://httpkit.com",
  "docs": "http://httpkit.com/echo"
}

Just like that we have used curl to make an HTTP Request. The method, or “verb”, curl uses, by default, is GET. The resource, or “noun”, we are requestion is addressed by the URL pointing to the httpkit echo service, http://echo.httpkit.com.

You can add path and query string parameters right to the URL.

Request
curl http://echo.httpkit.com/path?query=string
Response
{ ...
  "uri": "/path?query=string",
  "path": {
    "name": "/path",
    "query": "?query=string",
    "params": {
      "query": "string"
    }
  }, ...
}

Set the Request Method

The curl default HTTP method, GET, can be set to any method you would like using the -X option. The usual suspects POST, PUT, DELETE, and even custom methods, can be specified.

Request
curl -X POST echo.httpkit.com
Response
{
    "method": "POST",
    ...
}

As you can see, the http:// protocol prefix can be dropped with curl because it is assumed by default. Let’s give DELETE a try, too.

Request
curl -X DELETE echo.httpkit.com
Response
{
    "method": "DELETE",
    ...
}

Set Request Headers

Request headers allow clients to provide servers with meta information about things such as authorization, capabilities, and body content-type. OAuth2 uses an Authorization header to pass access tokens, for example. Custom headers are set in curl using the -H option.

Request
curl -H "Authorization: OAuth 2c4419d1aabeec" 
     http://echo.httpkit.com
Response
{...
"headers": {
    "host": "echo.httpkit.com",
    "authorization": "OAuth 2c4419d1aabeec",
  ...},
...}

Multiple headers can be set by using the -H option multiple times.

Request
curl -H "Accept: application/json" 
     -H "Authorization: OAuth 2c3455d1aeffc" 
     http://echo.httpkit.com
Response
{ ...
  "headers": { ...
    "host": "echo.httpkit.com",
    "accept": "application/json",
    "authorization": "OAuth 2c3455d1aeffc" 
   }, ...
}

Send a Request Body

Many popular HTTP APIs today POST and PUT resources using application/json or application/xml rather than in an HTML form data. Let’s try PUTing some JSON data to the server.

Request
curl -X PUT 
     -H 'Content-Type: application/json' 
     -d '{"firstName":"Kris", "lastName":"Jordan"}'
     echo.httpkit.com
Response
{
   "method": "PUT", ...
   "headers": { ...
     "content-type": "application/json",
     "content-length": "40"
   },
   "body": "{"firstName":"Kris","lastName":"Jordan"}",
   ...
 }

Use a File as a Request Body

Escaping JSON/XML at the command line can be a pain and sometimes the body payloads are large files. Luckily, cURL’s @readfile macro makes it easy to read in the contents of a file. If we had the above example’s JSON in a file named “example.json” we could have run it like this, instead:

Request
curl -X PUT 
     -H 'Content-Type: application/json' 
     -d @example.json
     echo.httpkit.com

POST HTML Form Data

Being able to set a custom method, like POST, is of little use if we can’t also send a request body with data. Perhaps we are testing the submission of an HTML form. Using the -d option we can specify URL encoded field names and values.

Request
curl -d "firstName=Kris" 
     -d "lastName=Jordan" 
     echo.httpkit.com
Response
{
  "method": "POST", ...
  "headers": {
    "content-length": "30",
    "content-type":"application/x-www-form-urlencoded"
  },
  "body": "firstName=Kris&lastName=Jordan", ...
}

Notice the method is POST even though we did not specify it. When curl sees form field data it assumes POST. You can override the method using the -X flag discussed above. The “Content-Type” header is also automatically set to “application/x-www-form-urlencoded” so that the web server knows how to parse the content. Finally, the request body is composed by URL encoding each of the form fields.

POST HTML Multipart / File Forms

What about HTML forms with file uploads? As you know from writing HTML file upload form, these use a multipart/form-data Content-Type, with the enctype attribute in HTML. In cURL we can pair the -F option and the @readFile macro covered above.

Request
curl -F "firstName=Kris" 
     -F "publicKey=@idrsa.pub;type=text/plain" 
     echo.httpkit.com
Response
{
  "method": "POST",
  ...
  "headers": {
    "content-length": "697",
    "content-type": "multipart/form-data;
    boundary=----------------------------488327019409",
    ... },
  "body": "------------------------------488327019409rn
           Content-Disposition: form-data;
           name="firstName"rnrn
           Krisrn
           ------------------------------488327019409rn
           Content-Disposition: form-data;
           name="publicKey";
           filename="id_rsa.pub"rn
           Content-Type: text/plainrnrn
           ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAkq1lZYUOJH2
           ... more [a-zA-Z0-9]* ...
           naZXJw== krisjordan@gmail.comnrn
           ------------------------------488327019409
           --rn",
...}

Like with the -d flag, when using -F curl will automatically default to the POST method, the multipart/form-data content-type header, calculate length, and compose the multipart body for you. Notice how the @readFile macro will read the contents of a file into any string, it’s not just a standalone operator. The “;text/plain” specifies the MIME content-type of the file. Left unspecified, curl will attempt to sniff the content-type for you.

Test Virtual Hosts, Avoid DNS

Testing a virtual host or a caching proxy before modifying DNS and without overriding hosts is useful on occassion. With cURL just point the request at your host’s IP address and override the default Host header cURL sets up.

Request
curl -H "Host: google.com" 50.112.251.120
Response
{
  "method": "GET", ...
  "headers": {
    "host": "google.com", ...
  }, ...
}

View Response Headers

APIs are increasingly making use of response headers to provide information on authorization, rate limiting, caching, etc. With cURL you can view the headers and the body using the -i flag.

Request
curl -i echo.httpkit.com 
Response
HTTP/1.1 200 OK
Server: nginx/1.1.19
Date: Wed, 29 Aug 2012 04:18:19 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 391
Connection: keep-alive
X-Powered-By: http://httpkit.com

{
  "method": "GET",
  "uri": "/", ...
}

Shameless plug: Do you hack on REST API integrations or implementations? Wiretap is an HTTP debugger you can use to see every request and response between any client and HTTP API in real time. It’s entering private beta soon. Help test it!

on an Ubuntu system (probably Debian too)

$ sudo apt-get install php5-curl

The basic idea behind the cURL functions is that you initialize a cURL session using the curl_init(), then you can set all your options for the transfer via the curl_setopt(), then you can execute the session with the curl_exec() and then you finish off your session using the curl_close(). Here is an example that uses the cURL functions to fetch the example.com homepage into a file:

<?php

$ch = curl_init("http://example.iana.org/");
$fp = fopen("example_homepage.txt", "w");

curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, 0);

curl_exec($ch);
curl_close($ch);
fclose($fp);
?>

Disk utilities

EaseUS Disk Copy Home is a free disk/partition clone software for home users only. Regardless of your operating system, file system and partition scheme, through creating a bootable CD it can sector-by-sector copy you disk to assure you a 100% identical copy of the original one. It is a perfect free tool for Data Recovery Wizard to recover files from a backup disk.

EaseUS Disk Copy makes it utterly simple to create a bootable disk for your system on a CD or DVD, USB drive, or ISO image file, and use it to copy or clone disk partitions and recover data and partitions from backups, including sector-by-sector copying for total compatibility. With it, you can perform disk operations that usually require more than one drive (even more than one computer), such as recovering a backup of your main drive.

EaseUS Disk Copy is fully portable, so it runs as soon as you click its program file without having to be installed, even from a USB drive or similar device. The program’s disk wizard is a simple dialog box with three choices for creating a bootable drive, with drop-down lists for multiple destinations: USB, CD/DVD, and Export ISO (you browse to select a destination for an ISO file for further use). We inserted a blank DVD-R into our disk tray, and EaseUS Disk Copy’s built-in burning software recognized it. We selected CD/DVD and pressed Proceed. Immediately the software began analyzing our system and burning our bootable drive. The whole process was finished quickly. We removed the disk and labeled it, since a bootable disk you can’t find or identify doesn’t help much when your system is kaput. We reinserted the disk, rebooted out system, accessed the boot menu, and selected CD-ROM. As it should, our system booted to EaseUS Disk Copy’s menu.

At this point we could choose to continue into Disk Copy, boot from the first hard drive, or select an additional partition to boot from (handy for multi-OS systems). We selected Disk Copy, and the program’s disk copying and cloning wizard opened. This wizard walked us through each step of choosing a disk or partition as well as operations and options. The sector-by-sector option takes more time and uses more space, since it creates a one-for-one clone of your disk.

For a simple, free way to create bootable disks to use with backups and to copy your hard drives and partitions, it’s hard to do better than EaseUS Disk Copy.

Read more: EaseUS Disk Copy Home Edition – CNET Download.com http://download.cnet.com/EaseUS-Disk-Copy-Home-Edition/3000-2242_4-10867157.html#ixzz2UcWTJqM0


G4L is a hard disk and partition imaging and cloning tool. The created images are optionally compressed and transferred to an FTP server or cloned locally. CIFS(Windows), SSHFS and NFS support included, and udpcast and fsarchiver options. .
.
GPT partition support was added in version 0.41.

Backing up Windows partitions requires the use of a bootable G4L CD or running g4l via grub4dos..

G4L Web Site›


Clonezilla is a partition and disk imaging/cloning program similar to Norton Ghost®. It saves and restores only used blocks in hard drive. Two types of Clonezilla are available, Clonezilla live and Clonezilla SE (Server Edition).


Darik’s Boot and Nuke (DBAN) is free erasure software designed for consumer use. DBAN users should be aware of some product limitations, including:
•No guarantee that data is removed
•Limited hardware support (e.g. no RAID dismantling)
•No customer support

DBAN is a self-contained boot disk that automatically deletes the contents of any hard disk that it can detect. This method can help prevent identity theft before recycling a computer. It is also a solution commonly used to remove viruses and spyware from Microsoft Windows installations. DBAN prevents all known techniques of hard disk forensic analysis. It does not provide users with a proof of erasure, such as an audit-ready erasure report.

Professional data erasure tools are recommended for company and organizational users. For secure data erasure with audit-ready reporting, contact Blancco or download a free evaluation license.


Unlocker Portable 1.9.0

File eraser,a freeware to delete stubborn files easily, kill stubborn files.

 

  • Ever had such an annoying message given by Windows?

It has many other flavors:

Cannot delete file: Access is denied
There has been a sharing violation.
The source or destination file may be in use.
The file is in use by another program or user.
Make sure the disk is not full or write-protected and that the file is not currently in use.