Structure of HTTP Transactions

Like most network protocols, HTTP uses the client-server model: An HTTP client opens a connection and sends a request message to an HTTP server; the server then returns a response message, usually containing the resource that was requested. After delivering the response, the server closes the connection (making HTTP a stateless protocol, i.e. not maintaining any connection information between transactions).

The format of the request and response messages are similar, and English-oriented. Both kinds of messages consist of:

an initial line,
zero or more header lines,
a blank line (i.e. a CRLF by itself), and
an optional message body (e.g. a file, or query data, or query output).

Put another way, the format of an HTTP message is:

<initial line, different for request vs. response>
Header1: value1
Header2: value2
Header3: value3

<optional message body goes here, like file contents or query data;
 it can be many lines long, or even binary data $&*%@!^$@>

Initial Request Line

The initial line is different for the request than for the response. A request line has three parts, separated by spaces: a method name, the local path of the requested resource, and the version of HTTP being used. A typical request line is:

GET /path/to/file/index.html HTTP/1.0
Notes:

GET is the most common HTTP method; it says "give me this resource". Other methods include POST and HEAD-- more on those later. Method names are always uppercase.
The path is the part of the URL after the host name, also called the request URI (a URI is like a URL, but more general).
The HTTP version always takes the form "HTTP/x.x", uppercase.
Return to Table of Contents

Initial Response Line (Status Line)

The initial response line, called the status line, also has three parts separated by spaces: the HTTP version, a response status code that gives the result of the request, and an English reason phrase describing the status code. Typical status lines are:

HTTP/1.0 200 OK
or

HTTP/1.0 404 Not Found
Notes:

The HTTP version is in the same format as in the request line, "HTTP/x.x".
The status code is meant to be computer-readable; the reason phrase is meant to be human-readable, and may vary.
The status code is a three-digit integer, and the first digit identifies the general category of response:
1xx indicates an informational message only
2xx indicates success of some kind
3xx redirects the client to another URL
4xx indicates an error on the client's part
5xx indicates an error on the server's part

Sample HTTP Exchange

To retrieve the file at the URL

http://www.somehost.com/path/file.html
first open a socket to the host www.somehost.com, port 80 (use the default port of 80 because none is specified in the URL). Then, send something like the following through the socket:

GET /path/file.html HTTP/1.0
From: someuser@jmarshall.com
User-Agent: HTTPTool/1.0
[blank line here]

The server should respond with something like the following, sent back through the same socket:

HTTP/1.0 200 OK
Date: Fri, 31 Dec 1999 23:59:59 GMT
Content-Type: text/html
Content-Length: 1354

<html>
<body>
<h1>Happy New Millennium!</h1>
(more file contents)
  .
  .
  .
</body>
</html>

After sending the response, the server closes the socket.

Other HTTP Methods, Like HEAD and POST

The POST Method

A POST request is used to send data to the server to be processed in some way, like by a CGI script. A POST request is different from a GET request in the following ways:

There's a block of data sent with the request, in the message body. There are usually extra headers to describe this message body, like Content-Type: and Content-Length:.
The request URI is not a resource to retrieve; it's usually a program to handle the data you're sending.
The HTTP response is normally program output, not a static file.
The most common use of POST, by far, is to submit HTML form data to CGI scripts. In this case, the Content-Type: header is usually application/x-www-form-urlencoded, and the Content-Length: header gives the length of the URL-encoded form data (here's a note on URL-encoding). The CGI script receives the message body through STDIN, and decodes it. Here's a typical form submission, using POST:

POST /path/script.cgi HTTP/1.0
From: frog@jmarshall.com
User-Agent: HTTPTool/1.0
Content-Type: application/x-www-form-urlencoded
Content-Length: 32

home=Cosby&favorite+flavor=flies

You can use a POST request to send whatever data you want, not just form submissions. Just make sure the sender and the receiving program agree on the format.

URL-encoding

HTML form data is usually URL-encoded to package it in a GET or POST submission. In a nutshell, here's how you URL-encode the name-value pairs of the form data:

Convert all "unsafe" characters in the names and values to "%xx", where "xx" is the ascii value of the character, in hex. "Unsafe" characters include =, &, %, +, non-printable characters, and any others you want to encode-- there's no danger in encoding too many characters. For simplicity, you might encode all non-alphanumeric characters.
Change all spaces to plusses.
String the names and values together with = and &, like
name1=value1&name2=value2&name3=value3
This string is your message body for POST submissions, or the query string for GET submissions.
For example, if a form has a field called "name" that's set to "Lucy", and a field called "neighbors" that's set to "Fred & Ethel", the URL-encoded form data would be
name=Lucy&neighbors=Fred+%26+Ethel
with a length of 34.
Technically, the term "URL-encoding" refers only to step 1 above; it's defined in RFC 2396, section 2.4 (previously in RFC 1738, section 2.2). Commonly, the term refers to this entire process of packaging form data into one long string.

Common HTTP Headers

Request header

Accept: Content-Types that are acceptable for the response.
Content-Type: The MIME type of the body of the request (used with POST and PUT requests)

Response header

HTTP parameter ??

body parameter ?
The values are sent in the request body, in the format that the content type specifies.

Usually the content type is application/x-www-form-urlencoded, so the request body uses the same format as the query string:

parameter=value&also=another
When you use a file upload in the form, you use the multipart/form-data encoding instead, which has a different format. It's more complicated, but you usually don't need to care what it looks like, so I won't show an example, but it can be good to know that it exists.

query parameter

CURL

Options

-H, --header
-X, --request
-k
-d, --data
-v

Make a GET request without any data:

curl http://yilmazhuseyin.com/blog/dev/
curl --request GET 'http://yilmazhuseyin.com/blog/dev/'
curl --X GET 'http://yilmazhuseyin.com/blog/dev/' # -X is same as --request

Both usages are actually the same. However, I try to use second one all the time. With --request (or -X) parameter, we choose our http method to use for our requests. Its values can be GET, POST, DELETE, PUT etc.. If we don't specify this parameter, GET method will be used by default. That is why first version works same as the second one.

ake requests with different HTTP method type without data:

curl --request POST 'http://www.somedomain.com/'
curl --request DELETE 'http://www.somedomain.com/'
curl --request PUT 'http://www.somedomain.com/'

Make requests with data:

Since we learn how to make POST, GET, PUT, DELETE requests, we can now make same requests with data. In order to send data with those requests, we should use --data parameter. Here are some examples:

::::/bin/bash
# send login data with POST request
curl --request POST 'http://www.somedomain.com/login/' \
--data 'username=myusername&password=mypassword'

# send search data to with get request
curl --request GET 'http://www.youtube.com/results?search_query=my_keyword'

curl -i -H "Accept: application/json" \
-H "Content-Type: application/json" \
http://hostname/resource

# send PUT request with data
curl --request PUT 'http://www.somedomain.com/rest-api/user/12345/'\
--data 'email=myemail@gmail.com'

# same thing but this one get data from a file named data.txt
curl --request PUT 'http://www.somedomain.com/rest-api/user/12345/'\
--data @data.txt

cURL with elasticsearch query (CURD)

Create index and insert data

curl -XPUT "http://localhost:9200/movies/movie/1" -d'
{
    "title": "The Godfather",
    "director": "Francis Ford Coppola",
    "year": 1972,
    "genres": ["Crime", "Drama"]
}'

curl -XPUT "http://localhost:9200/movies/movie/2" -d'
{
    "title": "Lawrence of Arabia",
    "director": "David Lean",
    "year": 1962,
    "genres": ["Adventure", "Biography", "Drama"]
}'

Getting by ID

curl -XGET "http://localhost:9200/movies/movie/1" -d''

Deleting documents

curl -XDELETE "http://localhost:9200/movies/movie/1" -d''

Basic free text search (The _search endpoint)

In other words, in order to search for our movies we can make POST requests to either of the following URLs:

http://localhost:9200/_search - Search across all indexes and all types.
http://localhost:9200/movies/_search - Search across all types in the movies index.
http://localhost:9200/movies/movie/_search - Search explicitly for documents of type movie within the movies index.

curl -XPOST "http://localhost:9200/_search" -d'
{
    "query": {
        "query_string": {
            "query": "kill"
        }
    }
}'

HTTP Request