Accessing
Web Resources
with PHP

Matthew Turland

The World Wide Web

A visualization of Facebook friendships forming a global map

"Web 1.0" - 1990s

Search engines
Hyperlink directories
Beginnings of user-generated content

Yahoo.com circa 1999

"Web 2.0" - 2000s

Rich user interfaces
Social media and networks
Publishing user-generated content becomes trivially easy

Facebook circa 2011

The Semantic Web - 2010s?

Class linkages in open datasets

Diagram illustrating various microformats

Building Blocks

Technologies comprising the Semantic Web

The Semantic Web is a Long Way Off

A desert road leading off into the distance

REST and SOA Adoption is Increasing

A row of mobile phones from various vendors

Web Scraping is Still Used

http://phparch.com </shameless_plug>

Humble Beginnings...

Lafayette911.org

http://lafayette911.org

... Can Lead To Great Things

Listing of addresses ordered by highest traffic incident count

Pie chart showing frequency of various types of traffic incidents

Quality Assurance is Critical

A Nabaztag with flashing lights and a tilted ear

"Nabaztag" pronounced "na-poss-talk" or "na-boss-tug"

Got Skills?

Cat and skateboard in mid-air with the caption 'MAD SKILLZ i haz them'

HTTP
XML
XPath
CSS selectors

HTTP

Drawing highlighting HTTP as a common protocol between browsers and servers

Know Enough To Use Client Libraries

A number and variety of hammers on a table

Also Know Low-Level Details

A microscope

Anatomy of a Request

POST /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org wpAntispam=&wpSection=&wpStarttime=20080719022313

POST = method or operation
/wiki/Main_Page = Uniform Resource Identifier (URI)
HTTP/1.1 = client protocol and version
Host = header name, en.wikipedia.org = header value
Body contains entity data

URI Versus URL

Venn diagram indicating the relationship between URIs and URLs

Related PHP functions
See RFC 3986 for more information

Anatomy of a URL

http://user:pass@domain.com:8080/file.ext?query=&var=value

http = scheme or protocol
user:pass = username and password for authentication
domain.com = hostname
8080 = server port
/file.ext = path
query=&var=value = query string
- ? separates URI and query string
- key-value pairs separated by &
- keys and values separated by =

Anatomy of a Response

HTTP/1.1 200 OK Host: en.wikipedia.org Content-Length: 48363 Content-Type: text/html; charset=utf-8 <!DOCTYPE html> ...

HTTP/1.1 = server protocol and version
200 OK = status code and reason phrase
Host = header name, en.wikipedia.org = header value
Body contains entity data

Status Codes

GitHub 404 page

Divided into five classes:
- 1xx Information
- 2xx Success
- 3xx Redirection
- 4xx Client Error
- 5xx Server Error
See section 10 of RFC 2616 for more information

GET Requests

GET /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org

Devices used to consume e-books and audiobooks

Most commonly used operation
Requests a representation of a resource
Intended to be safe, idempotent, and generally cacheable

HEAD Requests

HEAD /wiki/MainPage HTTP/1.1 Host: en.wikipedia.org

Head of Buddha sculpture

Response contains no body
Header metadata with minimal bandwidth
Safe and idempotent

POST Requests

POST /wiki/ HTTP/1.1 Content-Type: application/x-www-form-urlencoded Content-Length: 57 Host: en.wikipedia.org wpAntispam=&wpSection=&wpStarttime=20080719022313

Captain Jean-Luc Picard from Star Trek: The Next Generation issuing his trademark phrase, "Make it so."

Second most commonly used operation
Accepts an entity as a new subordinate of the resource
Function varies; may result in creation of a new resource
Not safe or idempotent, generally not cacheable

PUT Requests

PUT /wiki/Main_Page HTTP/1.1 Content-Type: application/x-www-form-urlencoded Content-Length: 57 Host: en.wikipedia.org wpAntispam=&wpSection=&wpStarttime=20080719022313

A woman painting over wallpaper

Used with RESTful APIs
Accepts an entity to create or overwrite a resource
PUT URI identifies entity where POST URI handles it
Idempotent, not safe or cacheable

DELETE Requests

DELETE /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org

Strongbad facing a computer screen that says 'DELETED!!'

Used with RESTful APIs
Deletes the specified resource, possibly with a delay
Allows a body, but generally does not include one
Response may include a body
Idempotent, not safe or cacheable

Headers

Custom license plate that says "METADATA"

All-purpose method of including metadata in requests and responses
Used to facilitate a variety of features
- Cookies
- Redirection
- Referring URLs
- Persistent connections
- Content caching
- User agents
- Authentication
See section 14 of RFC 2616 for more information

Cookies

Come to the Dark Side - WE HAVE COOKIES!

Used to retain state between requests, particularly to restrict access to content in web applications
Most clients support handling cookies, some require explicit instruction to do so
Header value contains one name-value pair and zero or more attribute-value pairs
Presence of an "expires" attribute indicates whether a cookie is a session cookie
See RFC 2109 and RFC 2965 for more information

Cookies Example

GET /search?q=test HTTP/1.1 Host: google.com
HTTP/1.1 200 OK Host: google.com Set-Cookie: PREF=ID...; expires=Thu, 11-Apr-2013... <!doctype html><head><title>test - Google Search</title>...
GET /search?q=test HTTP/1.1 Host: google.com Cookie: PREF=ID...; expires=Thu, 11-Apr-2013...

Redirection

Blowfish in a sports car from the Torchwood episode "Kiss Kiss, Bang Bang"

Intended for 201 or 3xx responses
Used by web applications to redirect users after processing POST requests
Misbehaving web applications can instigate infinite redirection loops
Most client libraries offer a way to turn off or limit automatic processing of redirects
See section 14.30 of RFC 2616 for more information

Redirection Example

GET /search?q=test HTTP/1.1 Host: google.com
HTTP/1.1 301 Moved Permanently Host: google.com Location: http://www.google.com/search?q=test
GET /search?q=test HTTP/1.1 Host: www.google.com

Referring URLs

Screenshot of referring sites from Google Analytics

Used by the client to tell the server what URL referred the client to the URL of the current request
Applications:
- traffic analytics
- asset hot-linking prevention
- enforcement of web application workflow
- detection of violations of web application workflow
Common example of a misspelling in a standard
See section 14.36 of RFC 2616 for more information

Referring URL Example

GET /wiki/Main_Page HTTP/1.1 Host: en.wikipedia.org Referer: http://www.google.com/search?q=wikipedia

Persistent Connections

Diagram showing the difference between multiple versus persistent connections

One connection per request involves significant overhead
Persistent connections facilitate multiple requests per connection
HTTP 1.1 uses persistent connections by default
See sections 8.1 and 19.6.2 of RFC 2616 for more information

Persistent Connections Example

GET /search?q=test HTTP/1.1 Host: www.google.com
HTTP/1.1 200 OK Host: google.com ...
GET /images/nav_logo.png HTTP/1.1 Host: www.google.com Connection: close

Content Caching

A calculator displaying a number

Client requests a resource
Server includes headers used to determine if or when that resource may have changed
Client later requests that resource and includes those header values
Server only includes a body if that resource has changed
When dealing with HTTP 1.0, be aware of the Pragma header
See sections 13 and 14 of RFC 2616 for more information

Content Caching - Initial Request

GET /search?q=test HTTP/1.1 Host: www.google.com
HTTP/1.1 200 OK Host: google.com Date: Wed, 12 Apr 2011 02:01:52 GMT Last-Modified: Wed, 13 Apr 2011 03:00:51 GMT ETag: f9edd12ec24f90a6efbcab7be46e3202 Expires: Fri, 13 May 2011 03:00:51 GMT Cache-Control: public, max-age=2592000 ...

Content Caching - Later Requests

GET /search?q=test HTTP/1.1 Host: www.google.com If-Modified-Since: Wed, 13 Apr 2011 03:00:51 GMT If-None-Match: f9edd12ec24f90a6efbcab7be46e3202
HTTP/1.1 304 Not Modified Host: google.com ...

User Agents

Agent Smith and his clones from The Matrix trilogy

Used by the client to identify itself to the server
Sometimes analyzed by servers to determine client capabilities and alter responses accordingly
(ex: OpenDDR, WURFL)
Can be falsified by the client to force the server to deliver a desired response
See section 14.43 of RFC 2616 for more information

User Agent Example

GET /search?q=test HTTP/1.1 Host: www.google.com User-Agent: Mozilla/5.0 (iPad; U; CPU OS 3_2_1 ...

Authentication

Hand pressed onto a biometric identification panel

Client sends a request with no authentication information
Server sends a response requesting authentication
Client resends the request with authentication information
Server sends the resource or a response indicating authentication failure
See RFC 2617 for more information

Authentication - Initial Request

GET /search?q=test HTTP/1.1 Host: www.google.com
HTTP/1.1 401 Unauthorized Host: google.com WWW-Authenticate: Basic realm="Secure Area" ...

Authentication - Later Request

GET /search?q=test HTTP/1.1 Host: www.google.com Authorization: Basic YmlnYmFkd29sZjpsZXRtZWlu
HTTP/1.1 403 Forbidden Host: google.com ...

Know Your Libraries

A corner of the stacks of a library

Lots of HTTP client libraries out there
None are necessarily better; which is most appropriate depends on the circumstances
Research, choose carefully, design to allow for a change of mind later

Sockets

A socket wrench

Pros
- Native language feature
- Simple API
- Flexible
- Allows for low-level debugging
Cons
- Procedural API
- Very low-level
- Limited to features you write yourself
- Requires intimate knowledge of the HTTP protocol

Sockets Example

Streams

Photo of the Ghostbusters with the caption "Don't Cross the Streams. Good advice is timeless."

Pros
- Native language feature
- Simple API
- Good for basic quick-and-dirty one-offs
Cons
- Procedural API
- Clunky error handling
- Little debugging functionality
- Few features

Streams Example

cURL

Pros
- Mature
- Featureful
- Core PHP extension
- Sufficient debugging features
Cons
- Procedural API
- Learning curve is a bit steep
Use rolling-curl for an OO API and non-blocking parallel requests
curl_setopt() function PHP manual page lists configuration constants

cURL Example

rolling-curl Example

pecl_http

Pros
- Built on the same library as the cURL extension
- Both procedural and OO APIs
Cons
- Less availability than the cURL extension
- Occasionally introduces its own bugs, such as bug #16893

pecl_http Example

PEAR

Pros
- Good PHP version coverage (4.2, 4.3, 5.2, 5.3)
- Good environmental coverage (from sockets to cURL)
- Flexibility via subclassing or listeners
- Implementations can be debugged in userland
Cons
- Slightly less performance than C implementations
- Higher minimum PHP version = lower stability
- Lower minimum PHP version = lower likelihood for updates
- Adapters wrap use of extensions and hide some features
Alternative: Zend_Http_Client

PEAR Example

HTTP Tips

A clothes washer/dryer set

Minimize and cache DNS lookups
- cURL and pecl_http handle this natively
- For other extensions, install a local DNS cache like nscd or dnsmasq if possible
- Extend userland code to add DNS caching using gethostbyname() (ex: ZF-5144)
Use batch jobs, request pooling, parallel processes, and client-side caching when available/appropriate
Plan for failure

HTTP Resources

HTTP: The Definitive Guide (ISBN 1565925092)
HTTP Pocket Reference: HyperText Transfer Protocol (ISBN 1565928628)
HTTP Developer's Handbook (ISBN 0672324547) by Chris Shiflett
Ben Ramsey's HTTP blog post series

Tidy

Can make markup more easily consumable by XML extensions
- Corrects malformations in markup
- Converts HTML to XHTML
Formats markup for human readability
Can introduce data loss into extremely malformed markup
See the Tidy PHP manual section for more information

Tidy Example

XML Extensions

Four commonly used extensions
- libxml
- SimpleXML
- DOM
- XMLReader
Two types of parsers
- tree parsers
- pull parsers

libxml Extension

libxml PHP manual section

XML/Tree Concepts

XPath

Regular Expressions : Strings :: XPath : XML
Supported by extensions implementing tree parsers
Excellent tutorial by Tobias Schlitt

SimpleXML

Tree-based parser
Facilitates XML access through PHP object/array access syntax
Best for quick-and-dirty scripts dealing with simple XHTML and XML documents
See the SimpleXML PHP manual section for more information

SimpleXML Example

DOM

Tree parser
Implements the Document Object Model standard
Best for long-lived scripts dealing with complex HTML, XHTML, and XML documents
See the DOM PHP manual section for more information

DOM Example

XMLReader

Pull parser
Works in an iterator-like fashion
Best for scripts that must be memory-conservative or that work with large documents
See the XMLReader PHP manual section for more information

XMLReader Example

CSS Selectors

Can be used as alternative to XPath
Supported by third-party libraries
Excellent visual examples in the jQuery documentation

Regular Expressions

Useful for validating extracted data
Use a concise syntax of meta-characters to describe patterns in strings
Mastering Regular Expressions (MRE) is a great reference
See the PCRE PHP manual section for more information

Who I Work For

Synacor Logo

Synacor Headquarters - Buffalo, NY

Internet solutions for ISPs, media companies, and advertisers
Offices in Buffalo and New York City
Clientele includes most of the top 20 USA cable providers
Great company to work for - join us!
http://www.synacor.com
http://twitter.com/synacor

That's All, Folks

Questions? Comments? Concerns?
Please leave feedback!
http://matthewturland.com
me@matthewturland.com
Elazar on Twitter, GitHub, and Freenode