@elazar @elazar@phpc.social #longhornphp

Automating the Web

Matthew Turland

Hello!

My name is Matt
It's nice to see you
Thank you for coming

Forrest Gump waving

TL;DR

Web scraping (extracting data from web sites)
Programmatic interaction with web sites
Consuming web services / APIs
Acceptance testing

Forrest Gump from the film by the same name running out of a gate and down a road

There Will Be Slides

matthewturland.com/presentations

joind.in/talk/5cf41

Michael Scott in The Office repeatedly saying 'PowerPoint.'

Go Ahead, Hashtag It

Feel free to live-tweet/toot!
Hashtag: #longhornphp
Feel free to @ me

Deadpool instructing Negasonic Teenage Warhead to finish typing a tweet into her phone

The Book Was Better

Composer

Packages: find, install, and autoload them

Obligatory Disclaimer

Some uses of this content may have unclear or lacking legality
This content is intended to be strictly educational, not prescriptive or legal advice
If in doubt, consult a lawyer (i.e. not me)

Dr. Ian Malcolm from the film Jurassic Park saying, 'Yeah, yeah, but your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should.'

Storytime

Grandpa from the film 'The Princess Bride' beginning to read

This is Lafayette

A map of Louisiana with Lafayette Parish marked in red

This is Lafayette 911

A screenshot of lafayette911.org

lafayette911.org

This is Ray

Raymond Camden

raymondcamden.com

Ray Created a Viewer

Screenshot of Ray's 911 viewer mapping incidents using Google Maps

Blog Post circa 2010

Six Months Later...

From the Simpsons -- Homer: "No way! When?" Ned Flanders: "Six months ago."

... He Had a Lot of Data

Visualizations of 911 statistics gathered by Ray

Blog Post circa 2010

Can We Recreate It?

Darkwing Duck saying 'Let's get dangerous.'

Use the Source, Luke

Luke Skywalker in Return of the Jedi saying, 'I warn you not to underestimate my power.'

Chrome / Firefox: Right-click > View Page Source

Use the Source, Luke

Into the Fire & Frames

Guitar Hero gameplay of Through the Fire and Flames

Into the Fire & Frames

A screenshot of lafayette911.org with no incidents listed

Into the Fire & Frames

Where's the Data?

Data from Star Trek: The Next Generation walking offscreen

Some Possible Sources

XmlHttpRequest (XHR)
JavaScript-initiated refresh
- e.g. assign(), reload() or replace() methods of Location
Markup-initiated refresh
- e.g. <meta http-equiv="refresh">

Learn the DevTools

Chrome: View > Developer > Developer Tools
Firefox: Tools > Browser Tools > Web Developer Tools

Luke Danes from Gilmore Girls saying, 'With my tools.'

Check for XHRs

Chrome / Firefox: DevTools > Network tab
Chrome: Fetch/XHR filter, Firefox: XHR filter
Chrome / Firefox: click request in table

Network tab in Firefox Developer Tools showing XHRs made by lafayette911.org

Inspect the Request

Firefox Developer Tools showing XHRs made by lafayette911.org

Inspect the Request


POST /L911/Service2.svc/getTrafficIncidents HTTP/2
Host: apps.lafayettela.gov

...

Maurice Moss from the IT Crowd saying, 'I can't turn down a friend request from my mum.'

Inspect the Request

POST = method or operation
/L911/Service2.svc/getTrafficIncidents = Uniform Resource Identifier (URI)
HTTP/2 = client protocol and version
Host = header name
apps.lafayettela.gov = header value
... = request body

Inspect the Response

Firefox Developer Tools showing an XHR response

Inspect the Response

Malcolm Reynolds from Firefly attempting to speak and failing

Inspect the Response

HTTP/2 = server protocol and version
200 = status code
OK = status description
cache-control = header name
private = header value
{"d":"..."} = response body

HTTP

RFC 7230: Hypertext Transfer Protocol (HTTP/1.1): Message Syntax and Routing
RFC 6265: HTTP State Management Mechanism
"Learning HTTP/2: A Practical Guide for Beginners" by Stephen Ludin and Javier Garza
"HTTP: The Definitive Guide" by David Gourley and Brian Totty
Ben Ramsey's blog series on HTTP status codes

Mimic the Request

Streams / Filesystem

Mimic the Request

Streams / Filesystem

Text Fu

Programmers manipulate text the same way woodworkers shape wood. "The Pragmatic Programmer: Your Journey to Mastery, 20th Anniversary Edition" by David Thomas and Andrew Hunt

Neo from the film The Matrix saying, 'I know kung fu.'

Extract the Markup

Apu from The Simpsons saying, 'Look at the outrageous markup.'

Inspect the Markup

Handle Malformations

Ruby Rhod in the film The Fifth Element saying, 'I think mine is broken. Why I gotta get the broke one?'

Handle Malformations

Install tidy extension
Optionally, configure its options
Parse markup
Verify that malformations don't cause data loss

Glen Matthews (AKA Janitor) in the TV series Scrubs making a 'represent' gesture with the caption 'The Clean Up Crew'

Handle Malformations

Handle Malformations

Parse the Markup

Parse the Markup

Parse the Markup

For complex queries: XPath + DOMXPath

Parse the Markup

To use CSS selectors: symfony/css-selector

MDN: Tutorial, Reference

CSS Selectors

A comic strip by Julia Evans on CSS selectors

Trim the Address

String / Multibyte String

Parse the Address

Regular Expressions

/ ... / - expression delimiters
[ ... ] - character class or range
+ - quantifier for 1 or more instances
| - alternation
\s - escape sequence for whitespace characters
{2,} - quantifier for 2 or more instances

Pattern Syntax / Modifiers

Regular Expressions

"Mastering Regular Expressions" by Jeffrey E. F. Friedl
Regular Expressions: Now You Have Two Problems by Jeff Atwood
Regular expressions! by Eric Wastl

Parse the Date

Thank This Guy

Derick Rethans

We Did It!

Borat saying 'Great Success'

Now What?

Store data in JSON files or a database
Put it behind a web server or API
Add a Google Maps frontend
Profit!

An underpants gnome from South Park asking, 'Hey, what's phase two?!'

Lafayette Traffic

Google Maps showing Lafayette, LA with traffic incidents marked Lafayette Traffic showing a live city traffic camera feed

Google Play / GitHub Android, Data

So What?

Gregory House from the TV series House, MD shrugging with the caption 'and?'

On JIRA, Briefly

Screenshot of a project board in JIRA

Standards

cURL

CLI client utility and library for HTTP/HTTPS
Related: PHP cURL extension
CLI client utility is useful for debugging and recreating requests in PHP

A man participating the sport of curling

Copy as cURL

Chrome: DevTools > Network tab > right-click request > Copy > Copy as cURL
Firefox: DevTools > Network tab > right-click request > Copy Value > Copy as cURL

Chris Pratt feigning a bicep curl with a beer

Recreate Requests

Install frizz925/curl-parser
Provide copied cURL command to parser

Recreate Requests

Extract request data

Recreate Requests

Get request object

Send a Request

Using Responses

Debug Requests

Install alexkart/curl-builder
Download cURL

Debug Requests

Convert request to server request

Debug Requests

Convert server request to cURL command

Debug Requests

Tweak cURL command as needed

Add the -v flag to get more verbose output
If the request uses the POST method and has no body, add -X flag with POST as its argument

Debug Requests

Run the cURL command

Repeat Requests

Install deviantintegral/har
Conflicts with frizz925/curl-parser due to
disparities in guzzlehttp/psr7 versions

A repeater from the game Plants v. Zombies

Repeat Requests

Chrome: DevTools > Network tab > right-click in request pane > Save all as HAR with content
Firefox: DevTools > Network tab > right-click in request pane > Save All as HAR
HAR Analyzer

Repeat Requests

Bring Out the Big Guns

Korben Dallas in the film The Fifth Element pointing a gun and laughing

Automate the Browser

Automate the Browser

symfony/dom-crawler

Automate the Browser

This is Fine

WAF = Web Application Firewall
May detect and block automated clients
Related blog posts: Part 1, Part 2

Meme from Adult Swim of a cartoon dog sitting in a house on fire saying, 'This is fine.'

WebDriver Flag

Required by WebDriver implementations
Reveals control by automated means
How to remove it (h/t SO)

WebDriver Flag

Blink Flag

Blink = rendering engine in Chrome browsers
How to disable the flag (h/t SO)
Index of Blink runtime features

Other WebDriver Flag

Required by WebDriver implementations
Reveals control by automated means
Not especially helpful mention of it (thanks SO)

Other WebDriver Flag

Headless Mode

How it's enabled (h/t SO)
Really helpful related article

Quick Demo

Debugging Tips

Use a logger, like monolog/monolog
Use sleep() to block execution

Free Project Idea

PsySH
Add integration with symfony/panther
e.g. custom commands to interactively fetch pages, filter and interact with elements, etc.

Fry from the TV series Futurama saying, 'Shut up and take my money!'

Other Resources

Leonardo DiCaprio in a tuxedo smiling and raising a glass toward the viewer

That's All, Folks

joind.in/talk/5cf41 - Please leave feedback!
matthewturland.com
me@matthewturland.com

Outro for Looney Tunes with Porky Pig saying, 'That's all, folks.'