Comments on: Gotcha on Scraping .NET Applications with PHP and cURL http://matthewturland.com/2010/06/30/gotcha-on-scraping-net-applications-with-php-and-curl/ Sat, 31 Dec 2011 15:29:33 +0000 hourly 1 http://wordpress.org/?v=3.3.1 By: EvNix http://matthewturland.com/2010/06/30/gotcha-on-scraping-net-applications-with-php-and-curl/comment-page-1/#comment-5246 EvNix Sun, 04 Jul 2010 12:24:06 +0000 http://matthewturland.com/?p=365#comment-5246 nice tutorial i prefer using snoopy class though! i guess snoopy uses fsockopen but i am not very sure nice tutorial i prefer using snoopy class though!
i guess snoopy uses fsockopen
but i am not very sure

]]> By: Matthew Turland http://matthewturland.com/2010/06/30/gotcha-on-scraping-net-applications-with-php-and-curl/comment-page-1/#comment-5203 Matthew Turland Thu, 01 Jul 2010 21:15:30 +0000 http://matthewturland.com/?p=365#comment-5203 @Daniel Thanks for the taking the time to comment and make that clarification. I didn't mean to imply that this was an error on the part of libcurl or its PHP extension, merely a behavior of the latter that's easy to miss if you don't read the docs carefully. I agree that the end result is that the POST data is formatted differently and the difference in header value is more a consequence of that and a sympton rather than a cause. One thing I didn't mention explicitly is that I've never known PHP to behave like .NET does in this situation, so the behavior came as somewhat of a surprise to me. @Daniel Thanks for the taking the time to comment and make that clarification. I didn’t mean to imply that this was an error on the part of libcurl or its PHP extension, merely a behavior of the latter that’s easy to miss if you don’t read the docs carefully.

I agree that the end result is that the POST data is formatted differently and the difference in header value is more a consequence of that and a sympton rather than a cause. One thing I didn’t mention explicitly is that I’ve never known PHP to behave like .NET does in this situation, so the behavior came as somewhat of a surprise to me.

]]> By: Daniel Stenberg http://matthewturland.com/2010/06/30/gotcha-on-scraping-net-applications-with-php-and-curl/comment-page-1/#comment-5202 Daniel Stenberg Thu, 01 Jul 2010 20:59:32 +0000 http://matthewturland.com/?p=365#comment-5202 Hi. I'm the main author of libcurl, and while I'm not fluent in PHP nor in the curl extension for PHP, I must say that your post/explanation here greatly misses the point. The problem is really _not_ the Content-Type: header. Most likely servers don't care one yota about that header. What counts, is that you made a multipart formpost instead of a regular one. The entire POST was done in a completely different encoding as url-encoding and multipart are far far away from each other. (Me personally, I don't like how the curl exension made the two different posts that easy to mix up.) Have fun with that scraping! Hi.

I’m the main author of libcurl, and while I’m not fluent in PHP nor in the curl extension for PHP, I must say that your post/explanation here greatly misses the point. The problem is really _not_ the Content-Type: header. Most likely servers don’t care one yota about that header.

What counts, is that you made a multipart formpost instead of a regular one. The entire POST was done in a completely different encoding as url-encoding and multipart are far far away from each other.

(Me personally, I don’t like how the curl exension made the two different posts that easy to mix up.)

Have fun with that scraping!

]]> By: Daniel Ice http://matthewturland.com/2010/06/30/gotcha-on-scraping-net-applications-with-php-and-curl/comment-page-1/#comment-5197 Daniel Ice Thu, 01 Jul 2010 14:46:31 +0000 http://matthewturland.com/?p=365#comment-5197 Great tip. I have been doing lots of spidering over time and never seen this tip. Thanks. Great tip. I have been doing lots of spidering over time and never seen this tip. Thanks.

]]>