Author Topic: Help with making a POST request (Read 10672 times)

Joshua Flynn · « **on:** July 18, 2011, 12:06:36 pm »

I am unfamiliar with the processes behind making a POST request to a website. I get that you need to use SetField, but I am unsure how to particularly use it.

I want to be able to use it with this site here:
http://jobseekers.direct.gov.uk/

To submit a request to search for particular jobs in a particular area [the two fields at the top] (if it's possible to set the other options too I'd love to know how).

Presently, the program I've built will automatically parse the website, grab the re-direct, change the HTML code into characters, and connect to the re-direct but presently I can't get my program to send a POST request to begin the search. So any help on how would be greatly appreciated.

Thank you.

Laurent · « **Reply #1 on:** July 18, 2011, 12:19:25 pm »

Hi

Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?

Quote

I get that you need to use SetField

nop, SetField only sets header fields. For POST requests, everything's in the body, it looks like this:

Code: [Select]

param1=xxx&param2=yyy&param3=zzz

Joshua Flynn · « **Reply #2 on:** July 18, 2011, 12:22:20 pm »

Quote from: "Laurent"

Hi

Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?

I admittedly haven't tried anything because I don't know what to send as I am unfamiliar with how a POST request works. I am using SFML 1.6. If there is someway to upload files then I can supply the two header files and .cpp.

I can't do a minimalist as the site uses a re-direction that needs to be parsed and altered before being re-submitted.

Joshua Flynn · « **Reply #3 on:** July 18, 2011, 12:24:22 pm »

Quote from: "Laurent"

Hi

Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?

Quote
I get that you need to use SetField

nop, SetField only sets header fields. For POST requests, everything's in the body, it looks like this:
Code: [Select]
param1=xxx&param2=yyy&param3=zzz

Okay, well, as I said I am unfamiliar with how it works. Say I wanted to alter the job description field... do I access the page, get a copy of the body (which it now presently does)... then do I parse and alter parts of the text and resend it back? Or do I create a new body from scratch?

Sorry no idea what I am doing here.

Joshua Flynn · « **Reply #4 on:** July 18, 2011, 12:43:47 pm »

As for your code example request, I couldn't simplify it, but I merged the header files and .cpp together:

http://pastebin.com/ACBu5FjC

It should work but it's untested.

Laurent · « **Reply #5 on:** July 18, 2011, 01:13:01 pm »

Before doing anything you need to find out how the page works. Are you sure that it uses POST? What parameters are understood by the page? What are their names? ...

And maybe there's simply no way to fill the page fields automatically. If the website's programers haven't implemented a GET or POST method to do so, you won't be able to do anything.

Joshua Flynn · « **Reply #6 on:** July 18, 2011, 01:33:11 pm »

Quote from: "Laurent"

Before doing anything you need to find out how the page works. Are you sure that it uses POST? What parameters are understood by the page? What are their names? ...

And maybe there's simply no way to fill the page fields automatically. If the website's programers haven't implemented a GET or POST method to do so, you won't be able to do anything.

I am completely unfamiliar with HTML in that respect. But getting the body works (you get the entire page, all the options etc). There must be some way of inputting data into those fields surely? If not by POST, by some other method implementable via C++?

Argh another idea scrapped.

Laurent · « **Reply #7 on:** July 18, 2011, 01:39:48 pm »

If there's no GET or POST method to fill the page, the only option that is left to you is to use javascript to populate the fields after the page is loaded. You won't be able to do anything in C++.

Joshua Flynn · « **Reply #8 on:** July 18, 2011, 02:01:10 pm »

Quote from: "Laurent"

If there's no GET or POST method to fill the page, the only option that is left to you is to use javascript to populate the fields after the page is loaded. You won't be able to do anything in C++.

Do you know an example of a site that does GET and POST, just so I can test this out?

Laurent · « **Reply #9 on:** July 18, 2011, 02:07:45 pm »

A GET example is easy to find, since everything appears in the URL. You can test Google for example: http://www.google.fr/search?q=sfml

Which decomposes into:
- server: "http://www.google.fr"
- URI: "search?q=sfml"

For a POST example, you must know the code of the website since everything's hidden in the body of the request. Or use something that can show you the full contents of the request that is sent to the website.

But anyway, that won't help you for your particular website.

Joshua Flynn · « **Reply #10 on:** July 18, 2011, 02:18:48 pm »

Quote from: "Laurent"

But anyway, that won't help you for your particular website.

I got the GET working. It pulls back the site code (I do this with the NOAA ACE satellite server to get the text documents).

But that's okay, if I can get it to work with other sites I might be able to work around the problem.

Quote from: "Laurent"

Or use something that can show you the full contents of the request that is sent to the website.

Do you know of any free software that can do this? I'd love to know what is being sent so I can deconstruct it and learn from it.

Laurent · « **Reply #11 on:** July 18, 2011, 02:27:07 pm »

Quote

Do you know of any free software that can do this? I'd love to know what is being sent so I can deconstruct it and learn from it.

No, but that shouldn't be hard to find. Maybe your web browser can do it (with a plugin).

Joshua Flynn · « **Reply #12 on:** July 19, 2011, 04:18:27 pm »

Hi, just updating this thread for anyone interested.

The tool I used was an stand-alone that integrates itself with most browsers called 'Fiddler' or 'Fiddler2' depending. Allows analysis of POST and GET data. In this case, I use the request builder to analyse how the POST data was being sent raw, and duplicated it's contents.

Effectively spoofing a previous request works, a bit hacky, but works. I am hoping to reconstruct it so it works above board. Lots of redirects involved.

Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).

Laurent · « **Reply #13 on:** July 19, 2011, 04:21:06 pm »

Quote

Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).

Can you explain what you mean?

Joshua Flynn · « **Reply #14 on:** July 21, 2011, 03:25:40 pm »

Quote from: "Laurent"

Quote
Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).

Can you explain what you mean?

Well, normally, when the site supplies a 302 (redirection), I have to write a custom function (that may not be best optimised, especially if part of SFML already knows the redirect link) to parse the GetBody data (in a seperate char array) to extract the supplied redirection link, then to recall the request function to follow it. I was wondering if SFML had a 'follow redirect' type function to perhaps optimise and simplify the processing of URL re-directs.