SFML community forums
Help => Network => Topic started by: Joshua Flynn on July 18, 2011, 12:06:36 pm
-
I am unfamiliar with the processes behind making a POST request to a website. I get that you need to use SetField, but I am unsure how to particularly use it.
I want to be able to use it with this site here:
http://jobseekers.direct.gov.uk/
To submit a request to search for particular jobs in a particular area [the two fields at the top] (if it's possible to set the other options too I'd love to know how).
Presently, the program I've built will automatically parse the website, grab the re-direct, change the HTML code into characters, and connect to the re-direct but presently I can't get my program to send a POST request to begin the search. So any help on how would be greatly appreciated.
Thank you.
-
Hi
Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?
I get that you need to use SetField
nop, SetField only sets header fields. For POST requests, everything's in the body, it looks like this:
param1=xxx¶m2=yyy¶m3=zzz
-
Hi
Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?
I admittedly haven't tried anything because I don't know what to send as I am unfamiliar with how a POST request works. I am using SFML 1.6. If there is someway to upload files then I can supply the two header files and .cpp.
I can't do a minimalist as the site uses a re-direction that needs to be parsed and altered before being re-submitted.
-
Hi
Can you show us what've you've tried so far? Are you using SFML 1.6 or 2?
I get that you need to use SetField
nop, SetField only sets header fields. For POST requests, everything's in the body, it looks like this:
param1=xxx¶m2=yyy¶m3=zzz
Okay, well, as I said I am unfamiliar with how it works. Say I wanted to alter the job description field... do I access the page, get a copy of the body (which it now presently does)... then do I parse and alter parts of the text and resend it back? Or do I create a new body from scratch?
Sorry no idea what I am doing here.
-
As for your code example request, I couldn't simplify it, but I merged the header files and .cpp together:
http://pastebin.com/ACBu5FjC
It should work but it's untested.
-
Before doing anything you need to find out how the page works. Are you sure that it uses POST? What parameters are understood by the page? What are their names? ...
And maybe there's simply no way to fill the page fields automatically. If the website's programers haven't implemented a GET or POST method to do so, you won't be able to do anything.
-
Before doing anything you need to find out how the page works. Are you sure that it uses POST? What parameters are understood by the page? What are their names? ...
And maybe there's simply no way to fill the page fields automatically. If the website's programers haven't implemented a GET or POST method to do so, you won't be able to do anything.
I am completely unfamiliar with HTML in that respect. But getting the body works (you get the entire page, all the options etc). There must be some way of inputting data into those fields surely? If not by POST, by some other method implementable via C++?
Argh another idea scrapped.
-
If there's no GET or POST method to fill the page, the only option that is left to you is to use javascript to populate the fields after the page is loaded. You won't be able to do anything in C++.
-
If there's no GET or POST method to fill the page, the only option that is left to you is to use javascript to populate the fields after the page is loaded. You won't be able to do anything in C++.
Do you know an example of a site that does GET and POST, just so I can test this out?
-
A GET example is easy to find, since everything appears in the URL. You can test Google for example: http://www.google.fr/search?q=sfml
Which decomposes into:
- server: "http://www.google.fr"
- URI: "search?q=sfml"
For a POST example, you must know the code of the website since everything's hidden in the body of the request. Or use something that can show you the full contents of the request that is sent to the website.
But anyway, that won't help you for your particular website.
-
But anyway, that won't help you for your particular website.
I got the GET working. It pulls back the site code (I do this with the NOAA ACE satellite server to get the text documents).
But that's okay, if I can get it to work with other sites I might be able to work around the problem.
Or use something that can show you the full contents of the request that is sent to the website.
Do you know of any free software that can do this? I'd love to know what is being sent so I can deconstruct it and learn from it.
-
Do you know of any free software that can do this? I'd love to know what is being sent so I can deconstruct it and learn from it.
No, but that shouldn't be hard to find. Maybe your web browser can do it (with a plugin).
-
Hi, just updating this thread for anyone interested.
The tool I used was an stand-alone that integrates itself with most browsers called 'Fiddler' or 'Fiddler2' depending. Allows analysis of POST and GET data. In this case, I use the request builder to analyse how the POST data was being sent raw, and duplicated it's contents.
Effectively spoofing a previous request works, a bit hacky, but works. I am hoping to reconstruct it so it works above board. Lots of redirects involved.
Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).
-
Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).
Can you explain what you mean?
-
Laurent, I am not sure if you have this feature already available, but you should have the option to return a re-direction link (or follow a re-direction link depending).
Can you explain what you mean?
Well, normally, when the site supplies a 302 (redirection), I have to write a custom function (that may not be best optimised, especially if part of SFML already knows the redirect link) to parse the GetBody data (in a seperate char array) to extract the supplied redirection link, then to recall the request function to follow it. I was wondering if SFML had a 'follow redirect' type function to perhaps optimise and simplify the processing of URL re-directs.
-
I think that what's contained in the body is simply a default, human-readable page that allows the user to know what happened in case the browser does not implement the redirection. But if you want to implement the redirection, you should rather look into the header's fields -- I think the new URL is under "Location" for a redirection.
std::string newUri = response.GetField("Location");
-
I think that what's contained in the body is simply a default, human-readable page that allows the user to know what happened in case the browser does not implement the redirection. But if you want to implement the redirection, you should rather look into the header's fields -- I think the new URL is under "Location" for a redirection.
std::string newUri = response.GetField("Location");
Sorry I took some time. Was developing the program.
Is there a way to see the header fields (like accessing it like a string), so I can see what's inside and thus know which field to reference it to?
-
Is there a way to see the header fields
No.
-
Is there a way to see the header fields
No.
Okay. Is there any way to know what the header fields contain? Given there is no way I can guess what they contain.
And, is it possible to access/download images using the Get method (or something similar)?
-
Is there any way to know what the header fields contain?
How is it different from your previous question? :D
And, is it possible to access/download images using the Get method (or something similar)?
Yes, they are like any other web page. You just have to interpret the body differently, based on the MIME type returned in the header.
-
Yes, they are like any other web page. You just have to interpret the body differently, based on the MIME type returned in the header.
So assuming I knew what to expect (the image is a GIF, two versions, one is static, one is multiple images in a loop), and I can obtain the link for it's location, how would I download the image, and save it as a file?
How is it different from your previous question? :D
I suppose I was trying to somehow broaden it. Is there a tool that can access the header? I mean, how do you know a location exists inside the header if it's not possible to see it before?
-
So assuming I knew what to expect (the image is a GIF, two versions, one is static, one is multiple images in a loop), and I can obtain the link for it's location, how would I download the image, and save it as a file?
You build a GET request with the image location, and you directly dump the body of the response to a file. Normally.
Is there a tool that can access the header? I mean, how do you know a location exists inside the header if it's not possible to see it before?
You're supposed to know which header field to retrieve (after reading carefully the HTTP RFC (http://www.w3.org/Protocols/rfc2616/rfc2616.html)).
-
You build a GET request with the image location, and you directly dump the body of the response to a file. Normally.
Okay. Sounds good and straight forward enough to me.
You're supposed to know which header field to retrieve (after reading carefully the HTTP RFC (http://www.w3.org/Protocols/rfc2616/rfc2616.html)).
I'll read through that.
It gives me the impression - is SFML capable of sending emails? And if so, how would you configure it to do so?
-
It gives me the impression - is SFML capable of sending emails? And if so, how would you configure it to do so?
No no. E-mails have their own protocol (SMTP, for sending), it's a totally different thing and SFML doesn't implement that.
-
No no. E-mails have their own protocol (SMTP, for sending), it's a totally different thing and SFML doesn't implement that.
Okay. Last couple of questions. For writing the images, would I use normal (text) or binary mode for that?
And, GET response returns things like '%3f' to represent question marks, etc - what representative method is this (is there a list of all these I can use)? And is there some way of automatically converting one from the other without having to implement my own functions?
-
Okay. Last couple of questions. For writing the images, would I use normal (text) or binary mode for that?
I have no idea, that's where my knowledge ends, sorry ;)
Maybe you'll need to decode the response body first, it might be encoded differently.
And, GET response returns things like '%3f' to represent question marks, etc - what representative method is this (is there a list of all these I can use)? And is there some way of automatically converting one from the other without having to implement my own functions?
You should have a look at this:
http://en.wikipedia.org/wiki/Percent-encoding
I think you should read articles/tutorials about the concepts involved here, since you're new to many of them. A good global understanding is necessary before trying to implement things.